Real Python

How to Use Generators and yield in Python

2019-09-25T15:52:37+00:00

Have you ever had to work with a dataset so large that it overwhelmed your machine’s memory? Or maybe you have a complex function that needs to maintain an internal state every time it’s called, but the function is too small to justify creating its own class. In these cases and more, generators and the Python yield statement are here to help.

By the end of this article, you’ll know:

What generators are and how to use them
How to create generator functions and expressions
How the Python yield statement works
How to use multiple Python yield statements in a generator function
How to use advanced generator methods
How to build data pipelines with multiple generators

If you’re a beginner or intermediate Pythonista and you’re interested in learning how to work with large datasets in a more Pythonic fashion, then this is the tutorial for you.

You can get a copy of the dataset used in this tutorial by clicking the link below:

Download Dataset: Click here to download the dataset you'll use in this tutorial to learn about generators and yield in Python.

Using Generators

Introduced with PEP 255, generator functions are a special kind of function that return a lazy iterator. These are objects that you can loop over like a list. However, unlike lists, lazy iterators do not store their contents in memory. For an overview of iterators in Python, take a look at Python “for” Loops (Definite Iteration).

Now that you have a rough idea of what a generator does, you might wonder what they look like in action. Let’s take a look at two examples. In the first, you’ll see how generators work from a bird’s eye view. Then, you’ll zoom in and examine each example more thoroughly.

Example 1: Reading Large Files

A common use case of generators is to work with data streams or large files, like CSV files. These text files separate data into columns by using commas. This format is a common way to share data. Now, what if you want to count the number of rows in a CSV file? The code block below shows one way of counting those rows:

csv_gen = csv_reader("some_csv.txt")
row_count = 0

for row in csv_gen:
    row_count += 1

print(f"Row count is {row_count}")

Looking at this example, you might expect csv_gen to be a list. To populate this list, csv_reader() opens a file and loads its contents into csv_gen. Then, the program iterates over the list and increments row_count for each row.

This is a reasonable explanation, but would this design still work if the file is very large? What if the file is larger than the memory you have available? To answer this question, let’s assume that csv_reader() just opens the file and reads it into an array:

def csv_reader(file_name):
    file = open(file_name)
    result = file.read().split("\n")
    return result

This function opens a given file and uses file.read() along with .split() to add each line as a separate element to a list. If you were to use this version of csv_reader() in the row counting code block you saw further up, then you’d get the following output:

>>>

Traceback (most recent call last):
  File "ex1_naive.py", line 22, in <module>
    main()
  File "ex1_naive.py", line 13, in main
    csv_gen = csv_reader("file.txt")
  File "ex1_naive.py", line 6, in csv_reader
    result = file.read().split("\n")
MemoryError

In this case, open() returns a generator object that you can lazily iterate through line by line. However, file.read().split() loads everything into memory at once, causing the MemoryError.

Before that happens, you’ll probably notice your computer slow to a crawl. You might even need to kill the program with a KeyboardInterrupt. So, how can you handle these huge data files? Take a look at a new definition of csv_reader():

def csv_reader(file_name):
    for row in open(file_name, "r"):
        yield row

In this version, you open the file, iterate through it, and yield a row. This code should produce the following output, with no memory errors:

Row count is 64186394

What’s happening here? Well, you’ve essentially turned csv_reader() into a generator function. This version opens a file, loops through each line, and yields each row, instead of returning it.

You can also define a generator expression (also called a generator comprehension), which has a very similar syntax to list comprehensions. In this way, you can use the generator without calling a function:

csv_gen = (row for row in open(file_name))

This is a more succinct way to create the list csv_gen. You’ll learn more about the Python yield statement soon. For now, just remember this key difference:

Using yield will result in a generator object.
Using return will result in the first line of the file only.

Example 2: Generating an Infinite Sequence

Let’s switch gears and look at infinite sequence generation. In Python, to get a finite sequence, you call range() and evaluate it in a list context:

>>>

>>> a = range(5)
>>> list(a)
[0, 1, 2, 3, 4, 5]

Generating an infinite sequence, however, will require the use of a generator, since your computer memory is finite:

def infinite_sequence():
    num = 0
    while True:
        yield num
        num += 1

This code block is short and sweet. First, you initialize the variable num and start an infinite loop. Then, you immediately yield num so that you can capture the initial state. This mimics the action of range().

After yield, you increment num by 1. If you try this with a for loop, then you’ll see that it really does seem infinite:

>>>

>>> for i in infinite_sequence():
...     print(i, end=" ")
...
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42
[...]
6157818 6157819 6157820 6157821 6157822 6157823 6157824 6157825 6157826 6157827
6157828 6157829 6157830 6157831 6157832 6157833 6157834 6157835 6157836 6157837
6157838 6157839 6157840 6157841 6157842
KeyboardInterrupt
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>

The program will continue to execute until you stop it manually.

Instead of using a for loop, you can also call next() on the generator object directly. This is especially useful for testing a generator in the console:

>>>

>>> gen = infinite_sequence()
>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
3

Here, you have a generator called gen, which you manually iterate over by repeatedly calling next(). This works as a great sanity check to make sure your generators are producing the output you expect.

Note: When you use next(), Python calls .__next__() on the function you pass in as a parameter. There are some special effects that this parameterization allows, but it goes beyond the scope of this article. Experiment with changing the parameter you pass to next() and see what happens!

Example 3: Detecting Palindromes

You can use infinite sequences in many ways, but one practical use for them is in building palindrome detectors. A palindrome detector will locate all sequences of letters or numbers that are palindromes. These are words or numbers that are read the same forward and backward, like 121. First, define your numeric palindrome detector:

def is_palindrome(num):
    # Skip single-digit inputs
    if num // 10 == 0:
        return False
    temp = num
    reversed_num = 0

    while temp != 0:
        reversed_num = (reversed_num * 10) + (temp % 10)
        temp = temp // 10

    if num == reversed_num:
        return num
    else:
        return False

Don’t worry too much about understanding the underlying math in this code. Just note that the function takes an input number, reverses it, and checks to see if the reversed number is the same as the original. Now you can use your infinite sequence generator to get a running list of all numeric palindromes:

>>>

>>> for i in infinite_sequence():
...     pal = is_palindrome(i)
...     if pal:
...         print(pal)
...
11
22
33
[...]
99799
99899
99999
100001
101101
102201
KeyboardInterrupt
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 5, in is_palindrome

In this case, the only numbers that are printed to the console are those that are the same forward or backward.

Note: In practice, you’re unlikely to write your own infinite sequence generator. The itertools module provides a very efficient infinite sequence generator with itertools.count().

Now that you’ve seen a simple use case for an infinite sequence generator, let’s dive deeper into how generators work.

Understanding Generators

So far, you’ve learned about the two primary ways of creating generators: by using generator functions and generator expressions. You might even have an intuitive understanding of how generators work. Let’s take a moment to make that knowledge a little more explicit.

Generator functions look and act just like regular functions, but with one defining characteristic. Generator functions use the Python yield keyword instead of return. Recall the generator function you wrote earlier:

def infinite_sequence():
    num = 0
    while True:
        yield num
        num += 1

This looks like a typical function definition, except for the Python yield statement and the code that follows it. yield indicates where a value is sent back to the caller, but unlike return, you don’t exit the function afterward.

Instead, the state of the function is remembered. That way, when next() is called on a generator object (either explicitly or implicitly within a for loop), the previously yielded variable num is incremented, and then yielded again. Since generator functions look like other functions and act very similarly to them, you can assume that generator expressions are very similar to other comprehensions available in Python.

Note: Are you rusty on Python’s list, set, and dictionary comprehensions? You can check out Using List Comprehensions Effectively.

Building Generators With Generator Expressions

Like list comprehensions, generator expressions allow you to quickly create a generator object in just a few lines of code. They’re also useful in the same cases where list comprehensions are used, with an added benefit: you can create them without building and holding the entire object in memory before iteration. In other words, you’ll have no memory penalty when you use generator expressions. Take this example of squaring some numbers:

>>>

>>> nums_squared_lc = [num**2 for num in range(5)]
>>> nums_squared_gc = (num**2 for num in range(5))

Both nums_squared_lc and nums_squared_gc look basically the same, but there’s one key difference. Can you spot it? Take a look at what happens when you inspect each of these objects:

>>>

>>> nums_squared_lc
[0, 1, 4, 9, 16]
>>> nums_squared_gc
<generator object <genexpr> at 0x107fbbc78>

The first object used brackets to build a list, while the second created a generator expression by using parentheses. The output confirms that you’ve created a generator object and that it is distinct from a list.

Profiling Generator Performance

You learned earlier that generators are a great way to optimize memory. While an infinite sequence generator is an extreme example of this optimization, let’s amp up the number squaring examples you just saw and inspect the size of the resulting objects. You can do this with a call to sys.getsizeof():

>>>

>>> import sys
>>> nums_squared_lc = [i * 2 for i in range(10000)]
>>> sys.getsizeof(nums_squared_lc)
87624
>>> nums_squared_gc = (i ** 2 for i in range(10000))
>>> print(sys.getsizeof(nums_squared_gc))
120

In this case, the list you get from the list comprehension is 87,624 bytes, while the generator object is only 120. This means that the list is over 700 times larger than the generator object!

There is one thing to keep in mind, though. If the list is smaller than the running machine’s available memory, then list comprehensions can be faster to evaluate than the equivalent generator expression. To explore this, let’s sum across the results from the two comprehensions above. You can generate a readout with cProfile.run():

>>>

>>> import cProfile
>>> cProfile.run('sum([i * 2 for i in range(10000)])')
         5 function calls in 0.001 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.001    0.001    0.001    0.001 <string>:1(<listcomp>)
        1    0.000    0.000    0.001    0.001 <string>:1(<module>)
        1    0.000    0.000    0.001    0.001 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


>>> cProfile.run('sum((i * 2 for i in range(10000)))')
         10005 function calls in 0.003 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10001    0.002    0.000    0.002    0.000 <string>:1(<genexpr>)
        1    0.000    0.000    0.003    0.003 <string>:1(<module>)
        1    0.000    0.000    0.003    0.003 {built-in method builtins.exec}
        1    0.001    0.001    0.003    0.003 {built-in method builtins.sum}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Here, you can see that summing across all values in the list comprehension took about a third of the time as summing across the generator. If speed is an issue and memory isn’t, then a list comprehension is likely a better tool for the job.

Note: These measurements aren’t only valid for objects made with generator expressions. They’re also the same for objects made from the analogous generator function since the resulting generators are equivalent.

Remember, list comprehensions return full lists, while generator expressions return generators. Generators work the same whether they’re built from a function or an expression. Using an expression just allows you to define simple generators in a single line, with an assumed yield at the end of each inner iteration.

The Python yield statement is certainly the linchpin on which all of the functionality of generators rests, so let’s dive into how yield works in Python.

Understanding the Python Yield Statement

On the whole, yield is a fairly simple statement. Its primary job is to control the flow of a generator function in a way that’s similar to return statements. As briefly mentioned above, though, the Python yield statement has a few tricks up its sleeve.

When you call a generator function or use a generator expression, you return a special iterator called a generator. You can assign this generator to a variable in order to use it. When you call special methods on the generator, such as next(), the code within the function is executed up to yield.

When the Python yield statement is hit, the program suspends function execution and returns the yielded value to the caller. (In contrast, return stops function execution completely.) When a function is suspended, the state of that function is saved. This includes any variable bindings local to the generator, the instruction pointer, the internal stack, and any exception handling.

This allows you to resume function execution whenever you call one of the generator’s methods. In this way, all function evaluation picks back up right after yield. You can see this in action by using multiple Python yield statements:

>>>

>>> def multi_yield():
...     yield_str = "This will print the first string"
...     yield yield_str
...     yield_str = "This will print the second string"
...     yield yield_str
...
>>> multi_obj = multi_yield()
>>> print(next(multi_obj))
This will print the first string
>>> print(next(multi_obj))
This will print the second string
>>> print(next(multi_obj))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Take a closer look at that last calls to next(). You can see that execution has blown up with a traceback. This is because generators, like all iterators, can be exhausted. Unless your generator is infinite, you can iterate through it one time only. Once all values have been evaluated, iteration will stop and the for loop will exit. If you used next(), then instead you’ll get an explicit StopIteration exception.

Note: StopIteration is a natural exception that’s raised to signal the end of an iterator. for loops, for example, are built around StopIteration. You can even implement your own for loop by using a while loop:

>>>

>>> letters = ["a", "b", "c", "y"]
>>> it = iter(letters)
>>> while True:
...     try:
...         letter = next(it)
...     except StopIteration:
...         break
...     print(letter)
...
a
b
c
y

You can read more about StopIteration in the Python documentation on exceptions. For more on iteration in general, check out Python “for” Loops (Definite Iteration) and Python “while” Loops (Indefinite Iteration).

yield can be used in many ways to control your generator’s execution flow. The use of multiple Python yield statements can be leveraged as far as your creativity allows.

Using Advanced Generator Methods

You’ve seen the most common uses and constructions of generators, but there are a few more tricks to cover. In addition to yield, generator objects can make use of the following methods:

.send()
.throw()
.close()

How to Use `.send()`

For this next section, you’re going to build a program that makes use of all three methods. This program will print numeric palindromes like before, but with a few tweaks. Upon encountering a palindrome, your new program will add a digit and start a search for the next one from there. You’ll also handle exceptions with .throw() and stop the generator after a given amount of digits with .close(). First, let’s recall the code for your palindrome detector:

def is_palindrome(num):
    # Skip single-digit inputs
    if num // 10 == 0:
        return False
    temp = num
    reversed_num = 0

    while temp != 0:
        reversed_num = (reversed_num * 10) + (temp % 10)
        temp = temp // 10

    if num == reversed_num:
        return True
    else:
        return False

This is the same code you saw earlier, except that now the program returns strictly True or False. You’ll also need to modify your original infinite sequence generator, like so:

 1 def infinite_palindromes():
 2     num = 0
 3     while True:
 4         if is_palindrome(num):
 5             i = (yield num)
 6             if i is not None:
 7                 num = i
 8         num += 1

There are a lot of changes here! The first one you’ll see is in line 5, where i = (yield num). Though you learned earlier that yield is a statement, that isn’t quite the whole story.

As of Python 2.5 (the same release that introduced the methods you are learning about now), yield is an expression, rather than a statement. Of course, you can still use it as a statement. But now, you can also use it as you see in the code block above, where i takes the value that is yielded. This allows you to manipulate the yielded value. More importantly, it allows you to .send() a value back to the generator. When execution picks up after yield, i will take the value that is sent.

You’ll also check if i is not None, which could happen if next() is called on the generator object. (This can also happen when you iterate with a for loop.) If i has a value, then you update num with the new value. But regardless of whether or not i holds a value, you’ll then increment num and start the loop again.

Now, take a look at the main function code, which sends the lowest number with another digit back to the generator. For example, if the palindrome is 121, then it will .send() 1000:

pal_gen = infinite_palindromes()
for i in pal_gen:
    digits = len(str(i))
    pal_gen.send(10 ** (digits))

With this code, you create the generator object and iterate through it. The program only yields a value once a palindrome is found. It uses len() to determine the number of digits in that palindrome. Then, it sends 10 ** digits to the generator. This brings execution back into the generator logic and assigns 10 ** digits to i. Since i now has a value, the program updates num, increments, and checks for palindromes again.

Once your code finds and yields another palindrome, you’ll iterate via the for loop. This is the same as iterating with next(). The generator also picks up at line 5 with i = (yield num). However, now i is None, because you didn’t explicitly send a value.

What you’ve created here is a coroutine, or a generator function into which you can pass data. These are useful for constructing data pipelines, but as you’ll see soon, they aren’t necessary for building them. (If you’re looking to dive deeper, then this course on coroutines and concurrency is one of the most comprehensive treatments available.)

Now that you’ve learned about .send(), let’s take a look at .throw().

How to Use `.throw()`

.throw() allows you to throw exceptions with the generator. In the below example, you raise the exception in line 6. This code will throw a ValueError once digits reaches 5:

 1 pal_gen = infinite_palindromes()
 2 for i in pal_gen:
 3     print(i)
 4     digits = len(str(i))
 5     if digits == 5:
 6         pal_gen.throw(ValueError("We don't like large palindromes"))
 7     pal_gen.send(10 ** (digits))

This is the same as the previous code, but now you’ll check if digits is equal to 5. If so, then you’ll .throw() a ValueError. To confirm that this works as expected, take a look at the code’s output:

>>>

11
111
1111
10101
Traceback (most recent call last):
  File "advanced_gen.py", line 47, in <module>
    main()
  File "advanced_gen.py", line 41, in main
    pal_gen.throw(ValueError("We don't like large palindromes"))
  File "advanced_gen.py", line 26, in infinite_palindromes
    i = (yield num)
ValueError: We don't like large palindromes

.throw() is useful in any areas where you might need to catch an exception. In this example, you used .throw() to control when you stopped iterating through the generator. You can do this more elegantly with .close().

How to Use `.close()`

As its name implies, .close() allows you to stop a generator. This can be especially handy when controlling an infinite sequence generator. Let’s update the code above by changing .throw() to .close() to stop the iteration:

 1 pal_gen = infinite_palindromes()
 2 for i in pal_gen:
 3     print(i)
 4     digits = len(str(i))
 5     if digits == 5:
 6         pal_gen.close()
 7     pal_gen.send(10 ** (digits))

Instead of calling .throw(), you use .close() in line 6. The advantage of using .close() is that it raises StopIteration, an exception used to signal the end of a finite iterator:

>>>

11
111
1111
10101
Traceback (most recent call last):
  File "advanced_gen.py", line 46, in <module>
    main()
  File "advanced_gen.py", line 42, in main
    pal_gen.send(10 ** (digits))
StopIteration

Now that you’ve learned more about the special methods that come with generators, let’s talk about using generators to build data pipelines.

Creating Data Pipelines With Generators

Data pipelines allow you to string together code to process large datasets or streams of data without maxing out your machine’s memory. Imagine that you have a large CSV file:

permalink,company,numEmps,category,city,state,fundedDate,raisedAmt,raisedCurrency,round
digg,Digg,60,web,San Francisco,CA,1-Dec-06,8500000,USD,b
digg,Digg,60,web,San Francisco,CA,1-Oct-05,2800000,USD,a
facebook,Facebook,450,web,Palo Alto,CA,1-Sep-04,500000,USD,angel
facebook,Facebook,450,web,Palo Alto,CA,1-May-05,12700000,USD,a
photobucket,Photobucket,60,web,Palo Alto,CA,1-Mar-05,3000000,USD,a

This example is pulled from the TechCrunch Continental USA set, which describes funding rounds and dollar amounts for various startups based in the USA. Click the link below to download the dataset:

Download Dataset: Click here to download the dataset you'll use in this tutorial to learn about generators and yield in Python.

It’s time to do some processing in Python! To demonstrate how to build pipelines with generators, you’re going to analyze this file to get the total and average of all series A rounds in the dataset.

Let’s think of a strategy:

Read every line of the file.
Split each line into a list of values.
Extract the column names.
Use the column names and lists to create a dictionary.
Filter out the rounds you aren’t interested in.
Calculate the total and average values for the rounds you are interested in.

Normally, you can do this with a package like pandas, but you can also achieve this functionality with just a few generators. You’ll start by reading each line from the file with a generator expression:

 1 file_name = "techcrunch.csv"
 2 lines = (line for line in open(file_name))

Then, you’ll use another generator expression in concert with the previous one to split each line into a list:

 3 list_line = (s.rstrip().split(",") for s in lines)

Here, you created the generator list_line, which iterates through the first generator lines. This is a common pattern to use when designing generator pipelines. Next, you’ll pull the column names out of techcrunch.csv. Since the column names tend to make up the first line in a CSV file, you can grab that with a short next() call:

 4 cols = next(list_line)

This call to next() advances the iterator over the list_line generator one time. Put it all together, and your code should look something like this:

 1 file_name = "techcrunch.csv"
 2 lines = (line for line in open(file_name))
 3 list_line = (s.rstrip().split(",") for s in lines)
 4 cols = next(list_line)

To sum this up, you first create a generator expression lines to yield each line in a file. Next, you iterate through that generator within the definition of another generator expression called list_line, which turns each line into a list of values. Then, you advance the iteration of list_line just once with next() to get a list of the column names from your CSV file.

Note: Watch out for trailing newlines! This code takes advantage of .rstrip() in the list_line generator expression to make sure there are no trailing newline characters, which can be present in CSV files.

To help you filter and perform operations on the data, you’ll create dictionaries where the keys are the column names from the CSV:

 5 company_dicts = (dict(zip(cols, data)) for data in list_line)

This generator expression iterates through the lists produced by list_line. Then, it uses zip() and dict() to create the dictionary as specified above. Now, you’ll use a fourth generator to filter the funding round you want and pull raisedAmt as well:

 6 funding = (
 7     int(company_dict["raisedAmt"])
 8     for company_dict in company_dicts
 9     if company_dict["round"] == "a"
10 )

In this code snippet, your generator expression iterates through the results of company_dicts and takes the raisedAmt for any company_dict where the round key is A.

Remember, you aren’t iterating through all these at once in the generator expression. In fact, you aren’t iterating through anything until you actually use a for loop or a function that works on iterables, like sum(). In fact, call sum() now to iterate through the generators:

11 total_series_a = sum(funding)

Putting this all together, you’ll produce the following script:

 1 file_name = "techcrunch.csv"
 2 lines = (line for line in open(file_name))
 3 list_line = (s.rstrip()split(",") for s in lines)
 4 cols = next(list_line)
 5 company_dicts = (dict(zip(cols, data)) for data in list_line)
 6 funding = (
 7     int(company_dict["raisedAmt"])
 8     for company_dict in company_dicts
 9     if company_dict["round"] == "A"
10 )
11 total_series_a = sum(funding)
12 print(f"Total series A fundraising: ${total_series_a}")

This script pulls together every generator you’ve built, and they all function as one big data pipeline. Here’s a line by line breakdown:

Line 2 reads in each line of the file.
Line 3 splits each line into values and puts the values into a list.
Line 4 uses next() to store the column names in a list.
Line 5 creates dictionaries and unites them with a zip() call:
- The keys are the column names cols from line 4.
- The values are the rows in list form, created in line 3.
Line 6 gets each company’s series A funding amounts. It also filters out any other raised amount.
Line 11 begins the iteration process by calling sum() to get the total amount of series A funding found in the CSV.

When you run this code on techcrunch.csv, you should find a total of $4,376,015,000 raised in series A funding rounds.

Note: The methods for handling CSV files developed in this tutorial are important for understanding how to use generators and the Python yield statement. However, when you work with CSV files in Python, you should instead use the csv module included in Python’s standard library. This module has optimized methods for handling CSV files efficiently.

To dig even deeper, try figuring out the average amount raised per company in a series A round. This is a bit trickier, so here are some hints:

Generators exhaust themselves after being iterated over fully.
You will still need the sum() function.

Good luck!

Conclusion

In this tutorial, you’ve learned about generator functions and generator expressions.

You now know:

How to use and write generator functions and generator expressions
How the all-important Python yield statement enables generators
How to use multiple Python yield statements in a generator function
How to use .send() to send data to a generator
How to use .throw() to raise generator exceptions
How to use .close() to stop a generator’s iteration
How to build a generator pipeline to efficiently process large CSV files

You can get the dataset you used in this tutorial at the link below:

Download Dataset: Click here to download the dataset you'll use in this tutorial to learn about generators and yield in Python.

How have generators helped you in your work or projects? If you’re just learning about them, then how do you plan to use them in the future? Did you find a good solution to the data pipeline problem? Let us know in the comments below!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Thonny: The Beginner-Friendly Python Editor

2019-09-24T14:00:00+00:00

Are you a Python beginner looking for a tool that can support your learning? This course is for you! Every programmer needs a place to write their code. This course will cover an awesome tool called Thonny that will enable you to start working with Python in a beginner-friendly environment.

In this course, you’ll learn:

How to install Thonny on your computer
How to navigate Thonny’s user interface to use its built-in features
How to use Thonny to write and run your code
How to use Thonny to debug your code

By the end of this course, you’ll be comfortable with the development workflow in Thonny and ready to use it for your Python learning. So what is Thonny? Great question!

Thonny is a free Python Integrated Development Environment (IDE) that was especially designed with the beginner Pythonista in mind. Specifically, it has a built-in debugger that can help when you run into nasty bugs, and it allows you to do step-through expression evaluation, among other really awesome features.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Getting Started With Async Features in Python

2019-09-23T14:00:00+00:00

Have you heard of asynchronous programming in Python? Are you curious to know more about Python async features and how you can use them in your work? Perhaps you’ve even tried to write threaded programs and run into some issues. If you’re looking to understand how to use Python async features, then you’ve come to the right place.

In this article, you’ll learn:

What a synchronous program is
What an asynchronous program is
Why you might want to write an asynchronous program
How to use Python async features

All of the example code in this article have been tested with Python 3.7.2. You can grab a copy to follow along by clicking the link below:

Download Code: Click here to download the code you'll use to learn about async features in Python in this tutorial.

Understanding Asynchronous Programming

A synchronous program is executed one step at a time. Even with conditional branching, loops and function calls, you can still think about the code in terms of taking one execution step at a time. When each step is complete, the program moves on to the next one.

Here are two examples of programs that work this way:

Batch processing programs are often created as synchronous programs. You get some input, process it, and create some output. Steps follow one after the other until the program reaches the desired output. The program only needs to pay attention to the steps and their order.
Command-line programs are small, quick processes that run in a terminal. These scripts are used to create something, transform one thing into something else, generate a report, or perhaps list out some data. This can be expressed as a series of program steps that are executed sequentially until the program is done.

An asynchronous program behaves differently. It still takes one execution step at a time. The difference is that the system may not wait for an execution step to be completed before moving on to the next one.

This means that the program will move on to future execution steps even though a previous step hasn’t yet finished and is still running elsewhere. This also means that the program knows what to do when a previous step does finish running.

Why would you want to write a program in this manner? The rest of this article will help you answer that question and give you the tools you need to elegantly solve interesting asynchronous problems.

Building a Synchronous Web Server

A web server’s basic unit of work is, more or less, the same as batch processing. The server will get some input, process it, and create the output. Written as a synchronous program, this would create a working web server.

It would also be an absolutely terrible web server.

Why? In this case, one unit of work (input, process, output) is not the only purpose. The real purpose is to handle hundreds or even thousands of units of work as quickly as possible. This can happen over long periods of time, and several work units may even arrive all at once.

Can a synchronous web server be made better? Sure, you could optimize the execution steps so that all the work coming in is handled as quickly as possible. Unfortunately, there are limitations to this approach. The result could be a web server that doesn’t respond fast enough, can’t handle enough work, or even one that times out when work gets stacked up.

Note: There are other limitations you might see if you tried to optimize the above approach. These include network speed, file IO speed, database query speed, and the speed of other connected services, to name a few. What these all have in common is that they are all IO functions. All of these items are orders of magnitude slower than the CPU’s processing speed.

In a synchronous program, if an execution step starts a database query, then the CPU is essentially idle until the database query is returned. For batch-oriented programs, this isn’t a priority most of the time. Processing the results of that IO operation is the goal. Often, this can take longer than the IO operation itself. Any optimization efforts would be focused on the processing work, not the IO.

Asynchronous programming techniques allow your programs to take advantage of relatively slow IO processes by freeing the CPU to do other work.

Thinking Differently About Programming

When you start trying to understand asynchronous programming, you might see a lot of discussion about the importance of blocking, or writing non-blocking code. (Personally, I struggled to get a good grasp of these concepts from the people I asked and the documentation I read.)

What is non-blocking code? What’s blocking code, for that matter? Would the answers to these questions help you write a better web server? If so, how could you do it? Let’s find out!

Writing asynchronous programs requires that you think differently about programming. While this new way of thinking can be hard to wrap your head around, it’s also an interesting exercise. That’s because the real world is almost entirely asynchronous, and so is how you interact with it.

Imagine this: you’re a parent trying to do several things at once. You have to balance the checkbook, do the laundry, and keep an eye on the kids. Somehow, you’re able to do all of these things at the same time without even thinking about it! Let’s break it down:

Balancing the checkbook is a synchronous task. One step follows another until it’s done. You’re doing all the work yourself.
However, you can break away from the checkbook to do laundry. You unload the dryer, move clothes from the washer to the dryer, and start another load in the washer.
Working with the washer and dryer is a synchronous task, but the bulk of the work happens after the washer and dryer are started. Once you’ve got them going, you can walk away and get back to the checkbook task. At this point, the washer and dryer tasks have become asynchronous. The washer and dryer will run independently until the buzzer goes off (notifying you that the task needs attention).
Watching your kids is another asynchronous task. Once they are set up and playing, they can do so independently for the most part. This changes when someone needs attention, like when someone gets hungry or hurt. When one of your kids yells in alarm, you react. The kids are a long-running task with high priority. Watching them supersedes any other tasks you might be doing, like the checkbook or laundry.

These examples can help to illustrate the concepts of blocking and non-blocking code. Let’s think about this in programming terms. In this example, you’re like the CPU. While you’re moving the laundry around, you (the CPU) are busy and blocked from doing other work, like balancing the checkbook. But that’s okay because the task is relatively quick.

On the other hand, starting the washer and dryer does not block you from performing other tasks. It’s an asynchronous function because you don’t have to wait for it to finish. Once it’s started, you can go back to something else. This is called a context switch: the context of what you’re doing has changed, and the machine’s buzzer will notify you sometime in the future when the laundry task is complete.

As a human, this is how you work all the time. You naturally juggle multiple things at once, often without thinking about it. As a developer, the trick is how to translate this kind of behavior into code that does the same kind of thing.

Programming Parents: Not as Easy as It Looks!

If you recognize yourself (or your parents) in the example above, then that’s great! You’ve got a leg up in understanding asynchronous programming. Again, you’re able to switch contexts between competing tasks fairly easily, picking up some tasks and resuming others. Now you’re going to try and program this behavior into virtual parents!

Thought Experiment #1: The Synchronous Parent

How would you create a parent program to do the above tasks in a completely synchronous manner? Since watching the kids is a high-priority task, perhaps your program would do just that. The parent watches over the kids while waiting for something to happen that might need their attention. However, nothing else (like the checkbook or laundry) would get done in this scenario.

Now, you can re-prioritize the tasks any way you want, but only one of them would happen at any given time. This is the result of a synchronous, step-by-step approach. Like the synchronous web server described above, this would work, but it might not be the best way to live. The parent wouldn’t be able to complete any other tasks until the kids fell asleep. All other tasks would happen afterward, well into the night. (A couple of weeks of this and many real parents might jump out the window!)

Thought Experiment #2: The Polling Parent

If you used polling, then you could change things up so that multiple tasks are completed. In this approach, the parent would periodically break away from the current task and check to see if any other tasks need attention.

Let’s make the polling interval something like fifteen minutes. Now, every fifteen minutes your parent checks to see if the washer, dryer or kids need any attention. If not, then the parent can go back to work on the checkbook. However, if any of those tasks do need attention, then the parent will take care of it before going back to the checkbook. This cycle continues on until the next timeout out of the polling loop.

This approach works as well since multiple tasks are getting attention. However, there are a couple of problems:

The parent may spend a lot of time checking on things that don’t need attention: The washer and dryer haven’t yet finished, and the kids don’t need any attention unless something unexpected happens.
The parent may miss completed tasks that do need attention: For instance, if the washer finished its cycle at the beginning of the polling interval, then it wouldn’t get any attention for up to fifteen minutes! What’s more, watching the kids is supposedly the highest priority task. They couldn’t tolerate fifteen minutes with no attention when something might be going drastically wrong.

You could address these issues by shortening the polling interval, but now your parent (the CPU) would be spending more time context switching between tasks. This is when you start to hit a point of diminishing returns. (Once again, a couple of weeks living like this and, well… See the previous comment about windows and jumping.)

Thought Experiment #3: The Threading Parent

“If I could only clone myself…” If you’re a parent, then you’ve probably had similar thoughts! Since you’re programming virtual parents, you can essentially do this by using threading. This is a mechanism that allows multiple sections of one program to run at the same time. Each section of code that runs independently is known as a thread, and all threads share the same memory space.

If you think of each task as a part of one program, then you can separate them and run them as threads. In other words, you can “clone” the parent, creating one instance for each task: watching the kids, monitoring the washer, monitoring the dryer, and balancing the checkbook. All of these “clones” are running independently.

This sounds like a pretty nice solution, but there are some issues here as well. One is that you’ll have to explicitly tell each parent instance what to do in your program. This can lead to some problems since all instances share everything in the program space.

For example, say that Parent A is monitoring the dryer. Parent A sees that the clothes are dry, so they take control of the dryer and begin unloading the clothes. At the same time, Parent B sees that the washer is done, so they take control of the washer and begin removing clothes. However, Parent B also needs to take control of the dryer so they can put the wet clothes inside. This can’t happen, because Parent A currently has control of the dryer.

After a short while, Parent A has finished unloading clothes. Now they want to take control of the washer and start moving clothes into the empty dryer. This can’t happen, either, because Parent B currently has control of the washer!

These two parents are now deadlocked. Both have control of their own resource and want control of the other resource. They’ll wait forever for the other parent instance to release control. As the programmer, you’d have to write code to work this situation out.

Note: Threaded programs allow you to create multiple, parallel paths of execution that all share the same memory space. This is both an advantage and a disadvantage. Any memory shared between threads is subject to one or more threads trying to use the same shared memory at the same time. This can lead to data corruption, data read in an invalid state, and data that’s just messy in general.

In threaded programming, the context switch happens under system control, not the programmer. The system controls when to switch contexts and when to give threads access to shared data, thereby changing the context of how the memory is being used. All of these kinds of problems are manageable in threaded code, but it’s difficult to get right, and hard to debug when it’s wrong.

Here’s another issue that might arise from threading. Suppose that a child gets hurt and needs to be taken to urgent care. Parent C has been assigned the task of watching over the kids, so they take the child right away. At the urgent care, Parent C needs to write a fairly large check to cover the cost of seeing the doctor.

Meanwhile, Parent D is at home working on the checkbook. They’re unaware of this large check being written, so they’re very surprised when the family checking account is suddenly overdrawn!

Remember, these two parent instances are working within the same program. The family checking account is a shared resource, so you’d have to work out a way for the child-watching parent to inform the checkbook-balancing parent. Otherwise, you’d need to provide some kind of locking mechanism so that the checkbook resource can only be used by one parent at a time, with updates.

Using Python Async Features in Practice

Now you’re going to take some of the approaches outlined in the thought experiments above and turn them into functioning Python programs.

All of the examples in this article have been tested with Python 3.7.2. The requirements.txt file indicates which modules you’ll need to install to run all the examples. If you haven’t yet downloaded the file, you can do so now:

Download Code: Click here to download the code you'll use to learn about async features in Python in this tutorial.

You also might want to set up a Python virtual environment to run the code so you don’t interfere with your system Python.

Synchronous Programming

This first example shows a somewhat contrived way of having a task retrieve work from a queue and process that work. A queue in Python is a nice FIFO (first in first out) data structure. It provides methods to put things in a queue and take them out again in the order they were inserted.

In this case, the work is to get a number from the queue and have a loop count up to that number. It prints to the console when the loop begins, and again to output the total. This program demonstrates one way for multiple synchronous tasks to process the work in a queue.

The program named example_1.py in the repository is listed in full below:

 1 import queue
 2 
 3 def task(name, work_queue):
 4     if work_queue.empty():
 5         print(f"Task {name} nothing to do")
 6     else:
 7         while not work_queue.empty():
 8             count = work_queue.get()
 9             total = 0
10             print(f"Task {name} running")
11             for x in range(count):
12                 total += 1
13             print(f"Task {name} total: {total}")
14 
15 def main():
16     """
17     This is the main entry point for the program.
18     """
19     # Create the queue of 'work'
20     work_queue = queue.Queue()
21 
22     # Put some 'work' in the queue
23     for work in [15, 10, 5, 2]:
24         work_queue.put(work)
25 
26     # Create some synchronous tasks
27     tasks = [
28         (task, "One", work_queue),
29         (task, "Two", work_queue)
30     ]
31 
32     # Run the tasks
33     for t, n, q in tasks:
34         t(n, q)
35 
36 if __name__ == "__main__":
37     main()

Let’s take a look at what each line does:

Line 1 imports the queue module. This is where the program stores work to be done by the tasks.
Lines 3 to 13 define task(). This function pulls work out of work_queue and processes the work until there isn’t any more to do.
Line 15 defines main() to run the program tasks.
Line 20 creates the work_queue. All tasks use this shared resource to retrieve work.
Lines 23 to 24 put work in work_queue. In this case, it’s just a random count of values for the tasks to process.
Lines 27 to 29 create a list of task tuples, with the parameter values those tasks will be passed.
Lines 33 to 34 iterate over the list of task tuples, calling each one and passing the previously defined parameter values.
Line 36 calls main() to run the program.

The task in this program is just a function accepting a string and a queue as parameters. When executed, it looks for anything in the queue to process. If there is work to do, then it pulls values off the queue, starts a for loop to count up to that value, and outputs the total at the end. It continues getting work off the queue until there is nothing left and it exits.

When this program is run, it produces the output you see below:

Task One running
Task One total: 15
Task One running
Task One total: 10
Task One running
Task One total: 5
Task One running
Task One total: 2
Task Two nothing to do

This shows that Task One does all the work. The while loop that Task One hits within task() consumes all the work on the queue and processes it. When that loop exits, Task Two gets a chance to run. However, it finds that the queue is empty, so Task Two prints a statement that says it has nothing to do and then exits. There’s nothing in the code to allow both Task One and Task Two to switch contexts and work together.

Simple Cooperative Concurrency

The next version of the program allows the two tasks to work together. Adding a yield statement means the loop will yield control at the specified point while still maintaining its context. This way, the yielding task can be restarted later.

The yield statement turns task() into a generator. A generator function is called just like any other function in Python, but when the yield statement is executed, control is returned to the caller of the function. This is essentially a context switch, as control moves from the generator function to the caller.

The interesting part is that control can be given back to the generator function by calling next() on the generator. This is a context switch back to the generator function, which picks up execution with all function variables that were defined before the yield still intact.

The while loop in main() takes advantage of this when it calls next(t). This statement restarts the task at the point where it previously yielded. All of this means that you’re in control when the context switch happens: when the yield statement is executed in task().

This is a form of cooperative multitasking. The program is yielding control of its current context so that something else can run. In this case, it allows the while loop in main() to run two instances of task() as a generator function. Each instance consumes work from the same queue. This is sort of clever, but it’s also a lot of work to get the same results as the first program. The program example_2.py demonstrates this simple concurrency and is listed below:

 1 import queue
 2 
 3 def task(name, queue):
 4     while not queue.empty():
 5         count = queue.get()
 6         total = 0
 7         print(f"Task {name} running")
 8         for x in range(count):
 9             total += 1
10             yield
11         print(f"Task {name} total: {total}")
12 
13 def main():
14     """
15     This is the main entry point for the program.
16     """
17     # Create the queue of 'work'
18     work_queue = queue.Queue()
19 
20     # Put some 'work' in the queue
21     for work in [15, 10, 5, 2]:
22         work_queue.put(work)
23 
24     # Create some tasks
25     tasks = [
26         task("One", work_queue),
27         task("Two", work_queue)
28     ]
29 
30     # Run the tasks
31     done = False
32     while not done:
33         for t in tasks:
34             try:
35                 next(t)
36             except StopIteration:
37                 tasks.remove(t)
38             if len(tasks) == 0:
39                 done = True
40 
41 if __name__ == "__main__":
42     main()

Here’s what’s happening in the code above:

Lines 3 to 11 define task() as before, but the addition of yield on Line 10 turns the function into a generator. This where the context switch is made and control is handed back to the while loop in main().
Lines 25 to 28 create the task list, but in a slightly different manner than you saw in the previous example code. In this case, each task is called with its parameters as its entered in the tasks list variable. This is necessary to get the task() generator function running the first time.
Lines 34 to 39 are the modifications to the while loop in main() that allow task() to run cooperatively. This is where control returns to each instance of task() when it yields, allowing the loop to continue and run another task.
Line 35 gives control back to task(), and continues its execution after the point where yield was called.
Line 39 sets the done variable. The while loop ends when all tasks have been completed and removed from tasks.

This is the output produced when you run this program:

Task One running
Task Two running
Task Two total: 10
Task Two running
Task One total: 15
Task One running
Task Two total: 5
Task One total: 2

You can see that both Task One and Task Two are running and consuming work from the queue. This is what’s intended, as both tasks are processing work, and each is responsible for two items in the queue. This is interesting, but again, it takes quite a bit of work to achieve these results.

The trick here is using the yield statement, which turns task() into a generator and performs a context switch. The program uses this context switch to give control to the while loop in main(), allowing two instances of a task to run cooperatively.

Notice how Task Two outputs its total first. This might lead you to think that the tasks are running asynchronously. However, this is still a synchronous program. It’s structured so the two tasks can trade contexts back and forth. The reason why Task Two outputs its total first is that it’s only counting to 10, while Task One is counting to 15. Task Two simply arrives at its total first, so it gets to print its output to the console before Task One.

Cooperative Concurrency With Blocking Calls

The next version of the program is the same as the last, except for the addition of time.sleep(delay) in the body of your task loop. This adds a delay based on the value retrieved from the work queue to every iteration of the task loop. The delay is added to simulate the effect of a blocking call occurring in your task.

A blocking call is code that stops the CPU from doing anything else for some time. In the thought experiments above, if a parent wasn’t able to break away from balancing the checkbook until it was complete, then that would be a blocking call.

time.sleep(delay) does the same thing in this example, because the CPU can’t do anything else but wait for the delay to expire.

elapsed_time provides a way to get the elapsed time from when an instance of the class is created until it’s called as a function. The program example_3.py is listed below:

 1 import time
 2 import queue
 3 from lib.elapsed_time import ET
 4 
 5 def task(name, queue):
 6     while not queue.empty():
 7         delay = queue.get()
 8         et = ET()
 9         print(f"Task {name} running")
10         time.sleep(delay)
11         print(f"Task {name} total elapsed time: {et():.1f}")
12         yield
13 
14 def main():
15     """
16     This is the main entry point for the program.
17     """
18     # Create the queue of 'work'
19     work_queue = queue.Queue()
20 
21     # Put some 'work' in the queue
22     for work in [15, 10, 5, 2]:
23         work_queue.put(work)
24 
25     tasks = [
26         task("One", work_queue),
27         task("Two", work_queue)
28     ]
29 
30     # Run the tasks
31     et = ET()
32     done = False
33     while not done:
34         for t in tasks:
35             try:
36                 next(t)
37             except StopIteration:
38                 tasks.remove(t)
39             if len(tasks) == 0:
40                 done = True
41 
42     print(f"\nTotal elapsed time: {et():.1f}")
43 
44 if __name__ == "__main__":
45     main()

Here’s what’s different in the code above:

Line 1 imports the time module to give the program access to time.sleep().
Line 11 changes task() to include a time.sleep(delay) to mimic an IO delay. This replaces the for loop that did the counting in example_1.py.

When you run this program, you’ll see the following output:

Task One running
Task One total elapsed time: 15.0
Task Two running
Task Two total elapsed time: 10.0
Task One running
Task One total elapsed time: 5.0
Task Two running
Task Two total elapsed time: 2.0

Total elapsed time: 32.01021909713745

As before, both Task One and Task Two are running, consuming work from the queue and processing it. However, even with the addition of the delay, you can see that cooperative concurrency hasn’t gotten you anything. The delay stops the processing of the entire program, and the CPU just waits for the IO delay to be over.

This is exactly what’s meant by blocking code in Python async documentation. You’ll notice that the time it takes to run the entire program is just the cumulative time of all the delays. Running tasks this way is not a win.

Cooperative Concurrency With Non-Blocking Calls

The next version of the program has been modified quite a bit. It makes use of Python async features using asyncio/await provided in Python 3.

The time and queue modules have been replaced with the asyncio package. This gives your program access to asynchronous friendly (non-blocking) sleep and queue functionality. The change to task() defines it as asynchronous with the addition of the async prefix on line 4. This indicates to Python that the function will be asynchronous.

The other big change is removing the time.sleep(delay) and yield statements, and replacing them with await asyncio.sleep(delay). This creates a non-blocking delay that will perform a context switch back to the caller main().

The while loop inside main() no longer exists. Instead of task_array, there’s a call to await asyncio.gather(...). This tells asyncio two things:

Create two tasks based on task() and start running them.
Wait for both of these to be completed before moving forward.

The last line of the program asyncio.run(main()) runs main(). This creates what’s known as an event loop). It’s this loop that will run main(), which in turn will run the two instances of task().

The event loop is at the heart of the Python async system. It runs all the code, including main(). When task code is executing, the CPU is busy doing work. When the await keyword is reached, a context switch occurs, and control passes back to the event loop. The event loop looks at all the tasks waiting for an event (in this case, an asyncio.sleep(delay) timeout) and passes control to a task with an event that’s ready.

await asyncio.sleep(delay) is non-blocking in regards to the CPU. Instead of waiting for the delay to timeout, the CPU registers a sleep event on the event loop task queue and performs a context switch by passing control to the event loop. The event loop continuously looks for completed events and passes control back to the task waiting for that event. In this way, the CPU can stay busy if work is available, while the event loop monitors the events that will happen in the future.

Note: An asynchronous program runs in a single thread of execution. The context switch from one section of code to another that would affect data is completely in your control. This means you can atomize and complete all shared memory data access before making a context switch. This simplifies the shared memory problem inherent in threaded code.

The example_4.py code is listed below:

 1 import asyncio
 2 from lib.elapsed_time import ET
 3 
 4 async def task(name, work_queue):
 5     while not work_queue.empty():
 6         delay = await work_queue.get()
 7         et = ET()
 8         print(f"Task {name} running")
 9         await asyncio.sleep(delay)
10         print(f"Task {name} total elapsed time: {et():.1f}")
11 
12 async def main():
13     """
14     This is the main entry point for the program.
15     """
16     # Create the queue of 'work'
17     work_queue = asyncio.Queue()
18 
19     # Put some 'work' in the queue
20     for work in [15, 10, 5, 2]:
21         await work_queue.put(work)
22 
23     # Run the tasks
24     et = ET()
25     await asyncio.gather(
26         asyncio.create_task(task("One", work_queue)),
27         asyncio.create_task(task("Two", work_queue)),
28     )
29     print(f"\nTotal elapsed time: {et():.1f}")
30 
31 if __name__ == "__main__":
32     asyncio.run(main())

Here’s what’s different between this program and example_3.py:

Line 1 imports asyncio to gain access to Python async functionality. This replaces the time import.
Line 4 shows the addition of the async keyword in front of the task() definition. This informs the program that task can run asynchronously.
Line 9 replaces time.sleep(delay) with the non-blocking asyncio.sleep(delay), which also yields control (or switches contexts) back to the main event loop.
Line 17 creates the non-blocking asynchronous work_queue.
Lines 20 to 21 put work into work_queue in an asynchronous manner using the await keyword.
Lines 25 to 28 create the two tasks and gather them together, so the program will wait for both tasks to complete.
Line 32 starts the program running asynchronously. It also starts the internal event loop.

When you look at the output of this program, notice how both Task One and Task Two start at the same time, then wait at the mock IO call:

Task One running
Task Two running
Task Two total elapsed time: 10.0
Task Two running
Task One total elapsed time: 15.0
Task One running
Task Two total elapsed time: 5.0
Task One total elapsed time: 2.0

Total elapsed time: 17.0

This indicates that await asyncio.sleep(delay) is non-blocking, and that other work is being done.

At the end of the program, you’ll notice the total elapsed time is essentially half the time it took for example_3.py to run. That’s the advantage of a program that uses Python async features! Each task was able to run await asyncio.sleep(delay) at the same time. The total execution time of the program is now less than the sum of its parts. You’ve broken away from the synchronous model!

Synchronous (Blocking) HTTP Calls

The next version of the program is kind of a step forward as well as a step back. The program is doing some actual work with real IO by making HTTP requests to a list of URLs and getting the page contents. However, it’s doing so in a blocking (synchronous) manner.

The program has been modified to import the wonderful requests module to make the actual HTTP requests. Also, the queue now contains a list of URLs, rather than numbers. In addition, task() no longer increments a counter. Instead, requests gets the contents of a URL retrieved from the queue, and prints how long it took to do so.

The example_5.py code is listed below:

 1 import queue
 2 import requests
 3 from lib.elapsed_time import ET
 4 
 5 def task(name, work_queue):
 6     with requests.Session() as session:
 7         while not work_queue.empty():
 8             url = work_queue.get()
 9             print(f"Task {name} getting URL: {url}")
10             et = ET()
11             session.get(url)
12             print(f"Task {name} total elapsed time: {et():.1f}")
13             yield
14 
15 def main():
16     """
17     This is the main entry point for the program.
18     """
19     # Create the queue of 'work'
20     work_queue = queue.Queue()
21 
22     # Put some 'work' in the queue
23     for url in [
24         "http://google.com",
25         "http://yahoo.com",
26         "http://linkedin.com",
27         "http://apple.com",
28         "http://microsoft.com",
29         "http://facebook.com",
30         "http://twitter.com"
31     ]:
32         work_queue.put(url)
33 
34     tasks = [
35         task("One", work_queue),
36         task("Two", work_queue)
37     ]
38 
39     # Run the tasks
40     et = ET()
41     done = False
42     while not done:
43         for t in tasks:
44             try:
45                 next(t)
46             except StopIteration:
47                 tasks.remove(t)
48             if len(tasks) == 0:
49                 done = True
50 
51     print(f"\nTotal elapsed time: {et():.1f}")
52 
53 if __name__ == "__main__":
54     main()

Here’s what’s happening in this program:

Line 2 imports requests, which provides a convenient way to make HTTP calls.
Line 11 introduces a delay, similar to example_3.py. However, this time it calls session.get(url), which returns the contents of the URL retrieved from work_queue.
Lines 23 to 32 put the list of URLs into work_queue.

When you run this program, you’ll see the following output:

Task One getting URL: http://google.com
Task One total elapsed time: 0.3
Task Two getting URL: http://yahoo.com
Task Two total elapsed time: 0.8
Task One getting URL: http://linkedin.com
Task One total elapsed time: 0.4
Task Two getting URL: http://apple.com
Task Two total elapsed time: 0.3
Task One getting URL: http://microsoft.com
Task One total elapsed time: 0.5
Task Two getting URL: http://facebook.com
Task Two total elapsed time: 0.5
Task One getting URL: http://twitter.com
Task One total elapsed time: 0.4

Total elapsed time: 3.2

Just like in earlier versions of the program, yield turns task() into a generator. It also performs a context switch that lets the other task instance run.

Each task gets a URL from the work queue, retrieves the contents of the page, and reports how long it took to get that content.

As before, yield allows both your tasks to run cooperatively. However, since this program is running synchronously, each session.get() call blocks the CPU until the page is retrieved. Note the total time it took to run the entire program at the end. This will be meaningful for the next example.

Asynchronous (Non-Blocking) HTTP Calls

This version of the program modifies the previous one to use Python async features. It also imports the aiohttp module, which is a library to make HTTP requests in an asynchronous fashion using asyncio.

The tasks here have been modified to remove the yield call since the code to make the HTTP GET call is no longer blocking. It also performs a context switch back to the event loop.

The example_6.py program is listed below:

 1 import asyncio
 2 import aiohttp
 3 from lib.elapsed_time import ET
 4 
 5 async def task(name, work_queue):
 6     async with aiohttp.ClientSession() as session:
 7         while not work_queue.empty():
 8             url = await work_queue.get()
 9             print(f"Task {name} getting URL: {url}")
10             et = ET()
11             async with session.get(url) as response:
12                 await response.text()
13             print(f"Task {name} total elapsed time: {et():.1f}")
14 
15 async def main():
16     """
17     This is the main entry point for the program.
18     """
19     # Create the queue of 'work'
20     work_queue = asyncio.Queue()
21 
22     # Put some 'work' in the queue
23     for url in [
24         "http://google.com",
25         "http://yahoo.com",
26         "http://linkedin.com",
27         "http://apple.com",
28         "http://microsoft.com",
29         "http://facebook.com",
30         "http://twitter.com",
31     ]:
32         await work_queue.put(url)
33 
34     # Run the tasks
35     et = ET()
36     await asyncio.gather(
37         asyncio.create_task(task("One", work_queue)),
38         asyncio.create_task(task("Two", work_queue)),
39     )
40     print(f"\nTotal elapsed time: {et():.1f}")
41 
42 if __name__ == "__main__":
43     asyncio.run(main())

Here’s what’s happening in this program:

Line 2 imports the aiohttp library, which provides an asynchronous way to make HTTP calls.
Line 5 marks task() as an asynchronous function.
Line 6 creates an aiohttp session context manager.
Line 11 creates an aiohttp response context manager. It also makes an HTTP GET call to the URL taken from work_queue.
Line 12 uses the response to get the text retrieved from the URL asynchronously.

When you run this program, you’ll see the following output:

Task One getting URL: http://google.com
Task Two getting URL: http://yahoo.com
Task One total elapsed time: 0.3
Task One getting URL: http://linkedin.com
Task One total elapsed time: 0.3
Task One getting URL: http://apple.com
Task One total elapsed time: 0.3
Task One getting URL: http://microsoft.com
Task Two total elapsed time: 0.9
Task Two getting URL: http://facebook.com
Task Two total elapsed time: 0.4
Task Two getting URL: http://twitter.com
Task One total elapsed time: 0.5
Task Two total elapsed time: 0.3

Total elapsed time: 1.7

Take a look at the total elapsed time, as well as the individual times to get the contents of each URL. You’ll see that the duration is about half the cumulative time of all the HTTP GET calls. This is because the HTTP GET calls are running asynchronously. In other words, you’re effectively taking better advantage of the CPU by allowing it to make multiple requests at once.

Because the CPU is so fast, this example could likely create as many tasks as there are URLs. In this case, the program’s run time would be that of the single slowest URL retrieval.

Conclusion

This article has given you the tools you need to start making asynchronous programming techniques a part of your repertoire. Using Python async features gives you programmatic control of when context switches take place. This means that many of the tougher issues you might see in threaded programming are easier to deal with.

Asynchronous programming is a powerful tool, but it isn’t useful for every kind of program. If you’re writing a program that calculates pi to the millionth decimal place, for instance, then asynchronous code won’t help you. That kind of program is CPU bound, without much IO. However, if you’re trying to implement a server or a program that performs IO (like file or network access), then using Python async features could make a huge difference.

To sum it up, you’ve learned:

What synchronous programs are
How asynchronous programs are different, but also powerful and manageable
Why you might want to write asynchronous programs
How to use the built-in async features in Python

You can get the code for all of the example programs used in this tutorial:

Download Code: Click here to download the code you'll use to learn about async features in Python in this tutorial.

Now that you’re equipped with these powerful skills, you can take your programs to the next level!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Convert a Python String to int

2019-09-18T14:00:00+00:00

Integers are whole numbers. In other words, they have no fractional component. Two data types you can use to store an integer in Python are int and str. These types offer flexibility for working with integers in different circumstances. In this tutorial, you’ll learn how you can convert a Python string to an int. You’ll also learn how to convert an int to a string.

By the end of this tutorial, you’ll understand:

How to store integers using str and int
How to convert a Python string to an int
How to convert a Python int to a string

Let’s get started!

Python Pit Stop: This tutorial is a quick and practical way to find the info you need, so you’ll be back to your project in no time!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Representing Integers in Python

An integer can be stored using different types. Two possible Python data types for representing an integer are:

str
int

For example, you can represent an integer using a string literal:

>>>

>>> s = "110"

Here, Python understands you to mean that you want to store the integer 110 as a string. You can do the same with the integer data type:

>>>

>>> i = 110

It’s important to consider what you specifically mean by "110" and 110 in the examples above. As a human who has used the decimal number system for your whole life, it may be obvious that you mean the number one hundred and ten. However, there are several other number systems, such as binary and hexadecimal, which use different bases to represent an integer.

For example, you can represent the number one hundred and ten in binary and hexadecimal as 1101110 and 6e respectively.

You can also represent your integers with other number systems in Python using the str and int data types:

>>>

>>> binary = 0b1010
>>> hexadecimal = "0xa"

Notice that binary and hexadecimal use prefixes to identify the number system. All integer prefixes are in the form 0?, in which you replace ? with a character that refers to the number system:

b: binary (base 2)
o: octal (base 8)
d: decimal (base 10)
x: hexadecimal (base 16)

Technical Detail: The prefix is not required in either an integer or string representation when it can be inferred.

int assumes the literal integer to be decimal:

>>>

>>> decimal = 303
>>> hexadecimal_with_prefix = 0x12F
>>> hexadecimal_no_prefix = 12F
  File "<stdin>", line 1
    hexadecimal_no_prefix = 12F
                              ^
SyntaxError: invalid syntax

The string representation of an integer is more flexible because a string holds arbitrary text data:

>>>

>>> decimal = "303"
>>> hexadecimal_with_prefix = "0x12F"
>>> hexadecimal_no_prefix = "12F"

Each of these strings represent the same integer.

Now that you have some foundational knowledge about how to represent integers using str and int, you’ll learn how to convert a Python string to an int.

Converting a Python String to an `int`

If you have a decimal integer represented as a string and you want to convert the Python string to an int, then you just pass the string to int(), which returns a decimal integer:

>>>

>>> int("10")
10
>>> type(int("10"))
<class 'int'>

By default, int() assumes that the string argument represents a decimal integer. If, however, you pass a hexadecimal string to int(), then you’ll see a ValueError:

>>>

>>> int("0x12F")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '0x12F'

The error message says that the string is not a valid decimal integer.

Note:

It’s important to recognize the difference between two types of failed results of passing a string to int():

Syntax Error: A ValueError will occur when int() doesn’t know how to parse the string using the provided base (10 by default).
Logical Error: int() does know how to parse the string, but not the way you expected.

Here’s an example of a logical error:

>>>

>>> binary = "11010010"
>>> int(binary)  # Using the default base of 10, instead of 2
11010010

In this example, you meant for the result to be 210, which is the decimal representation of the binary string. Unfortunately, because you didn’t specify that behavior, int() assumed that the string was a decimal integer.

One good safeguard for this behavior is to always define your string representations using explicit bases:

>>>

>>> int("0b11010010")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '0b11010010'

Here, you get a ValueError because int() doesn’t know how to parse the binary string as a decimal integer.

When you pass a string to int(), you can specify the number system that you’re using to represent the integer. The way to specify the number system is to use base:

>>>

>>> int("0x12F", base=16)
303

Now, int() understands you are passing a hexadecimal string and expecting a decimal integer.

Technical Detail: The argument that you pass to base is not limited to 2, 8, 10, and 16:

>>>

>>> int("10", base=3)
3

Great! Now that you’re comfortable with the ins and outs of converting a Python string to an int, you’ll learn how to do the inverse operation.

Converting a Python `int` to a String

In Python, you can convert a Python int to a string using str():

>>>

>>> str(10)
'10'
>>> type(str(10))
<class 'str'>

By default, str() behaves like int() in that it results in a decimal representation:

>>>

>>> str(0b11010010)
'210'

In this example, str() is smart enough to interpret the binary literal and convert it to a decimal string.

If you want a string to represent an integer in another number system, then you use a formatted string, such as an f-string (in Python 3.6+), and an option that specifies the base:

>>>

>>> octal = 0o1073
>>> f"{octal}"  # Decimal
'571'
>>> f"{octal:x}"  # Hexadecimal
'23b'
>>> f"{octal:b}"  # Binary
'1000111011'

str is a flexible way to represent an integer in a variety of different number systems.

Conclusion

Congratulations! You’ve learned so much about integers and how to represent and convert them between Python string and int data types.

In this tutorial, you learned:

How to use str and int to store integers
How to specify an explicit number system for an integer representation
How to convert a Python string to an int
How to convert a Python int to a string

Now that you know so much about str and int, you can learn more about representing numerical types using float(), hex(), oct(), and bin()!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Debugging With pdb

2019-09-17T14:00:00+00:00

Nowadays, we often take for granted the excellent debuggers built into our favorite IDEs. But how do you debug your Python code when you don’t have the luxury of using an IDE?

pdb, short for Python DeBugger, is a module for interactive source code debugging. It’s built into the Python Standard Library, so it’s always accessible to you. Because it runs in the command line, it’s especially helpful when you’re developing on remote systems.

In this course, you’ll learn how to perform the most common debugging tasks using pdb, including setting breakpoints, stepping through code, viewing stack traces, creating watch lists, and more.

Free Bonus: Click here to get a printable "pdb Command Reference" (PDF) that you can keep on your desk and refer to while debugging.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyGame: A Primer on Game Programming in Python

2019-09-16T15:27:36+00:00

When I started learning computer programming late in the last millennium, it was driven by my desire to write computer games. I tried to figure out how to write games in every language and on every platform I learned, including Python. That’s how I discovered pygame and learned how to use it to write games and other graphical programs. At the time, I really wanted a primer on pygame.

By the end of this article, you’ll be able to:

Draw items on your screen
Play sound effects and music
Handle user input
Implement event loops
Describe how game programming differs from standard procedural Python programming

This primer assumes you have a basic understanding of writing Python programs, including user-defined functions, imports, loops, and conditionals. You should also be familiar with how to open files on your platform. A basic understanding of object-oriented Python is helpful as well. pygame works with most versions of Python, but Python 3.6 is recommended and used throughout this article.

You can get all of the code in this article to follow along:

Clone Repo: Click here to clone the repo you'll use to learn how to use PyGame in this tutorial.

Background and Setup

pygame is a Python wrapper for the SDL library, which stands for Simple DirectMedia Layer. SDL provides cross-platform access to your system’s underlying multimedia hardware components, such as sound, video, mouse, keyboard, and joystick. pygame started life as a replacement for the stalled PySDL project. The cross-platform nature of both SDL and pygame means you can write games and rich multimedia Python programs for every platform that supports them!

To install pygame on your platform, use the appropriate pip command:

$ pip install pygame

You can verify the install by loading one of the examples that comes with the library:

$ python3 -m pygame.examples.aliens

If a game window appears, then pygame is installed properly! If you run into problems, then the Getting Started guide outlines some known issues and caveats for all platforms.

Basic PyGame Program

Before getting down to specifics, let’s take a look at a basic pygame program. This program creates a window, fills the background with white, and draws a blue circle in the middle of it:

 1 # Simple pygame program
 2 
 3 # Import and initialize the pygame library
 4 import pygame
 5 pygame.init()
 6 
 7 # Set up the drawing window
 8 screen = pygame.display.set_mode([500, 500])
 9 
10 # Run until the user asks to quit
11 running = True
12 while running:
13 
14     # Did the user click the window close button?
15     for event in pygame.event.get():
16         if event.type == pygame.QUIT:
17             running = False
18 
19     # Fill the background with white
20     screen.fill((255, 255, 255))
21 
22     # Draw a solid blue circle in the center
23     pygame.draw.circle(screen, (0, 0, 255), (250, 250), 75)
24 
25     # Flip the display
26     pygame.display.flip()
27 
28 # Done! Time to quit.
29 pygame.quit()

When you run this program, you’ll see a window that looks like this:

Let’s break this code down, section by section:

Lines 4 and 5 import and initialize the pygame library. Without these lines, there is no pygame.
Line 8 sets up your program’s display window. You provide either a list or a tuple that specifies the width and height of the window to create. This program uses a list to create a square window with 500 pixels on each side.
Lines 11 and 12 set up a game loop to control when the program ends. You’ll cover game loops later on in this tutorial.
Lines 15 to 17 scan and handle events within the game loop. You’ll get to events a bit later as well. In this case, the only event handled is pygame.QUIT, which occurs when the user clicks the window close button.
Line 20 fills the window with a solid color. screen.fill() accepts either a list or tuple specifying the RGB values for the color. Since (255, 255, 255) was provided, the window is filled with white.
Line 23 draws a circle in the window, using the following parameters:
- screen: the window on which to draw
- (0, 0, 255): a tuple containing RGB color values
- (250, 250): a tuple specifying the center coordinates of the circle
- 75: the radius of the circle to draw in pixels
Line 26 updates the contents of the display to the screen. Without this call, nothing appears in the window!
Line 29 exits pygame. This only happens once the loop finishes.

That’s the pygame version of “Hello, World.” Now let’s dig a little deeper into the concepts behind this code.

PyGame Concepts

As pygame and the SDL library are portable across different platforms and devices, they both need to define and work with abstractions for various hardware realities. Understanding those concepts and abstractions will help you design and develop your own games.

Initialization and Modules

The pygame library is composed of a number of Python constructs, which include several different modules. These modules provide abstract access to specific hardware on your system, as well as uniform methods to work with that hardware. For example, display allows uniform access to your video display, while joystick allows abstract control of your joystick.

After importing the pygame library in the example above, the first thing you did was initialize PyGame using pygame.init(). This function calls the separate init() functions of all the included pygame modules. Since these modules are abstractions for specific hardware, this initialization step is required so that you can work with the same code on Linux, Windows, and Mac.

Displays and Surfaces

In addition to the modules, pygame also includes several Python classes, which encapsulate non-hardware dependent concepts. One of these is the Surface which, at its most basic, defines a rectangular area on which you can draw. Surface objects are used in many contexts in pygame. Later you’ll see how to load an image into a Surface and display it on the screen.

In pygame, everything is viewed on a single user-created display, which can be a window or a full screen. The display is created using .set_mode(), which returns a Surface representing the visible part of the window. It is this Surface that you pass into drawing functions like pygame.draw.circle(), and the contents of that Surface are pushed to the display when you call pygame.display.flip().

Images and Rects

Your basic pygame program drew a shape directly onto the display’s Surface, but you can also work with images on the disk. The image module allows you to load and save images in a variety of popular formats. Images are loaded into Surface objects, which can then be manipulated and displayed in numerous ways.

As mentioned above, Surface objects are represented by rectangles, as are many other objects in pygame, such as images and windows. Rectangles are so heavily used that there is a special Rect class just to handle them. You’ll be using Rect objects and images in your game to draw players and enemies, and to manage collisions between them.

Okay, that’s enough theory. Let’s design and write a game!

Basic Game Design

Before you start writing any code, it’s always a good idea to have some design in place. Since this is a tutorial game, let’s design some basic gameplay for it as well:

The goal of the game is to avoid incoming obstacles:
- The player starts on the left side of the screen.
- The obstacles enter randomly from the right and move left in a straight line.
The player can move left, right, up, or down to avoid the obstacles.
The player cannot move off the screen.
The game ends either when the player is hit by an obstacle or when the user closes the window.

When he was describing software projects, a former colleague of mine used to say, “You don’t know what you do until you know what you don’t do.” With that in mind, here are some things that won’t be covered in this tutorial:

No multiple lives
No scorekeeping
No player attack capabilities
No advancing levels
No boss characters

You’re free to try your hand at adding these and other features to your own program.

Let’s get started!

Importing and Initializing PyGame

After you import pygame, you’ll also need to initialize it. This allows pygame to connect its abstractions to your specific hardware:

 1 # Import the pygame module
 2 import pygame
 3 
 4 # Import pygame.locals for easier access to key coordinates
 5 # Updated to conform to flake8 and black standards
 6 from pygame.locals import (
 7     K_UP,
 8     K_DOWN,
 9     K_LEFT,
10     K_RIGHT,
11     K_ESCAPE,
12     KEYDOWN,
13     QUIT,
14 )
15 
16 # Initialize pygame
17 pygame.init()

The pygame library defines many things besides modules and classes. It also defines some local constants for things like keystrokes, mouse movements, and display attributes. You reference these constants using the syntax pygame.<CONSTANT>. By importing specific constants from pygame.locals, you can use the syntax <CONSTANT> instead. This will save you some keystrokes and improve overall readability.

Setting Up the Display

Now you need something to draw on! Create a screen to be the overall canvas:

 1 # Import the pygame module
 2 import pygame
 3 
 4 # Import pygame.locals for easier access to key coordinates
 5 # Updated to conform to flake8 and black standards
 6 from pygame.locals import (
 7     K_UP,
 8     K_DOWN,
 9     K_LEFT,
10     K_RIGHT,
11     K_ESCAPE,
12     KEYDOWN,
13     QUIT,
14 )
15 
16 # Initialize pygame
17 pygame.init()
18 
19 # Define constants for the screen width and height
20 SCREEN_WIDTH = 800
21 SCREEN_HEIGHT = 600
22 
23 # Create the screen object
24 # The size is determined by the constant SCREEN_WIDTH and SCREEN_HEIGHT
25 screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))

You create the screen to use by calling pygame.display.set_mode() and passing a tuple or list with the desired width and height. In this case, the window is 800x600, as defined by the constants SCREEN_WIDTH and SCREEN_HEIGHT on lines 20 and 21. This returns a Surface which represents the inside dimensions of the window. This is the portion of the window you can control, while the OS controls the window borders and title bar.

If you run this program now, then you’ll see a window pop up briefly and then immediately disappear as the program exits. Don’t blink or you might miss it! In the next section, you’ll focus on the main game loop to ensure that your program exits only when given the correct input.

Setting Up the Game Loop

Every game from Pong to Fortnite uses a game loop to control gameplay. The game loop does four very important things:

Processes user input
Updates the state of all game objects
Updates the display and audio output
Maintains the speed of the game

Every cycle of the game loop is called a frame, and the quicker you can do things each cycle, the faster your game will run. Frames continue to occur until some condition to exit the game is met. In your design, there are two conditions that can end the game loop:

The player collides with an obstacle. (You’ll cover collision detection later.)
The player closes the window.

The first thing the game loop does is process user input to allow the player to move around the screen. Therefore, you need some way to capture and process a variety of input. You do this using the pygame event system.

Processing Events

Key presses, mouse movements, and even joystick movements are some of the ways in which a user can provide input. All user input results in an event being generated. Events can happen at any time and often (but not always) originate outside the program. All events in pygame are placed in the event queue, which can then be accessed and manipulated. Dealing with events is referred to as handling them, and the code to do so is called an event handler.

Every event in pygame has an event type associated with it. For your game, the event types you’ll focus on are keypresses and window closure. Keypress events have the event type KEYDOWN, and the window closure event has the type QUIT. Different event types may also have other data associated with them. For example, the KEYDOWN event type also has a variable called key to indicate which key was pressed.

You access the list of all active events in the queue by calling pygame.event.get(). You then loop through this list, inspect each event type, and respond accordingly:

27 # Variable to keep the main loop running
28 running = True
29 
30 # Main loop
31 while running:
32     # Look at every event in the queue
33     for event in pygame.event.get():
34         # Did the user hit a key?
35         if event.type == KEYDOWN:
36             # Was it the Escape key? If so, stop the loop.
37             if event.key == K_ESCAPE:
38                 running = False
39 
40         # Did the user click the window close button? If so, stop the loop.
41         elif event.type == QUIT:
42             running = False

Let’s take a closer look at this game loop:

Line 28 sets up a control variable for the game loop. To exit the loop and the game, you set running = False. The game loop starts on line 29.
Line 31 starts the event handler, walking through every event currently in the event queue. If there are no events, then the list is empty, and the handler won’t do anything.
Lines 35 to 38 check if the current event.type is a KEYDOWN event. If it is, then the program checks which key was pressed by looking at the event.key attribute. If the key is the Esc key, indicated by K_ESCAPE, then it exits the game loop by setting running = False.
Lines 41 and 42 do a similar check for the event type called QUIT. This event only occurs when the user clicks the window close button. The user may also use any other operating system action to close the window.

When you add these lines to the previous code and run it, you’ll see a window with a blank or black screen:

The window won’t disappear until you press the Esc key, or otherwise trigger a QUIT event by closing the window.

Drawing on the Screen

In the sample program, you drew on the screen using two commands:

screen.fill() to fill the background
pygame.draw.circle() to draw a circle

Now you’ll learn about a third way to draw to the screen: using a Surface.

Recall that a Surface is a rectangular object on which you can draw, like a blank sheet of paper. The screen object is a Surface, and you can create your own Surface objects separate from the display screen. Let’s see how that works:

44 # Fill the screen with white
45 screen.fill((255, 255, 255))
46 
47 # Create a surface and pass in a tuple containing its length and width
48 surf = pygame.Surface((50, 50))
49 
50 # Give the surface a color to separate it from the background
51 surf.fill((0, 0, 0))
52 rect = surf.get_rect()

After the screen is filled with white on line 45, a new Surface is created on line 48. This Surface is 50 pixels wide, 50 pixels tall, and assigned to surf. At this point, you treat it just like the screen. So on line, 51 you fill it with black. You can also access its underlying Rect using .get_rect(). This is stored as rect for later use.

Using `.blit()` and `.flip()`

Just creating a new Surface isn’t enough to see it on the screen. To do that, you need to blit the Surface onto another Surface. The term blit stands for Block Transfer, and .blit() is how you copy the contents of one Surface to another. You can only .blit() from one Surface to another, but since the screen is just another Surface, that’s not a problem. Here’s how you draw surf on the screen:

54 # This line says "Draw surf onto the screen at the center"
55 screen.blit(surf, (SCREEN_WIDTH/2, SCREEN_HEIGHT/2))
56 pygame.display.flip()

The .blit() call on line 55 takes two arguments:

The Surface to draw
The location at which to draw it on the source Surface

The coordinates (SCREEN_WIDTH/2, SCREEN_HEIGHT/2) tell your program to place surf in the exact center of the screen, but it doesn’t quite look that way:

The reason why the image looks off-center is that .blit() puts the top-left corner of surf at the location given. If you want surf to be centered, then you’ll have to do some math to shift it up and to the left. You can do this by subtracting the width and height of surf from the width and height of the screen, dividing each by 2 to locate the center, and then passing those numbers as arguments to screen.blit():

54 # Put the center of surf at the center of the display
55 surf_center = (
56     (SCREEN_WIDTH-surf.get_width())/2,
57     (SCREEN_HEIGHT-surf.get_height())/2
58 )
59 
60 # Draw surf at the new coordinates
61 screen.blit(surf, surf_center)
62 pygame.display.flip()

Notice the call to pygame.display.flip() after the call to blit(). This updates the entire screen with everything that’s been drawn since the last flip. Without the call to .flip(), nothing is shown.

Sprites

In your game design, the player starts on the left, and obstacles come in from the right. You can represent all the obstacles with Surface objects to make drawing everything easier, but how do you know where to draw them? How do you know if an obstacle has collided with the player? What happens when the obstacle flies off the screen? What if you want to draw background images that also move? What if you want your images to be animated? You can handle all these situations and more with sprites.

In programming terms, a sprite is a 2D representation of something on the screen. Essentially, it’s a picture. pygame provides a Sprite class, which is designed to hold one or several graphical representations of any game object that you want to display on the screen. To use it, you create a new class that extends Sprite. This allows you to use its built-in methods.

Players

Here’s how you use Sprite objects with the current game to define the player. Insert this code after line 18:

20 # Define a Player object by extending pygame.sprite.Sprite
21 # The surface drawn on the screen is now an attribute of 'player'
22 class Player(pygame.sprite.Sprite):
23     def __init__(self):
24         super(Player, self).__init__()
25         self.surf = pygame.Surface((75, 25))
26         self.surf.fill((255, 255, 255))
27         self.rect = self.surf.get_rect()

You first define Player by extending pygame.sprite.Sprite on line 22. Then .__init__() uses .super() to call the .__init__() method of Sprite. For more info on why this is necessary, you can read Supercharge Your Classes With Python super().

Next, you define and initialize .surf to hold the image to display, which is currently a white box. You also define and initialize .rect, which you’ll use to draw the player later. To use this new class, you need to create a new object and change the drawing code as well. Expand the code block below to see it all together:

 1 # Import the pygame module
 2 import pygame
 3 
 4 # Import pygame.locals for easier access to key coordinates
 5 # Updated to conform to flake8 and black standards
 6 from pygame.locals import (
 7     K_UP,
 8     K_DOWN,
 9     K_LEFT,
10     K_RIGHT,
11     K_ESCAPE,
12     KEYDOWN,
13     QUIT,
14 )
15 
16 # Define constants for the screen width and height
17 SCREEN_WIDTH = 800
18 SCREEN_HEIGHT = 600
19 
20 # Define a player object by extending pygame.sprite.Sprite
21 # The surface drawn on the screen is now an attribute of 'player'
22 class Player(pygame.sprite.Sprite):
23     def __init__(self):
24         super(Player, self).__init__()
25         self.surf = pygame.Surface((75, 25))
26         self.surf.fill((255, 255, 255))
27         self.rect = self.surf.get_rect()
28 
29 # Initialize pygame
30 pygame.init()
31 
32 # Create the screen object
33 # The size is determined by the constant SCREEN_WIDTH and SCREEN_HEIGHT
34 screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
35 
36 # Instantiate player. Right now, this is just a rectangle.
37 player = Player()
38 
39 # Variable to keep the main loop running
40 running = True
41 
42 # Main loop
43 while running:
44     # for loop through the event queue
45     for event in pygame.event.get():
46         # Check for KEYDOWN event
47         if event.type == KEYDOWN:
48             # If the Esc key is pressed, then exit the main loop
49             if event.key == K_ESCAPE:
50                 running = False
51         # Check for QUIT event. If QUIT, then set running to false.
52         elif event.type == QUIT:
53             running = False
54 
55     # Fill the screen with black
56     screen.fill((0, 0, 0))
57 
58     # Draw the player on the screen
59     screen.blit(player.surf, (SCREEN_WIDTH/2, SCREEN_HEIGHT/2))
60 
61     # Update the display
62     pygame.display.flip()

Run this code. You’ll see a white rectangle at roughly the middle of the screen:

What do you think would happen if you changed line 59 to screen.blit(player.surf, player.rect)? Try it and see:

55 # Fill the screen with black
56 screen.fill((0, 0, 0))
57 
58 # Draw the player on the screen
59 screen.blit(player.surf, player.rect)
60 
61 # Update the display
62 pygame.display.flip()

When you pass a Rect to .blit(), it uses the coordinates of the top left corner to draw the surface. You’ll use this later to make your player move!

User Input

So far, you’ve learned how to set up pygame and draw objects on the screen. Now, the real fun starts! You’ll make the player controllable using the keyboard.

Earlier, you saw that pygame.event.get() returns a list of the events in the event queue, which you scan for KEYDOWN event types. Well, that’s not the only way to read keypresses. pygame also provides pygame.event.get_pressed(), which returns a dictionary containing all the current KEYDOWN events in the queue.

Put this in your game loop right after the event handling loop. This returns a dictionary containing the keys pressed at the beginning of every frame:

54 # Get the set of keys pressed and check for user input
55 pressed_keys = pygame.key.get_pressed()

Next, you write a method in Player to accepts that dictionary. This will define the behavior of the sprite based off the keys that are pressed. Here’s what that might look like:

29 # Move the sprite based on user keypresses
30 def update(self, pressed_keys):
31     if pressed_keys[K_UP]:
32         self.rect.move_ip(0, -5)
33     if pressed_keys[K_DOWN]:
34         self.rect.move_ip(0, 5)
35     if pressed_keys[K_LEFT]:
36         self.rect.move_ip(-5, 0)
37     if pressed_keys[K_RIGHT]:
38         self.rect.move_ip(5, 0)

K_UP, K_DOWN, K_LEFT, and K_RIGHT correspond to the arrow keys on the keyboard. If the dictionary entry for that key is True, then that key is down, and you move the player .rect in the proper direction. Here you use .move_ip(), which stands for move in place, to move the current Rect.

Then you can call .update() every frame to move the player sprite in response to keypresses. Add this call right after the call to .get_pressed():

52 # Main loop
53 while running:
54     # for loop through the event queue
55     for event in pygame.event.get():
56         # Check for KEYDOWN event
57         if event.type == KEYDOWN:
58             # If the Esc key is pressed, then exit the main loop
59             if event.key == K_ESCAPE:
60                 running = False
61         # Check for QUIT event. If QUIT, then set running to false.
62         elif event.type == QUIT:
63             running = False
64 
65     # Get all the keys currently pressed
66     pressed_keys = pygame.key.get_pressed()
67 
68     # Update the player sprite based on user keypresses
69     player.update(pressed_keys)
70 
71     # Fill the screen with black
72     screen.fill((0, 0, 0))

Now you can move your player rectangle around the screen with the arrow keys:

You may notice two small problems:

The player rectangle can move very fast if a key is held down. You’ll work on that later.
The player rectangle can move off the screen. Let’s solve that one now.

To keep the player on the screen, you need to add some logic to detect if the rect is going to move off screen. To do that, you check whether the rect coordinates have moved beyond the screen’s boundary. If so, then you instruct the program to move it back to the edge:

25 # Move the sprite based on user keypresses
26 def update(self, pressed_keys):
27     if pressed_keys[K_UP]:
28         self.rect.move_ip(0, -5)
29     if pressed_keys[K_DOWN]:
30         self.rect.move_ip(0, 5)
31     if pressed_keys[K_LEFT]:
32         self.rect.move_ip(-5, 0)
33     if pressed_keys[K_RIGHT]:
34         self.rect.move_ip(5, 0)
35 
36     # Keep player on the screen
37     if self.rect.left < 0:
38         self.rect.left = 0
39     if self.rect.right > SCREEN_WIDTH:
40         self.rect.right = SCREEN_WIDTH
41     if self.rect.top <= 0:
42         self.rect.top = 0
43     if self.rect.bottom >= SCREEN_HEIGHT:
44         self.rect.bottom = SCREEN_HEIGHT

Here, instead of using .move(), you just change the corresponding coordinates of .top, .bottom, .left, or .right directly. Test this, and you’ll find the player rectangle can no longer move off the screen.

Now let’s add some enemies!

Enemies

What’s a game without enemies? You’ll use the techniques you’ve already learned to create a basic enemy class, then create a lot of them for your player to avoid. First, import the random library:

 4 # Import random for random numbers
 5 import random

Then create a new sprite class called Enemy, following the same pattern you used for Player:

55 # Define the enemy object by extending pygame.sprite.Sprite
56 # The surface you draw on the screen is now an attribute of 'enemy'
57 class Enemy(pygame.sprite.Sprite):
58     def __init__(self):
59         super(Enemy, self).__init__()
60         self.surf = pygame.Surface((20, 10))
61         self.surf.fill((255, 255, 255))
62         self.rect = self.surf.get_rect(
63             center=(
64                 random.randint(SCREEN_WIDTH + 20, SCREEN_WIDTH + 100),
65                 random.randint(0, SCREEN_HEIGHT),
66             )
67         )
68         self.speed = random.randint(5, 20)
69 
70     # Move the sprite based on speed
71     # Remove the sprite when it passes the left edge of the screen
72     def update(self):
73         self.rect.move_ip(-self.speed, 0)
74         if self.rect.right < 0:
75             self.kill()

There are four notable differences between Enemy and Player:

On lines 62 to 67, you update rect to be a random location along the right edge of the screen. The center of the rectangle is just off the screen. It’s located at some position between 20 and 100 pixels away from the right edge, and somewhere between the top and bottom edges.
On line 68, you define .speed as a random number between 5 and 20. This specifies how fast this enemy moves towards the player.
On lines 73 to 76, you define .update(). It takes no arguments since enemies move automatically. Instead, .update() moves the enemy toward the left side of the screen at the .speed defined when it was created.
On line 74, you check whether the enemy has moved off-screen. To make sure the Enemy is fully off the screen and won’t just disappear while it’s still visible, you check that the right side of the .rect has gone past the left side of the screen. Once the enemy is off-screen, you call .kill() to prevent it from being processed further.

So, what does .kill() do? To figure this out, you have to know about Sprite Groups.

Sprite Groups

Another super useful class that pygame provides is the Sprite Group. This is an object that holds a group of Sprite objects. So why use it? Can’t you just track your Sprite objects in a list instead? Well, you can, but the advantage of using a Group lies in the methods it exposes. These methods help to detect whether any Enemy has collided with the Player, which makes updates much easier.

Let’s see how to create sprite groups. You’ll create two different Group objects:

The first Group will hold every Sprite in the game.
The second Group will hold just the Enemy objects.

Here’s what that looks like in code:

82 # Create the 'player'
83 player = Player()
84 
85 # Create groups to hold enemy sprites and all sprites
86 # - enemies is used for collision detection and position updates
87 # - all_sprites is used for rendering
88 enemies = pygame.sprite.Group()
89 all_sprites = pygame.sprite.Group()
90 all_sprites.add(player)
91 
92 # Variable to keep the main loop running
93 running = True

When you call .kill(), the Sprite is removed from every Group to which it belongs. This removes the references to the Sprite as well, which allows Python’s garbage collector to reclaim the memory as necessary.

Now that you have an all_sprites group, you can change how objects are drawn. Instead of calling .blit() on just Player, you can iterate over everything in all_sprites:

117 # Fill the screen with black
118 screen.fill((0, 0, 0))
119 
120 # Draw all sprites
121 for entity in all_sprites:
122     screen.blit(entity.surf, entity.rect)
123 
124 # Flip everything to the display
125 pygame.display.flip()

Now, anything put into all_sprites will be drawn with every frame, whether it’s an enemy or the player.

There’s just one problem… You don’t have any enemies! You could create a bunch of enemies at the beginning of the game, but the game would quickly become boring when they all left the screen a few seconds later. Instead, let’s explore how to keep a steady supply of enemies coming as the game progresses.

Custom Events

The design calls for enemies to appear at regular intervals. This means that at set intervals, you need to do two things:

Create a new Enemy.
Add it to all_sprites and enemies.

You already have code that handles random events. The event loop is designed to look for random events occurring every frame and deal with them appropriately. Luckily, pygame doesn’t restrict you to using only the event types it has defined. You can define your own events to handle as you see fit.

Let’s see how to create a custom event that’s generated every few seconds. You can create a custom event by naming it:

78 # Create the screen object
79 # The size is determined by the constant SCREEN_WIDTH and SCREEN_HEIGHT
80 screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
81 
82 # Create a custom event for adding a new enemy
83 ADDENEMY = pygame.USEREVENT + 1
84 pygame.time.set_timer(ADDENEMY, 250)
85 
86 # Instantiate player. Right now, this is just a rectangle.
87 player = Player()

pygame defines events internally as integers, so you need to define a new event with a unique integer. The last event pygame reserves is called USEREVENT, so defining ADDENEMY = pygame.USEREVENT + 1 on line 83 ensures it’s unique.

Next, you need to insert this new event into the event queue at regular intervals throughout the game. That’s where the time module comes in. Line 84 fires the new ADDENEMY event every 250 milliseconds, or four times per second. You call .set_timer() outside the game loop since you only need one timer, but it will fire throughout the entire game.

Add the code to handle your new event:

100 # Main loop
101 while running:
102     # Look at every event in the queue
103     for event in pygame.event.get():
104         # Did the user hit a key?
105         if event.type == KEYDOWN:
106             # Was it the Escape key? If so, stop the loop.
107             if event.key == K_ESCAPE:
108                 running = False
109 
110         # Did the user click the window close button? If so, stop the loop.
111         elif event.type == QUIT:
112             running = False
113 
114         # Add a new enemy?
115         elif event.type == ADDENEMY:
116             # Create the new enemy and add it to sprite groups
117             new_enemy = Enemy()
118             enemies.add(new_enemy)
119             all_sprites.add(new_enemy)
120 
121     # Get the set of keys pressed and check for user input
122     pressed_keys = pygame.key.get_pressed()
123     player.update(pressed_keys)
124 
125     # Update enemy position
126     enemies.update()

Whenever the event handler sees the new ADDENEMY event on line 115, it creates an Enemy and adds it to enemies and all_sprites. Since Enemy is in all_sprites, it will get drawn every frame. You also need to call enemies.update() on line 126, which updates everything in enemies, to ensure they move properly:

However, that’s not the only reason there’s a group for just enemies.

Collision Detection

Your game design calls for the game to end whenever an enemy collides with the player. Checking for collisions is a basic technique of game programming, and usually requires some non-trivial math to determine whether two sprites will overlap each other.

This is where a framework like pygame comes in handy! Writing collision detection code is tedious, but pygame has a LOT of collision detection methods available for you to use.

For this tutorial, you’ll use a method called .spritecollideany(), which is read as “sprite collide any.” This method accepts a Sprite and a Group as parameters. It looks at every object in the Group and checks if its .rect intersects with the .rect of the Sprite. If so, then it returns True. Otherwise, it returns False. This is perfect for this game since you need to check if the single player collides with one of a Group of enemies.

Here’s what that looks like in code:

130 # Draw all sprites
131 for entity in all_sprites:
132     screen.blit(entity.surf, entity.rect)
133 
134 # Check if any enemies have collided with the player
135 if pygame.sprite.spritecollideany(player, enemies):
136     # If so, then remove the player and stop the loop
137     player.kill()
138     running = False

Line 135 tests whether player has collided with any of the objects in enemies. If so, then player.kill() is called to remove it from every group to which it belongs. Since the only objects being rendered are in all_sprites, the player will no longer be rendered. Once the player has been killed, you need to exit the game as well, so you set running = False to break out of the game loop on line 138.

At this point, you’ve got the basic elements of a game in place:

Now, let’s dress it up a bit, make it more playable, and add some advanced capabilities to help it stand out.

Sprite Images

Alright, you have a game, but let’s be honest… It’s kind of ugly. The player and enemies are just white blocks on a black background. That was state-of-the-art when Pong was new, but it just doesn’t cut it anymore. Let’s replace all those boring white rectangles with some cooler images that will make the game feel like an actual game.

Earlier, you learned that images on disk can be loaded into a Surface with some help from the image module. For this tutorial, we made a little jet for the player and some missiles for the enemies. You’re welcome to use this art, draw your own, or download some free game art assets to use. You can click the link below to download the art used in this tutorial:

Clone Repo: Click here to clone the repo you'll use to learn how to use PyGame in this tutorial.

Altering the Object Constructors

Before you use images to represent the player and enemy sprites, you need to make some changes to their constructors. The code below replaces the code used previously:

 7 # Import pygame.locals for easier access to key coordinates
 8 # Updated to conform to flake8 and black standards
 9 # from pygame.locals import *
10 from pygame.locals import (
11     RLEACCEL,
12     K_UP,
13     K_DOWN,
14     K_LEFT,
15     K_RIGHT,
16     K_ESCAPE,
17     KEYDOWN,
18     QUIT,
19 )
20 
21 # Define constants for the screen width and height
22 SCREEN_WIDTH = 800
23 SCREEN_HEIGHT = 600
24 
25 
26 # Define the Player object by extending pygame.sprite.Sprite
27 # Instead of a surface, use an image for a better-looking sprite
28 class Player(pygame.sprite.Sprite):
29     def __init__(self):
30         super(Player, self).__init__()
31         self.image = pygame.image.load("jet.png").convert()
32         self.image.set_colorkey((255, 255, 255), RLEACCEL)
33         self.rect = self.image.get_rect()

Let’s unpack line 31 a bit. pygame.image.load() loads an image from the disk. You pass it a path to the file. It returns a Surface, and the .convert() call optimizes the Surface, making future .blit() calls faster.

Line 32 uses .set_colorkey() to indicate the color pygame will render as transparent. In this case, you choose white, because that’s the background color of the jet image. The RLEACCEL constant is an optional parameter that helps pygame render more quickly on non-accelerated displays. This is added to the pygame.locals import statement on line 11.

Nothing else needs to change. The image is still a Surface, except now it has a picture painted on it. You still use it in the same way.

Here’s what similar changes to the Enemy look like:

59 # Define the enemy object by extending pygame.sprite.Sprite
60 # Instead of a surface, use an image for a better-looking sprite
61 class Enemy(pygame.sprite.Sprite):
62     def __init__(self):
63         super(Enemy, self).__init__()
64         self.surf = pygame.image.load("missile.png").convert()
65         self.surf.set_colorkey((255, 255, 255), RLEACCEL)
66         # The starting position is randomly generated, as is the speed
67         self.rect = self.surf.get_rect(
68             center=(
69                 random.randint(SCREEN_WIDTH + 20, SCREEN_WIDTH + 100),
70                 random.randint(0, SCREEN_HEIGHT),
71             )
72         )
73         self.speed = random.randint(5, 20)

Running the program now should show that this is the same game you had before, except now you’ve added some nice graphics skins with images. But why stop at just making the player and enemy sprites look nice? Let’s add a few clouds going past to give the impression of a jet flying through the sky.

Adding Background Images

For background clouds, you use the same principles as you did for Player and Enemy:

Create the Cloud class.
Add an image of a cloud to it.
Create a method .update() that moves the cloud toward the left side of the screen.
Create a custom event and handler to create new cloud objects at a set time interval.
Add the newly created cloud objects to a new Group called clouds.
Update and draw the clouds in your game loop.

Here’s what Cloud looks like:

 83 # Define the cloud object by extending pygame.sprite.Sprite
 84 # Use an image for a better-looking sprite
 85 class Cloud(pygame.sprite.Sprite):
 86     def __init__(self):
 87         super(Cloud, self).__init__()
 88         self.surf = pygame.image.load("cloud.png").convert()
 89         self.surf.set_colorkey((0, 0, 0), RLEACCEL)
 90         # The starting position is randomly generated
 91         self.rect = self.surf.get_rect(
 92             center=(
 93                 random.randint(SCREEN_WIDTH + 20, SCREEN_WIDTH + 100),
 94                 random.randint(0, SCREEN_HEIGHT),
 95             )
 96 
 97     # Move the cloud based on a constant speed
 98     # Remove the cloud when it passes the left edge of the screen
 99     def update(self):
100         self.rect.move_ip(-5, 0)
101         if self.rect.right < 0:
102             self.kill()

That should all look very familiar. It’s pretty much the same as Enemy.

To have clouds appear at certain intervals, you’ll use event creation code similar to what you used to create new enemies. Put it right below the enemy creation event:

116 # Create custom events for adding a new enemy and a cloud
117 ADDENEMY = pygame.USEREVENT + 1
118 pygame.time.set_timer(ADDENEMY, 250)
119 ADDCLOUD = pygame.USEREVENT + 2
120 pygame.time.set_timer(ADDCLOUD, 1000)

This says to wait 1000 milliseconds, or one second, before creating the next cloud.

Next, create a new Group to hold each newly created cloud:

125 # Create groups to hold enemy sprites, cloud sprites, and all sprites
126 # - enemies is used for collision detection and position updates
127 # - clouds is used for position updates
128 # - all_sprites is used for rendering
129 enemies = pygame.sprite.Group()
130 clouds = pygame.sprite.Group()
131 all_sprites = pygame.sprite.Group()
132 all_sprites.add(player)

Next, add a handler for the new ADDCLOUD event in the event handler:

137 # Main loop
138 while running:
139     # Look at every event in the queue
140     for event in pygame.event.get():
141         # Did the user hit a key?
142         if event.type == KEYDOWN:
143             # Was it the Escape key? If so, then stop the loop.
144             if event.key == K_ESCAPE:
145                 running = False
146 
147         # Did the user click the window close button? If so, stop the loop.
148         elif event.type == QUIT:
149             running = False
150 
151         # Add a new enemy?
152         elif event.type == ADDENEMY:
153             # Create the new enemy and add it to sprite groups
154             new_enemy = Enemy()
155             enemies.add(new_enemy)
156             all_sprites.add(new_enemy)
157 
158         # Add a new cloud?
159         elif event.type == ADDCLOUD:
160             # Create the new cloud and add it to sprite groups
161             new_cloud = Cloud()
162             clouds.add(new_cloud)
163             all_sprites.add(new_cloud)

Finally, make sure the clouds are updated every frame:

167 # Update the position of enemies and clouds
168 enemies.update()
169 clouds.update()
170 
171 # Fill the screen with sky blue
172 screen.fill((135, 206, 250))

Line 172 updates the original screen.fill() to fill the screen with a pleasant sky blue color. You can change this color to something else. Maybe you want an alien world with a purple sky, a toxic wasteland in neon green, or the surface of Mars in red!

Note that each new Cloud and Enemy are added to all_sprites as well as clouds and enemies. This is done because each group is used for a separate purpose:

Rendering is done using all_sprites.
Position updates are done using clouds and enemies.
Collision detection is done using enemies.

You create multiple groups so that you can change the way sprites move or behave without impacting the movement or behavior of other sprites.

Game Speed

While testing the game you may have noticed that the enemies move a little fast. If not, then that’s okay, as different machines will see different results at this point.

The reason for this is that the game loop processes frames as fast as the processor and environment will allow. Since all the sprites move once per frame, they can move hundreds of times each second. The number of frames handled each second is called the frame rate, and getting this right is the difference between a playable game and a forgettable one.

Normally, you want as high a frame rate as possible, but for this game, you need to slow it down a bit for the game to be playable. Fortunately, the module time contains a Clock which is designed exactly for this purpose.

Using Clock to establish a playable frame rate requires just two lines of code. The first creates a new Clock before the game loop begins:

106 # Setup the clock for a decent framerate
107 clock = pygame.time.Clock()

The second calls .tick() to inform pygame that the program has reached the end of the frame:

188 # Flip everything to the display
189 pygame.display.flip()
190 
191 # Ensure program maintains a rate of 30 frames per second
192 clock.tick(30)

The argument passed to .tick() establishes the desired frame rate. To do this, .tick() calculates the number of milliseconds each frame should take, based on the desired frame rate. Then, it compares that number to the number of milliseconds that have passed since the last time .tick() was called. If not enough time has passed, then .tick() delays processing to ensure that it never exceeds the specified frame rate.

Passing in a smaller frame rate will result in more time in each frame for calculations, while a larger frame rate provides smoother (and possibly faster) gameplay:

Play around with this number to see what feels best for you!

Sound Effects

So far, you’ve focused on gameplay and the visual aspects of your game. Now let’s explore giving your game some auditory flavor as well. pygame provides mixer to handle all sound-related activities. You’ll use this module’s classes and methods to provide background music and sound effects for various actions.

The name mixer refers to the fact that the module mixes various sounds into a cohesive whole. Using the music sub-module, you can stream individual sound files in a variety of formats, such as MP3, Ogg, and Mod. You can also use Sound to hold a single sound effect to be played, in either Ogg or uncompressed WAV formats. All playback happens in the background, so when you play a Sound, the method returns immediately as the sound plays.

Note: The pygame documentation states that MP3 support is limited, and unsupported formats can cause system crashes. The sounds referenced in this article have been tested, and we recommend testing any sounds thoroughly before releasing your game.

As with most things pygame, using mixer starts with an initialization step. Luckily, this is already handled by pygame.init(). You only need to call pygame.mixer.init() if you want to change the defaults:

106 # Setup for sounds. Defaults are good.
107 pygame.mixer.init()
108 
109 # Initialize pygame
110 pygame.init()
111 
112 # Set up the clock for a decent framerate
113 clock = pygame.time.Clock()

pygame.mixer.init() accepts a number of arguments, but the defaults work fine in most cases. Note that if you want to change the defaults, you need to call pygame.mixer.init() before calling pygame.init(). Otherwise, the defaults will be in effect regardless of your changes.

After the system is initialized, you can get your sounds and background music setup:

135 # Load and play background music
136 # Sound source: http://ccmixter.org/files/Apoxode/59262
137 # License: https://creativecommons.org/licenses/by/3.0/
138 pygame.mixer.music.load("Apoxode_-_Electric_1.mp3")
139 pygame.mixer.music.play(loops=-1)
140 
141 # Load all sound files
142 # Sound sources: Jon Fincher
143 move_up_sound = pygame.mixer.Sound("Rising_putter.ogg")
144 move_down_sound = pygame.mixer.Sound("Falling_putter.ogg")
145 collision_sound = pygame.mixer.Sound("Collision.ogg")

Lines 138 and 139 load a background sound clip and begin playing it. You can tell the sound clip to loop and never end by setting the named parameter loops=-1.

Lines 143 to 145 load three sounds you’ll use for various sound effects. The first two are rising and falling sounds, which are played when the player moves up or down. The last is the sound used whenever there is a collision. You can add other sounds as well, such as a sound for whenever an Enemy is created, or a final sound for when the game ends.

So, how do you use the sound effects? You want to play each sound when a certain event occurs. For example, when the ship moves up, you want to play move_up_sound. Therefore, you add a call to .play() whenever you handle that event. In the design, that means adding the following calls to .update() for Player:

26 # Define the Player object by extending pygame.sprite.Sprite
27 # Instead of a surface, use an image for a better-looking sprite
28 class Player(pygame.sprite.Sprite):
29     def __init__(self):
30         super(Player, self).__init__()
31         self.surf = pygame.image.load("jet.png").convert()
32         self.surf.set_colorkey((255, 255, 255), RLEACCEL)
33         self.rect = self.surf.get_rect()
34 
35     # Move the sprite based on keypresses
36     def update(self, pressed_keys):
37         if pressed_keys[K_UP]:
38             self.rect.move_ip(0, -5)
39             move_up_sound.play()
40         if pressed_keys[K_DOWN]:
41             self.rect.move_ip(0, 5)
42             move_down_sound.play()

For a collision between the player and an enemy, you play the sound for when collisions are detected:

201 # Check if any enemies have collided with the player
202 if pygame.sprite.spritecollideany(player, enemies):
203     # If so, then remove the player
204     player.kill()
205 
206     # Stop any moving sounds and play the collision sound
207     move_up_sound.stop()
208     move_down_sound.stop()
209     collision_sound.play()
210 
211     # Stop the loop
212     running = False

Here, you stop any other sound effects first, because in a collision the player is no longer moving. Then you play the collision sound and continue execution from there.

Finally, when the game is over, all sounds should stop. This is true whether the game ends due to a collision or the user exits manually. To do this, add the following lines at the end of the program after the loop:

220 # All done! Stop and quit the mixer.
221 pygame.mixer.music.stop()
222 pygame.mixer.quit()

Technically, these last few lines are not required, as the program ends right after this. However, if you decide later on to add an intro screen or an exit screen to your game, then there may be more code running after the game ends.

That’s it! Test it again, and you should see something like this:

A Note on Sources

You may have noticed the comment on lines 136-137 when the background music was loaded, listing the source of the music and a link to the Creative Commons license. This was done because the creator of that sound required it. The license requirements stated that in order to use the sound, both proper attribution and a link to the license must be provided.

Here are some sources for music, sound, and art that you can search for useful content:

OpenGameArt.org: sounds, sound effects, sprites, and other artwork
Kenney.nl: sounds, sound effects, sprites, and other artwork
Gamer Art 2D: sprites and other artwork
CC Mixter: sounds and sound effects
Freesound: sounds and sound effects

As you make your games and use downloaded content such as art, music, or code from other sources, please be sure that you are complying with the licensing terms of those sources.

Conclusion

Throughout this tutorial, you’ve learned how game programming with pygame differs from standard procedural programming. You’ve also learned how to:

Implement event loops
Draw items on the screen
Play sound effects and music
Handle user input

To do this, you used a subset of the pygame modules, including the display, mixer and music, time, image, event, and key modules. You also used several pygame classes, including Rect, Surface, Sound, and Sprite. But these only scratch the surface of what pygame can do! Check out the official pygame documentation for a full list of available modules and classes.

You can find all of the code, graphics, and sound files for this article by clicking the link below:

Clone Repo: Click here to clone the repo you'll use to learn how to use PyGame in this tutorial.

Feel free to leave comments below as well. Happy Pythoning!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python vs C++: Selecting the Right Tool for the Job

2019-09-11T14:00:00+00:00

Are you a C++ developer comparing Python vs C++? Are you looking at Python and wondering what all the fuss is about? Do you wonder how Python compares to the concepts you already know? Or perhaps you have a bet on who would win if you locked C++ and Python in a cage and let them battle it out? Then this article is for you!

In this article, you’ll learn about:

Differences and similarities when you’re comparing Python vs C++
Times when Python might be a better choice for a problem and vice versa
Resources to turn to as you have questions while learning Python

This article is aimed at C++ developers who are learning Python. It assumes a basic knowledge of both languages and will use concepts from Python 3.6 and up, as well as C++11 or later.

Let’s dive into looking at Python vs C++!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Comparing Languages: Python vs C++

Frequently, you’ll find articles that extoll the virtues of one programming language over another. Quite often, they devolve into efforts to promote one language by degrading the other. This isn’t that type of article.

When you’re comparing Python vs C++, remember that they’re both tools, and they both have uses for different problems. Think about comparing a hammer and a screwdriver. You could use a screwdriver to drive in nails, and you could use a hammer to force in screws, but neither experience will be all that effective.

Using the right tool for the job is important. In this article, you’ll learn about the features of Python and C++ that make each of them the right choice for certain types of problems. So, don’t view the “vs” in Python vs C++ as meaning “against.” Rather, think of it as a comparison.

Compilation vs Virtual Machine

Let’s start with the biggest difference when you’re comparing Python vs C++. In C++, you use a compiler that converts your source code into machine code and produces an executable. The executable is a separate file that can then be run as a stand-alone program:

This process outputs actual machine instructions for the specific processor and operating system it’s built for. In this drawing, it’s a Windows program. This means you’d have to recompile your program separately for Windows, Mac, and Linux:

You’ll likely need to modify your C++ code to run on those different systems as well.

Python, on the other hand, uses a different process. Now, remember that you’ll be looking at CPython which is the standard implementation for the language. Unless you’re doing something special, this is the Python you’re running.

Python runs each time you execute your program. It compiles your source just like the C++ compiler. The difference is that Python compiles to bytecode instead of native machine code. Bytecode is the native instruction code for the Python virtual machine. To speed up subsequent runs of your program, Python stores the bytecode in .pyc files:

If you’re using Python 2, then you’ll find these files next to the .py files. For Python 3, you’ll find them in a __pycache__ directory.

The generated bytecode doesn’t run natively on your processor. Instead, it’s run by the Python virtual machine. This is similar to the Java virtual machine or the .NET Common Runtime Environment. The initial run of your code will result in a compilation step. Then, the bytecode will be interpreted to run on your specific hardware:

As long as the program hasn’t been changed, each subsequent run will skip the compilation step and use the previously compiled bytecode to interpret:

Interpreting code is going to be slower than running native code directly on the hardware. So why does Python work that way? Well, interpreting the code in a virtual machine means that only the virtual machine needs to be compiled for a specific operating system on a specific processor. All of the Python code it runs will run on any machine that has Python.

Note: CPython is written in C, so it can run on most systems that have a C compiler.

Another feature of this cross-platform support is that Python’s extensive standard library is written to work on all operating systems.

Using pathlib, for example, will manage path separators for you whether you’re on Windows, Mac, or Linux. The developers of those libraries spent a lot of time making it portable so you don’t need to worry about it in your Python program!

Before you move on, let’s start keeping track of a Python vs C++ comparison chart. As you cover new comparisons, they’ll be added in italics:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x

Now that you’ve seen the differences in run time when you’re comparing Python vs C++, let’s dig into the specifics of the languages’ syntax.

Syntax Differences

Python and C++ share many syntactical similarities, but there are a few areas worth discussing:

Whitespace
Boolean expressions
Variables and pointers
Comprehensions

Let’s start with the most contentious one first: whitespace.

Whitespace

The first thing most developers notice when comparing Python vs C++ is the “whitespace issue.” Python uses leading whitespace to mark scope. This means that the body of an if block or other similar structure is indicated by the level of indentation. C++ uses curly braces ({}) to indicate the same idea.

While the Python lexer will accept any whitespace as long as you’re consistent, PEP8 (the official style guide for Python) specifies 4 spaces for each level of indentation. Most editors can be configured to do this automatically.

There has been an enormous amount of writing, shouting, and ranting about Python’s whitespace rules already, so let’s just jump past that issue and on to other matters.

Instead of relying on a lexical marker like ; to end each statement, Python uses the end of the line. If you need to extend a statement over a single line, then you can use the backslash (\) to indicate that. (Note that if you’re inside a set of parentheses, then the continuation character is not needed.)

There are people who are unhappy on both sides of the whitespace issue. Some Python developers love that you don’t have to type out braces and semicolons. Some C++ developers hate the reliance on formatting. Learning to be comfortable with both is your best bet.

Now that you’ve looked at the whitespace issue, let’s move on to one that’s a bit less contentious: Boolean expressions.

Boolean Expressions

The way you’ll use Boolean expressions changes slightly in Python vs C++. In C++, you can use numeric values to indicate true or false, in addition to the built-in values. Anything that evaluates to 0 is considered false, while every other numeric value is true.

Python has a similar concept but extends it to include other cases. The basics are quite similar. The Python documentation states that the following items evaluate to False:

Constants defined as false:
- None
- False
Zeros of any numeric type:
- 0
- 0.0
- 0j
- Decimal(0)
- Fraction(0, 1)
Empty sequences and collections:
- ''
- ()
- []
- {}
- set()
- range(0)

All other items are True. This means that an empty list [] is False, while a list containing only zero [0] is still True.

Most objects will evaluate to True, unless the object has __bool__() which returns False or __len__() which returns 0. This allows you to extend your custom classes to act as Boolean expressions.

Python has a few slight changes from C++ in the Boolean operators as well. For starters, if and while statements do not require the surrounding parentheses as they do in C++. Parentheses can aid in readability, however, so use your best judgment.

Most C++ Boolean operators have similar operators in Python:

C++ Operator	Python Operator
`&&`	`and`
`\|\|`	`or`
`!`	`not`
`&`	`&`
`\|`	`\|`

Most of the operators are similar to C++, but if you want to brush up you can read Operators and Expressions in Python.

Variables and Pointers

When you first start using Python after writing in C++, you might not give variables much thought. They seem to generally work as they do in C++. However, they’re not the same. Whereas in C++ you use variables to reference values, in Python you use names.

Note: For this section, where you’re looking at variables and names in Python vs C++, you’ll use variables for C++ and names for Python. Elsewhere, they will both be called variables.

First, let’s back up a bit and take a broader look at Python’s object model.

In Python, everything is an object. Numbers are held in objects. Modules are held in objects. Both the object of a class and the class itself are objects. Functions are also objects:

>>>

>>> a_list_object = list()
>>> a_list_object
[]
>>> a_class_object = list
>>> a_class_object
<class 'list'>
>>> def sayHi(name):
...      print(f'Hello, {name}')
...
>>> a_function_object = sayHi
>>> a_function_object
<function sayHi at 0x7faa326ac048>

Calling list() creates a new list object, which you assign to a_list_object. Using the name of the class list by itself places a label on the class object. You can place a new label on a function as well. This is a powerful tool and, like all powerful tools, it can be dangerous. (I’m looking at you, Mr. Chainsaw.)

Note: The code above is shown running in a REPL, which stands for “Read, Eval, Print Loop.” This interactive environment is used frequently to try out ideas in Python and other interpreted languages.

If you type python at a command prompt, then it will bring up a REPL where you can start typing in code and trying things out for yourself!

Moving back to the Python vs C++ discussion, note that this is behavior is different from what you’ll see in C++. Unlike Python, C++ has variables that are assigned to a memory location, and you must indicate how much memory that variable will use:

int an_int;
float a_big_array_of_floats[REALLY_BIG_NUMBER];

In Python, all objects are created in memory, and you apply labels to them. The labels themselves don’t have types, and they can be put on any type of object:

>>>

>>> my_flexible_name = 1
>>> my_flexible_name
1
>>> my_flexible_name = 'This is a string'
>>> my_flexible_name
'This is a string'
>>> my_flexible_name = [3, 'more info', 3.26]
>>> my_flexible_name
[3, 'more info', 3.26]
>>> my_flexible_name = print
>>> my_flexible_name
<built-in function print>

You can assign my_flexible_name to any type of object, and Python will just roll with it.

When you’re comparing Python vs C++, the difference in variables vs names can be a bit confusing, but it comes with some excellent benefits. One is that in Python you don’t have pointers, and you never need to think about heap vs stack issues. You’ll dive into memory management a bit later in this article.

Comprehensions

Python has a language feature called list comprehensions. While it’s possible to emulate list comprehensions in C++, it’s fairly tricky. In Python, they’re a basic tool that’s taught to beginning programmers.

One way of thinking about list comprehensions is that they’re like a super-charged initializer for lists, dicts, or sets. Given one iterable object, you can create a list, and filter or modify the original as you do so:

>>>

>>> [x**2 for x in range(5)]
[0, 1, 4, 9, 16]

This script starts with the iterable range(5) and creates a list that contains the square for each item in the iterable.

It’s possible to add conditions to the values in the first iterable:

>>>

>>> odd_squares = [x**2 for x in range(5) if x % 2]
>>> odd_squares
[1, 9]

The if x % 2 at the end of this comprehension limits the numbers used from range(5) to only the odd ones.

At this point you might be having two thoughts:

That’s a powerful syntax trick that will simplify some parts of my code.
You can do the same thing in C++.

While it’s true that you can create a vector of the squares of the odd numbers in C++, doing so usually means a little more code:

std::vector<int> odd_squares;
for (int ii = 0; ii < 10; ++ii) {
    if (ii % 2) {
        odd_squares.push_back(ii);
    }
}

For developers coming from C-style languages, list comprehensions are one of the first noticeable ways they can write more Pythonic code. Many developers start writing Python with C++ structure:

odd_squares = []
for ii in range(5):
    if (ii % 2):
        odd_squares.append(ii)

This is perfectly valid Python. It will likely run more slowly, however, and it’s not as clear and concise as the list comprehension. Learning to use list comprehensions will not only speed up your code, but it will also make your code more Pythonic and easier to read!

Note: When you’re reading about Python, you’ll frequently see the word Pythonic used to describe something. This is just a term the community uses to describe code that is clean, elegant and looks like it was written by a Python Jedi.

Python’s `std::algorithms`

C++ has a rich set of algorithms built into the standard library. Python has a similar set of built-in functions that cover the same ground.

The first and most powerful of these is the in operator, which provides a quite readable test to see if an item is included in a list, set, or dictionary:

>>>

>>> x = [1, 3, 6, 193]
>>> 6 in x
True
>>> 7 in x
False
>>> y = { 'Jim' : 'gray', 'Zoe' : 'blond', 'David' : 'brown' }
>>> 'Jim' in y
True
>>> 'Fred' in y
False
>>> 'gray' in y
False

Note that the in operator, when used on dictionaries, only tests for keys, not values. This is shown by the final test, 'gray' in y.

in can be combined with not for quite readable syntax:

if name not in y:
    print(f"{name} not found")

Next up in your parade of Python built-in operators is any. This is a boolean function that returns True if any element of the given iterable evaluates to True. This can seem a little silly until you remember your list comprehensions! Combining these two can produce powerful, clear syntax for many situations:

>>>

>>> my_big_list = [10, 23, 875]
>>> my_small_list = [1, 2, 8]
>>> any([x < 3 for x in my_big_list])
False
>>> any([x < 3 for x in my_small_list])
True

Finally, you have all, which is similar to any. This returns True only if—you guessed it—all of the elements in the iterable are True. Again, combining this with list comprehensions produces a powerful language feature:

>>>

>>> list_a = [1, 2, 9]
>>> list_b = [1, 3, 9]
>>> all([x % 2 for x in list_a])
False
>>> all([x % 2 for x in list_b])
True

any and all can cover much of the same ground where C++ developers would look to std::find or std::find_if.

Note: In the any and all examples above, you can remove the brackets ([]) without any loss of functionality. (for example: all(x % 2 for x in list_a)) This makes use of generator expressions which, while quite handy, are beyond the scope of this article.

Before moving on to variable typing, let’s update your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x

Okay, now you’re ready to look at variable and parameter typing. Let’s go!

Static vs Dynamic Typing

Another large topic when you’re comparing Python vs C++ is the use of data types. C++ is a statically typed language, while Python is dynamically typed. Let’s explore what that means.

Static Typing

C++ is statically typed, which means that each variable you use in your code must have a specific data type like int, char, float, and so forth. You can only assign values of the correct type to a variable, unless you jump through some hoops.

This has some advantages for both the developer and the compiler. The developer gains the advantage of knowing what the type of a particular variable is ahead of time, and therefore which operations are allowed. The compiler can use the type information to optimize the code, making it smaller, faster, or both.

This advance knowledge comes at a cost, however. The parameters passed into a function must match the type expected by the function, which can reduce the flexibility and potential usefulness of the code.

Duck Typing

Dynamic typing is frequently referred to as duck typing. It’s an odd name, and you’ll read more about that in just a minute! But first, let’s start with an example. This function takes a file object and reads the first ten lines:

def read_ten(file_like_object):
    for line_number in range(10):
        x = file_like_object.readline()
        print(f"{line_number} = {x.strip()}")

To use this function, you’ll create a file object and pass it in:

with open("types.py") as f:
    read_ten(f)

This shows how the basic design of the function works. While this function was described as “reading the first ten lines from a file object,” there is nothing in Python that requires that file_like_object be a file. As long as the object passed in supports .readline(), the object can be of any type:

class Duck():
    def readline(self):
        return "quack"

my_duck = Duck()
read_ten(my_duck)

Calling read_ten() with a Duck object produces:

0 = quack
1 = quack
2 = quack
3 = quack
4 = quack
5 = quack
6 = quack
7 = quack
8 = quack
9 = quack

This is the essence of duck typing. The saying goes, “If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.”

In other words, if the object has the needed methods, then it’s acceptable to pass it in, regardless of the object’s type. Duck or dynamic typing gives you an enormous amount of flexibility, as it allows any type to be used where it meets the required interfaces.

However, there is a problem here. What happens if you pass in an object that doesn’t meet the required interface? For example, what if you pass in a number to read_ten(), like this: read_ten(3)?

This results in an exception being thrown. Unless you catch the exception, your program will blow up with a traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "duck_test.py", line 4, in read_ten
    x = file_like_object.readline()
AttributeError: 'int' object has no attribute 'readline'

Dynamic typing can be quite a powerful tool, but as you can see, you must use caution when employing it.

Note: Python and C++ are both considered strongly typed languages. Although C++ has a stronger type system, the details of this are generally not significant to someone learning Python.

Let’s move on to a feature that benefits from Python’s dynamic typing: templates.

Templates

Python doesn’t have templates like C++, but it generally doesn’t need them. In Python, everything is a subclass of a single base type. This is what allows you to create duck typing functions like the ones above.

The templating system in C++ allows you to create functions or algorithms that operate on multiple different types. This is quite powerful and can save you significant time and effort. However, it can also be a source of confusion and frustration, as compiler errors in templates can leave you baffled.

Being able to use duck typing instead of templates makes some things much easier. But this, too, can cause hard-to-detect issues. As in all complex decisions, there are trade-offs when you’re comparing Python vs C++.

Type Checking

There’s been a lot of interest and discussion in the Python community lately about static type checking in Python. Projects like mypy have raised the possibility of adding pre-runtime type checking to specific spots in the language. This can be quite useful in managing interfaces between portions of large packages or specific APIs.

It helps to address one of the downsides of duck typing. For developers using a function, it helps if they can fully understand what each parameter needs to be. This can be useful on large project teams where many developers need to communicate through APIs.

Once again, let’s take a look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x

Now you’re ready to move on to differences in object-oriented programming.

Object-Oriented Programming

Like C++, Python supports an object-oriented programming model. Many of the same concepts you learned in C++ carry over into Python. You’ll still need to make decisions about inheritance, composition, and multiple inheritance.

Similarities

Inheritance between classes works similarly in Python vs C++. A new class can inherit methods and attributes from one or more base classes, just like you’ve seen in C++. Some of the details are a bit different, however.

Base classes in Python do not have their constructor called automatically like they do in C++. This can be confusing when you’re switching languages.

Multiple inheritance also works in Python, and it has just as many quirks and strange rules as it does in C++.

Similarly, you can also use composition to build classes, where you have objects of one type hold other types. Considering everything is an object in Python, this means that classes can hold anything else in the language.

Differences

There are some differences, however, when you’re comparing Python vs C++. The first two are related.

The first difference is that Python has no concept of access modifiers for classes. Everything in a class object is public. The Python community has developed a convention that any member of a class starting with a single underscore is treated as private. This is in no way enforced by the language, but it seems to work out pretty well.

The fact that every class member and method is public in Python leads to the second difference: Python has far weaker encapsulation support than C++.

As mentioned, the single underscore convention makes this far less of an issue in practical codebases than it is in a theoretical sense. In general, any user that breaks this rule and depends on the internal workings of a class is asking for trouble.

Operator Overloads vs Dunder Methods

In C++, you can add operator overloads. These allow you to define the behavior of specific syntactical operators (like ==) for certain data types. Usually, this is used to add more natural usage of your classes. For the == operator, you can define exactly what it means for two objects of a class to be equal.

One difference that takes some developers a long time to grasp is how to work around the lack of operator overloads in Python. It’s great that Python’s objects all work in any of the standard containers, but what if you want the == operator to do a deep comparison between two objects of your new class? In C++, you would create an operator==() in your class and do the comparison.

Python has a similar structure that’s used quite consistently across the language: dunder methods. Dunder methods get their name because they all start and end with a double underscore, or “d-under.”

Many of the built-in functions that operate on objects in Python are handled by calls to that object’s dunder methods. For your example above, you can add __eq__() to your class to do whatever fancy comparison you like:

class MyFancyComparisonClass():
    def __eq__(self, other):
        return True

This produces a class that compares the same way as any other instance of its class. Not particularly useful, but it demonstrates the point.

There are a large number of dunder methods used in Python, and the built-in functions make use of them extensively. For example, adding __lt__() will allow Python to compare the relative order of two of your objects. This means that not only will the < operator now work, but that >, <=, and >= will also work as well.

Even better, if you have several objects of your new class in a list, then you can use sorted() on the list and they’ll be sorted using __lt__().

Once again, let’s take a look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x

Now that you’ve seen object-oriented coding across both languages, let’s look at how Python and C++ manage those objects in memory.

Memory Management

One of the biggest differences, when you’re comparing Python vs C++, is how they handle memory. As you saw in the section about variables in C++ and Python’s names, Python does not have pointers, nor does it easily let you manipulate memory directly. While there are times when you want to have that level of control, most of the time it’s not necessary.

Giving up direct control of memory locations brings a few benefits. You don’t need to worry about memory ownership, or making sure that memory is freed once (and only once) after it’s been allocated. You also never have to worry about whether or not an object was allocated on the stack or the heap, which tends to trip up beginning C++ developers.

Python manages all of these issues for you. To do this everything in Python is a derived class from Python’s object. This allows the Python interpreter to implement reference counting as a means of keeping track of which objects are still in use and which can be freed.

This convenience comes at a price, of course. To free allocated memory objects for you, Python will occasionally need to run what is called a garbage collector, which finds unused memory objects and frees them.

Note: CPython has a complex memory management scheme, which means that freeing memory doesn’t necessarily mean the memory gets returned to the operating system.

Python uses two tools to free memory:

The reference counting collector
The generational collector

Let’s look at each of these individually.

Reference Counting Collector

The reference counting collector is fundamental to the standard Python interpreter and is always running. It works by keeping track of how many times a given block of memory (which is always a Python object) has a name attached to it while your program is running. Many rules describe when the reference count is incremented or decremented, but an example of one case might clarify:

>>>

 1 >>> x = 'A long string'
 2 >>> y = x
 3 >>> del x
 4 >>> del y

In the above example, line 1 creates a new object containing the string "A long string". It then places the name x on this object, increasing the object’s reference count to 1:

On line 2 it assigns y to name the same object, which will increase the reference count to 2:

When you call del with x in line 3, you’re removing one of the references to the object, dropping the count back to 1:

Finally, when you remove y, the final reference to the object, its reference count drops to zero and it can be freed by the reference counting garbage collector. It may or may not be freed immediately at this point, but generally, that shouldn’t matter to the developer:

While this will take care of finding and freeing many of the objects that need to be freed, there are a few situations it will not catch. For that, you need the generational garbage collector.

Generational Garbage Collector

One of the big holes in the reference counting scheme is that your program can build a cycle of references, where object A has a reference to object B, which has a reference back to object A. It’s entirely possible to hit this situation and have nothing in your code referring to either object. In this case, neither of the objects will ever hit a reference count of 0.

The generational garbage collector involves a complex algorithm that is beyond the scope of this article, but it will find some of these orphaned reference cycles and free them for you. It runs on an occasional basis controlled by settings described in the documentation. One of these parameters is to disable this garbage collector entirely.

When You Don’t Want Garbage Collection

When you’re comparing Python vs C++, as when you’re comparing any two tools, each advantage comes with a trade-off. Python doesn’t require explicit memory management, but occasionally it will spend a longer amount of time than expected on garbage collection. The inverse is true for C++: your program will have consistent response times, but you’ll need to expend more effort in managing memory.

In many programs the occasional garbage collection hit is unimportant. If you’re writing a script that only runs for 10 seconds, then you’re unlikely to notice the difference. Some situations, however, require consistent response times. Real-time systems are a great example, where responding to a piece of hardware in a fixed amount of time can be essential to the proper operation of your system.

Systems with hard real-time requirements are some of the systems for which Python is a poor language choice. Having a tightly controlled system where you’re certain of the timing is a good use of C++. These are the types of issues to consider when you’re deciding on the language for a project.

Time to update your Python vs C++ chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x
Direct Memory Control		x
Garbage Collection	x

Threading, Multiprocessing, and Async IO

The concurrency models in C++ and Python are similar, but they have different results and benefits. Both languages have support for threading, multiprocessing, and Async IO operations. Let’s look at each of these.

Threading

While both C++ and Python have threading built into the language, the results can be markedly different, depending on the problem you’re solving. Frequently, threading is used to address performance problems. In C++, threading can provide a general speed-up for both computationally bound and I/O bound problems, as threads can take full advantage of the cores on a multiprocessor system.

Python, on the other hand, has made a design trade-off to use the Global Interpreter Lock, or the GIL, to simplify its threading implementation. There are many benefits to the GIL, but the drawback is that only one thread will be running at a single time, even if there are multiple cores.

If your problem is I/O bound, like fetching several web pages at once, then this limitation will not bother you in the least. You’ll appreciate Python’s easier threading model and built-in methods for inter-thread communications. If your problem is CPU-bound, however, then the GIL will restrict your performance to that of a single processor. Fortunately, Python’s multiprocessing library has a similar interface to its threading library.

Multiprocessing

Multiprocessing support in Python is built into the standard library. It has a clean interface that allows you to spin up multiple processes and share information between them. You can create a pool of processes and spread work across them using several techniques.

While Python still uses similar OS primitives to create the new processes, much of the low-level complication is hidden from the developer.

C++ relies on fork() to provide multiprocessing support. While this gives you direct access to all of the controls and issues of spawning multiple processes, it’s also much more complex.

Async IO

While both Python and C++ support Async IO routines, they’re handled differently. In C++, the std::async methods are likely to use threading to achieve the Async IO nature of their operations. In Python, Async IO code will only run on a single thread.

There are trade-offs here as well. Using separate threads allows the C++ Async IO code to perform faster on computationally bound problems. The Python tasks used in its Async IO implementation are more lightweight, so it’s faster to spin up a large number of them to handle I/O bound issues.

Your Python vs C++ comparison chart remains unchanged for this section. Both languages support a full range of concurrency options, with varying trade-offs between speed and convenience.

Miscellaneous Issues

If you’re comparing Python vs C++ and looking at adding Python to your toolbelt, then there are a few other things to consider. While your current editor or IDE will certainly work for Python, you might want to add certain extensions or language packs. It’s also worth giving PyCharm a look, as it’s Python-specific.

Several C++ projects have Python bindings. Things like Qt, WxWidgets, and many messaging APIs having multiple-language bindings.

If you want to embed Python in C++, then you can use the Python/C API.

Finally, there are several methods for using your C++ skills to extend Python and add functionality, or to call your existing C++ libraries from within your Python code. Tools like CTypes, Cython, CFFI, Boost.Python and Swig can help you combine these languages and use each for what it’s best at.

Summary: Python vs C++

You’ve spent some time reading and thinking about the differences between Python vs C++. While Python has easier syntax and fewer sharp edges, it’s not a perfect fit for all problems. You’ve looked at the syntax, memory management, processing, and several other aspects of these two languages.

Let’s take a final look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x
Direct Memory Control		x
Garbage Collection	x

If you’re comparing Python vs C++, then you can see from your chart that this is not a case where one is better than the other. Each of them is a tool that’s well crafted for various use cases. Just like you don’t use a hammer for driving screws, using the right language for the job will make your life easier!

Conclusion

Congrats! You’ve now seen some of the strengths and weaknesses of both Python and C++. You’ve learned some of the features of each language and how they are similar.

You’ve seen that C++ is great when you want:

Fast execution speed (potentially at the cost of development speed)
Complete control of memory

Conversely, Python is great when you want:

Fast development speed (potentially at the cost of execution speed)
Managed memory

You’re now ready to make a wise language choice when it comes to your next project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Absolute vs Relative Imports in Python

2019-09-10T14:00:00+00:00

If you’ve worked on a Python project that has more than one file, chances are you’ve had to use an import statement before. Even for Pythonistas with a couple of projects under their belt, imports can be confusing!

If you’re reading this because you’d like to gain a deeper understanding of imports in Python, particularly absolute and relative imports, then you’ve come to the right place! In this tutorial, you’ll learn the differences between the two, as well as their pros and cons.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

MATLAB vs Python: Why and How to Make the Switch

2019-09-09T14:00:00+00:00

MATLAB® is widely known as a high-quality environment for any work that involves arrays, matrices, or linear algebra. Python is newer to this arena but is becoming increasingly popular for similar tasks. As you’ll see in this article, Python has all of the computational power of MATLAB for science tasks and makes it fast and easy to develop robust applications. However, there are some important differences when comparing MATLAB vs Python that you’ll need to learn about to effectively switch over.

In this article, you’ll learn how to:

Evaluate the differences of using MATLAB vs Python
Set up an environment for Python that duplicates the majority of MATLAB functions
Convert scripts from MATLAB to Python
Avoid common issues you might have when switching from MATLAB to Python
Write code that looks and feels like Python

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

MATLAB vs Python: Comparing Features and Philosophy

Python is a high-level, general-purpose programming language designed for ease of use by human beings accomplishing all sorts of tasks. Python was created by Guido van Rossum and first released in the early 1990s. Python is a mature language developed by hundreds of collaborators around the world.

Python is used by developers working on small, personal projects all the way up to some of the largest internet companies in the world. Not only does Python run Reddit and Dropbox, but the original Google algorithm was written in Python. Also, the Python-based Django Framework runs Instagram and many other websites. On the science and engineering side, the data to create the 2019 photo of a black hole was processed in Python, and major companies like Netflix use Python in their data analytics work.

There is also an important philosophical difference in the MATLAB vs Python comparison. MATLAB is proprietary, closed-source software. For most people, a license to use MATLAB is quite expensive, which means that if you have code in MATLAB, then only people who can afford a license will be able to run it. Plus, users are charged for each additional toolbox they want to install to extend the basic functionality of MATLAB. Aside from the cost, the MATLAB language is developed exclusively by Mathworks. If Mathworks were ever to go out of business, then MATLAB would no longer be able to be developed and might eventually stop functioning.

On the other hand, Python is free and open-source software. Not only can you download Python at no cost, but you can also download, look at, and modify the source code as well. This is a big advantage for Python because it means that anyone can pick up the development of the language if the current developers were unable to continue for some reason.

If you’re a researcher or scientist, then using open-source software has some pretty big benefits. Paul Romer, the 2018 Nobel Laureate in Economics, is a recent convert to Python. By his estimation, switching to open-source software in general, and Python in particular, brought greater integrity and accountability to his research. This was because all of the code could be shared and run by any interested reader. Prof. Romer wrote an excellent article, Jupyter, Mathematica, and the Future of the Research Paper, about his experience with open-source software.

Moreover, since Python is available at no cost, a much broader audience can use the code you develop. As you’ll see a little later on in the article, Python has an awesome community that can help you get started with the language and advance your knowledge. There are tens of thousands of tutorials, articles, and books all about Python software development. Here are a few to get you started:

Plus, with so many developers in the community, there are hundreds of thousands of free packages to accomplish many of the tasks that you’ll want to do with Python. You’ll learn more about how to get these packages later on in this article.

Like MATLAB, Python is an interpreted language. This means that Python code can be ported between all of the major operating system platforms and CPU architectures out there, with only small changes required for different platforms. There are distributions of Python for desktop and laptop CPUs and microcontrollers like Adafruit. Python can also talk to other microcontrollers like Arduino with a simple programming interface that is almost identical no matter the host operating system.

For all of these reasons, and many more, Python is an excellent choice to replace MATLAB as your programming language of choice. Now that you’re convinced to try out Python, read on to find out how to get it on your computer and how to switch from MATLAB!

Note: GNU Octave is a free and open-source clone of MATLAB. In this sense, GNU Octave has the same philosophical advantages that Python has around code reproducibility and access to the software.

Octave’s syntax is mostly compatible with MATLAB syntax, so it provides a short learning curve for MATLAB developers who want to use open-source software. However, Octave can’t match Python’s community or the number of different kinds of applications that Python can serve, so we definitely recommend you switch whole hog over to Python.

Besides, this website is called Real Python, not Real Octave 😀

Setting Up Your Environment for Python

In this section, you’ll learn:

How to install Python on your computer for a seamless transition from MATLAB
How to install replacements for the MATLAB integrated development environment
How to use the replacements for MATLAB on your computer

Getting Python via Anaconda

Python can be downloaded from a number of different sources, called distributions. For instance, the Python that you can download from the official Python website is one distribution. Another very popular Python distribution, particularly for math, science, engineering, and data science applications, is the Anaconda distribution.

There are two main reasons that Anaconda is so popular:

Anaconda distributes pre-built packages for Windows, macOS, and Linux, which means that the installation process is really easy and the same for all three major platforms.
Anaconda includes all of the most popular packages for engineering and data science type workloads in one single installer.

For the purposes of creating an environment that is very similar to MATLAB, you should download and install Anaconda. As of this writing, there are two major versions of Python available: Python 2 and Python 3. You should definitely install the version of Anaconda for Python 3, since Python 2 will not be supported past January 1, 2020. Python 3.7 is the most recent version at the time of this writing, but Python 3.8 should be out a few months after this article is published. Either 3.7 or 3.8 will work the same for you, so choose the most recent version you can.

Once you have downloaded the Anaconda installer, you can follow the default set up procedures depending on your platform. You should install Anaconda in a directory that does not require administrator permission to modify, which is the default setting in the installer.

With Anaconda installed, there are a few specific programs you should know about. The easiest way to launch applications is to use the Anaconda Navigator. On Windows, you can find this in the Start Menu and on macOS you can find it in Launchpad. Here’s a screenshot of the Anaconda Navigator on Windows:

In the screenshot, you can see several installed applications, including JupyterLab, Jupyter Notebook, and Spyder, that you’ll learn more about later in this tutorial.

On Windows, there is one other application that you should know about. This is called Anaconda Prompt, and it is a command prompt set up specifically to work with conda on Windows. If you want to type conda commands in a terminal, rather than using the Navigator GUI, then you should use Anaconda Prompt on Windows.

On macOS, you can use any terminal application such as the default Terminal.app or iTerm2 to access conda from the command line. On Linux, you can use the terminal emulator of your choice and which specific emulator is installed will depend on your Linux distribution.

Terminology Note: You may be a little bit confused about conda versus Anaconda. The distinction is subtle but important. Anaconda is a distribution of Python that includes many of the necessary packages for scientific work of all kinds. conda is a cross-platform package management software that is included with the Anaconda distribution of Python. conda is the software that you use to build, install, and remove packages within the Anaconda distribution.

You can read all about how to use conda in Setting Up Python for Machine Learning on Windows. Although that tutorial focuses on Windows, the conda commands are the same on Windows, macOS, and Linux.

Python also includes another way to install packages, called pip. If you’re using Anaconda, you should always prefer to install packages using conda whenever possible. Sometimes, though, a package is only available with pip, and for those cases, you can read What Is Pip? A Guide for New Pythonistas.

Getting an Integrated Development Environment

One of the big advantages of MATLAB is that it includes a development environment with the software. This is the window that you’re most likely used to working in. There is a console in the center where you can type commands, a variable explorer on the right, and a directory listing on the left.

Unlike MATLAB, Python itself does not have a default development environment. It is up to each user to find one that fits their needs. Fortunately, Anaconda comes with two different integrated development environments (IDEs) that are similar to the MATLAB IDE to make your switch seamless. These are called Spyder and JupyterLab. In the next two sections, you’ll see a detailed introduction to Spyder and a brief overview of JupyterLab.

Spyder

Spyder is an IDE for Python that is developed specifically for scientific Python work. One of the really nice things about Spyder is that it has a mode specifically designed for people like you who are converting from MATLAB to Python. You’ll see that a little later on.

First, you should open Spyder. If you followed the instructions in the previous section, you can open Spyder using the Anaconda Navigator. Just find the Spyder icon and click the Launch button. You can also launch Spyder from the Start Menu if you’re using Windows or from Launchpad if you’re using macOS.

Changing the Default Window Layout in Spyder

The default window in Spyder looks like the image below. This is for version 3.3.4 of Spyder running on Windows 10. It should look quite similar on macOS or Linux:

Before you take a tour of the user interface, you can make the interface look a little more like MATLAB. In the View → Window layouts menu choose MATLAB layout. That will change the window automatically so it has the same areas that you’re used to from MATLAB, annotated on the figure below:

In the top left of the window is the File Explorer or directory listing. In this pane, you can find files that you want to edit or create new files and folders to work with.

In the top center is a file editor. In this editor, you can work on Python scripts that you want to save to re-run later on. By default, the editor opens a file called temp.py located in Spyder’s configuration directory. This file is meant as a temporary place to try things out before you save them in a file somewhere else on your computer.

In the bottom center is the console. Like in MATLAB, the console is where you can run commands to see what they do or when you want to debug some code. Variables created in the console are not saved if you close Spyder and open it up again. The console is technically running IPython by default.

Any commands that you type in the console will be logged into the history file in the bottom right pane of the window. Furthermore, any variables that you create in the console will be shown in the variable explorer in the top right pane.

Notice that you can adjust the size of any pane by putting your mouse over the divider between panes, clicking, and dragging the edge to the size that you want. You can close any of the panes by clicking the x in the top of the pane.

You can also break any pane out of the main window by clicking the button that looks like two windows in the top of the pane, right next to the x that closes the pane. When a pane is broken out of the main window, you can drag it around and rearrange it however you want. If you want to put the pane back in the main window, drag it with the mouse so a transparent blue or gray background appears and the neighboring panes resize, then let go and the pane will snap into place.

Once you have the panes arranged exactly how you want, you can ask Spyder to save the layout. Go to the View menu and find the Window layouts flyout again. Then click Save current layout and give it a name. This lets you reset to your preferred layout at any time if something gets changed by accident. You can also reset to one of the default configurations from this menu.

Running Statements in the Console in Spyder

In this section, you’re going to be writing some simple Python commands, but don’t worry if you don’t quite understand what they mean yet. You’ll learn more about Python syntax a little later on in this article. What you want to do right now is get a sense for how Spyder’s interface is similar to and different from the MATLAB interface.

You’ll be working a lot with the Spyder console in this article, so you should learn about how it works. In the console, you’ll see a line that starts with In [1]:, for input line 1. Spyder (really, the IPython console) numbers all of the input lines that you type. Since this is the first input you’re typing, the line number is 1. In the rest of this article, you’ll see references to “input line X,” where X is the number in the square brackets.

One of the first things I like to do with folks who are new to Python is show them the Zen of Python. This short poem gives you a sense of what Python is all about and how to approach working with Python.

To see the Zen of Python, type import this on input line 1 and then run the code by pressing Enter. You’ll see an output like below:

>>>

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This code has import this on input line 1. The output from running import this is to print the Zen of Python onto the console. We’ll return to several of the stanzas in this poem later on in the article.

In many of the code blocks in this article, you’ll see three greater-than signs (>>>) in the top right of the code block. If you click that, it will remove the input prompt and any output lines, so you can copy and paste the code right into your console.

Many Pythonistas maintain a healthy sense of humor. This is displayed in many places throughout the language, including the Zen of Python. For another one, in the Spyder console, type the following code, followed by Enter to run it:

>>>

In [2]: import antigravity

That statement will open your web browser to the webcomic called XKCD, specifically comic #353, where the author has discovered that Python has given him the ability to fly!

You’ve now successfully run your first two Python statements! Congratulations 😃🎉

If you look at the History Log, you should see the first two commands you typed in the console (import this and import antigravity). Let’s define some variables and do some basic arithmetic now. In the console, type the following statements, pressing Enter after each one:

>>>

In [3]: var_1 = 10
In [4]: var_2 = 20
In [5]: var_3 = var_1 + var_2
In [6]: var_3
Out[6]: 30

In this code, you defined 3 variables: var_1, var_2, and var_3. You assigned var_1 the value 10, var_2 the value 20, and var_3 the sum of var_1 and var_2. Then you showed the value of the var_3 variable by writing it as the only thing on the input line. The output from that statement is shown on the next Out line, and the number on the Out line matches the associated In line.

There are two main things for you to notice in these commands:

If a statement does not include an assignment (with an =), it is printed onto an Out line. In MATLAB, you would need to include a semicolon to suppress the output even from assignment statements, but that is not necessary in Python.
On input lines 3, 4, and 5, the Variable explorer in the top right pane updated.

After you run these three commands, your Variable explorer should look like the image below:

In this image, you can see a table with four columns:

Name shows the name that you gave to var_1, var_2, and var_3.
Type shows the Python type of the variable, in this case, all int for integer numbers.
Size shows the size of the data stored variable, which is more useful for lists and other data structures.
Value shows the current value of the variable.

Running Code in Files in Spyder

The last stop in our brief tour of the Spyder interface is the File editor pane. In this pane, you can create and edit Python scripts and run them using the console. By default, Spyder creates a temporary file called temp.py which is intended for you to temporarily store commands as you’re working before you move or save them in another file.

Let’s write some code into the temp.py file and see how to run it. The file starts with the following code, which you can just leave in place:

 1 # -*- coding: utf-8 -*-
 2 """
 3 Spyder Editor
 4 
 5 This is a temporary script file.
 6 """

In this code, you can see two Python syntax structures:

Line 1 has a comment. In Python, the comment character is the hash or pound sign (#). MATLAB uses the percent symbol (%) as the comment character. Anything following the hash on the line is a comment and is usually ignored by the Python interpreter.
Starting on line 2 is a string that provides some context for the contents of the file. This is often referred to as a documentation string or docstring for short. You’ll learn more about docstrings in a later section.

Now you can start adding code to this file. Starting on line 8 in temp.py, enter the following code that is similar to what you already typed in the console:

 8 var_4 = 10
 9 var_5 = 20
10 var_6 = var_4 + var_5

Then, there are three ways to run the code:

You can use the F5 keyboard shortcut to run the file just like in MATLAB.
You can click the green right-facing triangle in the menu bar just above the Editor and File explorer panes.
You can use the Run → Run menu option.

The first time you run a file, Spyder will open a dialog window asking you to confirm the options you want to use. For this test, the default options are fine and you can click Run at the bottom of the dialog box:

This will automatically execute the following code in the console:

>>>

In [7]: runfile('C:/Users/Eleanor/.spyder-py3/temp.py',
   ...:         wdir='C:/Users/Eleanor/.spyder-py3')

This code will run the file that you were working on. Notice that running the file added three variables into the Variable explorer: var_4, var_5, and var_6. These are the three variables that you defined in the file. You will also see runfile() added to the History log.

In Spyder, you can also create code cells that can be run individually. To create a code cell, add a line that starts with # %% into the file open in the editor:

11 # %% This is a code cell
12 var_7 = 42
13 var_8 = var_7 * 2
14 
15 # %% This is a second code cell
16 print("This code will be executed in this cell")

In this code, you have created your first code cell on line 11 with the # %% code. What follows is a line comment and is ignored by Python. On line 12, you are assigning var_7 to have the value 42 and then line 13 assigns var_8 to be var_7 times two. Line 15 starts another code cell that can be executed separately from the first one.

To execute the code cells, click the Run Current Cell or Run Current Cell and Go to the Next One buttons next to the generic Run button in the toolbar. You can also use the keyboard shortcuts Ctrl+Enter to run the current cell and leave it selected, or Shift+Enter to run the current cell and select the next cell.

Spyder also offers easy-to-use debugging features, just like in MATLAB. You can double-click any of the line numbers in the Editor to set a breakpoint in your code. You can run the code in debug mode using the blue right-facing triangle with two vertical lines from the toolbar, or the Ctrl+F5 keyboard shortcut. This will pause execution at any breakpoints you specify and open the ipdb debugger in the console which is an IPython-enhanced way to run the Python debugger pdb. You can read more in Python Debugging With pdb.

Summarizing Your Experience in Spyder

Now you have the basic tools to use Spyder as a replacement for the MATLAB integrated development environment. You know how to run code in the console or type code into a file and run the file. You also know where to look to see your directories and files, the variables that you’ve defined, and the history of the commands you typed.

Once you’re ready to start organizing your code into modules and packages, you can check out the following resources:

Spyder is a really big piece of software, and you’ve only just scratched the surface. You can learn a lot more about Spyder by reading the official documentation, the troubleshooting and FAQ guide, and the Spyder wiki.

JupyterLab

JupyterLab is an IDE developed by Project Jupyter. You may have heard of Jupyter Notebooks, particularly if you’re a data scientist. Well, JupyterLab is the next iteration of the Jupyter Notebook. Although at the time of this writing JupyterLab is still in beta, Project Jupyter expects that JupyterLab will eventually replace the current Notebook server interface. However, JupyterLab is fully compatible with existing Notebooks so the transition should be fairly seamless.

JupyterLab comes preinstalled with Anaconda, so you can launch it from the Anaconda Navigator. Find the JupyterLab box and click Launch. This will open your web browser to the address http://localhost:8888/lab.

The main JupyterLab window is shown in the picture below:

There are two main sections of the interface:

On the left is a File explorer that lets you open files from your computer.
On the right side of the window is how you can open create new Notebook files, work in an IPython console or system terminal, or create a new text file.

If you’re interested in learning more about JupyterLab, you can read a lot more about the next evolution of the Notebook in the blog post announcing the beta release or in the JupyterLab documentation. You can also learn about the Notebook interface in Jupyter Notebook: An Introduction and the Using Jupyter Notebooks course. One neat thing about the Jupyter Notebook-style document is that the code cells you created in Spyder are very similar to the code cells in a Jupyter Notebook.

Learning About Python’s Mathematical Libraries

Now you’ve got Python on your computer and you’ve got an IDE where you feel at home. So how do you learn about how to actually accomplish a task in Python? With MATLAB, you can use a search engine to find the topic you’re looking for just by including MATLAB in your query. With Python, you’ll usually get better search results if you can be a bit more specific in your query than just including Python.

In this section, you’ll take the next step to really feeling comfortable with Python by learning about how Python functionality is divided into several libraries. You’ll also learn what each library does so you can get top-notch results with your searches!

Python is sometimes called a batteries-included language. This means that most of the important functions you need are already included when you install Python. For instance, Python has a math library and a statistics library built-in that include the basic operations.

Sometimes, though, you want to do something that isn’t included in the language. One of the big advantages of Python is that someone else has probably done whatever you need to do and published the code to accomplish that task. There are several hundred-thousand publicly available and free packages that you can easily install to perform various tasks. These range from processing PDF files to building and hosting an interactive website to working with highly optimized mathematical and scientific functions.

Working with arrays or matrices, optimization, or plotting requires additional libraries to be installed. Fortunately, if you install Python with the Anaconda installer these libraries come preinstalled and you don’t need to worry. Even if you’re not using Anaconda, they are usually pretty easy to install for most operating systems.

The set of important libraries you’ll need to switch over from MATLAB are typically called the SciPy stack. At the base of the stack are libraries that provide fundamental array and matrix operations (NumPy), integration, optimization, signal processing, and linear algebra functions (SciPy), and plotting (Matplotlib). Other libraries that build on these to provide more advanced functionality include Pandas, scikit-learn, SymPy, and more.

NumPy (Numerical Python)

NumPy is probably the most fundamental package for scientific computing in Python. It provides a highly efficient interface to create and interact with multi-dimensional arrays. Nearly every other package in the SciPy stack uses or integrates with NumPy in some way.

NumPy arrays are the equivalent to the basic array data structure in MATLAB. With NumPy arrays, you can do things like inner and outer products, transposition, and element-wise operations. NumPy also contains a number of useful methods for reading text and binary data files, fitting polynomial functions, many mathematical functions (sine, cosine, square root, and so on), and generating random numbers.

The performance-sensitive parts of NumPy are all written in the C language, so they are very fast. NumPy can also take advantage of optimized linear algebra libraries such as Intel’s MKL or OpenBLAS to further increase performance.

Note:

Real Python has several articles that cover how you can use NumPy to speed up your Python code:

SciPy (Scientific Python)

The SciPy package (as distinct from the SciPy stack) is a library that provides a huge number of useful functions for scientific applications. If you need to do work that requires optimization, linear algebra or sparse linear algebra, discrete Fourier transforms, signal processing, physical constants, image processing, or numerical integration, then SciPy is the library for you! Since SciPy implements so many different features, it’s almost like having access to a bunch of the MATLAB toolboxes in one package.

SciPy relies heavily on NumPy arrays to do its work. Like NumPy, many of the algorithms in SciPy are implemented in C or Fortran, so they are also very fast. Also like NumPy, SciPy can take advantage of optimized linear algebra libraries to further improve performance.

Matplotlib (MATLAB-like Plotting Library)

Matplotlib is a library to produce high-quality and interactive two-dimensional plots. Matplotlib is designed to provide a plotting interface that is similar to the plot() function in MATLAB, so people switching from MATLAB should find it somewhat familiar. Although the core functions in Matplotlib are for 2-D data plots, there are extensions available that allow plotting in three dimensions with the mplot3d package, plotting geographic data with cartopy, and many more listed in the Matplotlib documentation.

Note:

Here are some more resources on Matplotlib:

Other Important Python Libraries

With NumPy, SciPy, and Matplotlib, you can switch a lot of your MATLAB code to Python. But there are a few more libraries that might be helpful to know about.

Pandas provides a DataFrame, an array with the ability to name rows and columns for easy access.
SymPy provides symbolic mathematics and a computer algebra system.
scikit-learn provides many functions related to machine learning tasks.
scikit-image provides functions related to image processing, compatible with the similar library in SciPy.
Tensorflow provides a common platform for many machine learning tasks.
Keras provides a library to generate neural networks.
multiprocessing provides a way to perform multi-process based parallelism. It’s built into Python.
Pint provides a unit library to conduct automatic conversion between physical unit systems.
PyTables provides a reader and writer for HDF5 format files.
PyMC3 provides Bayesian statistical modeling and probabilistic machine learning functionality.

Syntax Differences Between MATLAB® and Python

In this section, you’ll learn how to convert your MATLAB code into Python code. You’ll learn about the main syntax differences between MATLAB and Python, see an overview of basic array operations and how they differ between MATLAB and Python, and find out about some ways to attempt automatic conversion of your code.

The biggest technical difference between MATLAB and Python is that in MATLAB, everything is treated as an array, while in Python everything is a more general object. For instance, in MATLAB, strings are arrays of characters or arrays of strings, while in Python, strings have their own type of object called str. This has profound consequences for how you approach coding in each language, as you’ll see below.

With that out of the way, let’s get started! To help you, the sections below are organized into groups based on how likely you are to run into that syntax.

You Will Probably See This Syntax

The examples in this section represent code that you are very likely to see in the wild. These examples also demonstrate some of the more basic Python language features. You should make sure that you have a good grasp of these examples before moving on.

Comments Start With `#` in Python

In MATLAB, a comment is anything that follows a percent sign (%) on a line. In Python, comments are anything that follow the hash or pound sign (#). You already saw a Python comment in the earlier section about Spyder. In general, the Python interpreter ignores the content of comments, just like the MATLAB interpreter, so you can write whatever content you want in the comment. One exception to this rule in Python is the example you saw earlier in the section about Spyder:

# -*- coding: utf-8 -*-

When the Python interpreter reads this line, it will set the encoding that it uses to read the rest of the file. This comment must appear in one of the first two lines of the file to be valid.

Another difference between MATLAB and Python is in how inline documentation is written. In MATLAB, documentation is written at the start of a function in a comment, like the code sample below:

function [total] = addition(num_1,num_2)
% ADDITION  Adds two numbers together
%   TOTAL = ADDITION(NUM_1,NUM_2) adds NUM_1 and NUM_2 together
%
%   See also SUM and PLUS

However, Python does not use comments in this way. Instead, Python has an idea called documentation strings or docstrings for short. In Python, you would document the MATLAB function shown above like this:

def addition(num_1, num_2):
    """Adds two numbers together.

    Example
    -------
    >>> total = addition(10, 20)
    >>> total
    30

    """

Notice in this code that the docstring is between two sets of three quote characters ("""). This allows the docstring to run onto multiple lines with the whitespace and newlines preserved. The triple quote characters are a special case of string literals. Don’t worry too much about the syntax of defining a function yet. You’ll see more about that in a later section.

Whitespace at the Beginning of a Line Is Significant in Python

When you write code in MATLAB, blocks like if statements, for and while loops, and function definitions are finished with the end keyword. It is generally considered a good practice in MATLAB to indent the code within the blocks so that the code is visually grouped together, but it is not syntactically necessary.

For example, the following two blocks of code are functionally equivalent in MATLAB:

 1 num = 10;
 2 
 3 if num == 10
 4 disp("num is equal to 10")
 5 else
 6 disp("num is not equal to 10")
 7 end
 8 
 9 disp("I am now outside the if block")

In this code, you are first creating num to store the value 10 and then checking whether the value of num is equal to 10. If it is, you are displaying the phrase num is equal to 10 on the console from line 2. Otherwise, the else clause will kick in and display num is not equal to 10. Of course, if you run this code, you will see the num is equal to 10 output and then I am now outside the if block.

Now you should modify your code so it looks like the sample below:

 1 num = 10;
 2 
 3 if num == 10
 4     disp("num is equal to 10")
 5 else
 6     disp("num is not equal to 10")
 7 end
 8 
 9 disp("I am now outside the if block")

In this code, you have only changed lines 3 and 5 by adding some spaces or indentation in the front of the line. The code will perform identically to the previous example code, but with the indentation, it is much easier to tell what code goes in the if part of the statement and what code is in the else part of the statement.

In Python, indentation at the start of a line is used to delimit the beginning and end of class and function definitions, if statements, and for and while loops. There is no end keyword in Python. This means that indentation is very important in Python!

In addition, in Python the definition line of an if/else/elif statement, a for or while loop, a function, or a class is ended by a colon. In MATLAB, the colon is not used to end the line.

Consider this code example:

 1 num = 10
 2 
 3 if num == 10:
 4     print("num is equal to 10")
 5 else:
 6     print("num is not equal to 10")
 7 
 8 print("I am now outside the if block")

On the first line, you are defining num and setting its value to 10. On line 2, writing if num == 10: tests the value of num compared to 10. Notice the colon at the end of the line.

Next, line 3 must be indented in Python’s syntax. On that line, you are using print() to display some output to the console, in a similar way to disp() in MATLAB. You’ll read more about print() versus disp() in a later section.

On line 4, you are starting the else block. Notice that the e in the else keyword is vertically aligned with the i in the if keyword, and the line is ended by a colon. Because the else is dedented relative to print() on line 3, and because it is aligned with the if keyword, Python knows that the code within the if part of the block has finished and the else part is starting. Line 5 is indented by one level, so it forms the block of code to be executed when the else statement is satisfied.

Lastly, on line 6 you are printing a statement from outside the if/else block. This statement will be printed regardless of the value of num. Notice that the p in print() is vertically aligned with the i in if and the e in else. This is how Python knows that the code in the if/else block has ended. If you run the code above, Python will display num is equal to 10 followed by I am now outside the if block.

Now you should modify the code above to remove the indentation and see what happens. If you try to type the code without indentation into the Spyder/IPython console, you will get an IndentationError:

>>>

In [1]: num = 10
In [2]: if num == 10:
   ...: print("num is equal to 10")
  File "<ipython-input-2-f453ffd2bc4f>", line 2
    print("num is equal to 10")
        ^
IndentationError: expected an indented block

In this code, you first set the value of num to 10 and then tried to write the if statement without indentation. In fact, the IPython console is smart and automatically indents the line after the if statement for you, so you’ll have to delete the indentation to produce this error.

When you’re indenting your code, the official Python style guide called PEP 8 recommends using 4 space characters to represent one indentation level. Most text editors that are set up to work with Python files will automatically insert 4 spaces if you press the Tab key on your keyboard. You can choose to use the tab character for your code if you want, but you shouldn’t mix tabs and spaces or you’ll probably end up with a TabError if the indentation becomes mismatched.

Conditional Statements Use `elif` in Python

In MATLAB, you can construct conditional statements with if, elseif, and else. These kinds of statements allow you to control the flow of your program in response to different conditions.

You should try this idea out with the code below, and then compare the example of MATLAB vs Python for conditional statements:

 1 num = 10;
 2 if num == 10
 3     disp("num is equal to 10")
 4 elseif num == 20
 5     disp("num is equal to 20")
 6 else
 7     disp("num is neither 10 nor 20")
 8 end

In this code block, you are defining num to be equal to 10. Then you are checking if the value of num is 10, and if it is, using disp() to print output to the console. If num is 20, you are printing a different statement, and if num is neither 10 nor 20, you are printing the third statement.

In Python, the elseif keyword is replaced with elif:

 1 num = 10
 2 if num == 10:
 3     print("num is equal to 10")
 4 elif num == 20:
 5     print("num is equal to 20")
 6 else:
 7     print("num is neither 10 nor 20")

This code block is functionally equivalent to the previous MATLAB code block. There are 2 main differences. On line 4, elseif is replaced with elif, and there is no end statement to end the block. Instead, the if block ends when the next dedented line of code is found after the else. You can read more in the Python documentation for if statements.

Calling Functions and Indexing Sequences Use Different Brackets in Python

In MATLAB, when you want to call a function or when you want to index an array, you use round brackets (()), sometimes also called parentheses. Square brackets ([]) are used to create arrays.

You can test out the differences in MATLAB vs Python with the example code below:

>>>

>> arr = [10, 20, 30];
>> arr(1)

ans =

    10

>> sum(arr)

ans =

    60

In this code, you first create an array using the square brackets on the right side of the equal sign. Then, you retrieve the value of the first element by arr(1), using the round brackets as the indexing operator. On the third input line, you are calling sum() and using the round brackets to indicate the parameters that should be passed into sum(), in this case just arr. MATLAB computes the sum of the elements in arr and returns that result.

Python uses separate syntax for calling functions and indexing sequences. In Python, using round brackets means that a function should be executed and using square brackets will index a sequence:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[0]
Out[2]: 10

In [3]: sum(arr)
Out[3]: 60

In this code, you are defining a Python list on input line 1. Python lists have some important distinctions from arrays in MATLAB and arrays from the NumPy package. You can read more about Python lists in Lists and Tuples in Python, and you’ll learn more about NumPy arrays in a later section.

On the input line 2, you are displaying the value of the first element of the list with the indexing operation using square brackets. On input line 3, you are calling sum() using round brackets and passing in the list stored in arr. This results in the sum of the list elements being displayed on the last line. Notice that Python uses square brackets for indexing the list and round brackets for calling functions.

The First Index in a Sequence Is 0 in Python

In MATLAB, you can get the first value from an array by using 1 as the index. This style follows the natural numbering convention and starts how you would count the number of items in the sequence. You can try out the differences of MATLAB vs Python with this example:

>>>

>> arr = [10, 20, 30];
>> arr(1)

ans =

    10

>> arr(0)
Array indices must be positive integers or logical values.

In this code, you are creating an array with three numbers: 10, 20, and 30. Then you are displaying the value of the first element with the index 1, which is 10. Trying to access the zeroth element results in an error in MATLAB, as shown on the last two lines.

In Python, the index of the first element in a sequence is 0, not 1:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[0]
Out[2]: 10

In [3]: arr[1]
Out[3]: 20

In [4]: a_string = "a string"
In [5]: a_string[0]
Out[5]: 'a'

In [6]: a_string[1]
Out[6]: ' '

In this code, you are defining arr as a Python list with three elements on input line 1. On input line 2, you are displaying the value of the first element of the list, which has the index 0. Then you are displaying the second element of the list, which has the index 1.

On input lines 4, 5, and 6, you are defining a_string with the contents "a string" and then getting the first and second elements of the string. Notice that the second element (character) of the string is a space. This demonstrates a general Python feature, that many variable types operate as sequences and can be indexed, including lists, tuples, strings, and arrays.

The Last Element of a Sequence Has Index `-1` in Python

In MATLAB, you can get the last value from an array by using end as the index. This is really useful when you don’t know how long an array is, so you don’t know what number to access the last value.

Try out the differences in MATLAB vs Python with this example:

>>>

>> arr = [10, 20, 30];
>> arr(end)

ans =

    30

In this code, you are creating an array with three numbers, 10, 20, and 30. Then you are displaying the value of the last element with the index end, which is 30.

In Python, the last value in a sequence can be retrieved by using the index -1:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[-1]
Out[2]: 30

In this code, you are defining a Python list with three elements on input line 1. On input line 2, you are displaying the value of the last element of the list, which has the index -1 and the value 30.

In fact, by using negative numbers as the index values you can work your way backwards through the sequence:

>>>

In [3]: arr[-2]
Out[3]: 20

In [4]: arr[-3]
Out[4]: 10

In this code, you are retrieving the second-to-last and third-to-last elements from the list, which have values of 20 and 10, respectively.

Exponentiation Is Done With `**` in Python

In MATLAB, when you want to raise a number to a power you use the caret operator (^). The caret operator is a binary operator that takes two numbers. Other binary operators include addition (+), subtraction (-), multiplication (*), and division (/), among others. The number on the left of the caret is the base and the number on the right is the exponent.

Try out the differences of MATLAB vs Python with this example:

>>>

>> 10^2

ans =

    100

In this code, you are raising 10 to the power of 2 using the caret resulting an answer of 100.

In Python, you use two asterisks (**) when you want to raise a number to a power:

>>>

In [1]: 10 ** 2
Out[1]: 100

In this code, you are raising 10 to the power of 2 using two asterisks resulting an answer of 100. Notice that there is no effect of including spaces on either side of the asterisks. In Python, the typical style is to have spaces on both sides of a binary operator.

The Length of a Sequence Is Found With `len()` in Python

In MATLAB, you can get the length of an array with length(). This function takes an array as the argument and returns back the size of the largest dimension in the array. You can see the basics of this function with this example:

>>>

>> length([10, 20, 30])

ans =

    3

>> length("a string")

ans =

    1

In this code, on the first input line you are finding the length of an array with 3 elements. As expected, length() returns an answer of 3. On the second input line, you are finding the length of the string array that contains one element. Notice that MATLAB implicitly creates a string array, even though you did not use the square brackets to indicate it is an array.

In Python, you can get the length of a sequence with len():

>>>

In [1]: len([10, 20, 30])
Out[1]: 3

In [2]: len("a string")
Out[2]: 8

In this code, on the input line 1 you are finding the length of a list with 3 elements. As expected, len() returns a length of 3. On input line 2, you are finding the length of a string as the input. In Python, strings are sequences and len() counts the number of characters in the string. In this case, a string has 8 characters.

Console Output Is Shown With `print()` in Python

In MATLAB, you can use disp(), fprintf(), and sprintf() to print the value of variables and other output to the console. disp() is the simplest of these commands but does not give you any control over the format of the output. fprintf() and sprintf() give you total control over the output format, but the format specification is a little bit complicated.

In Python, print() serves the same function as disp(). Unlike disp(), print() can send its output to a file similar to fprintf().

Python’s print() will display any number of arguments passed to it, separating them by a space in the output. This is different from disp() in MATLAB, which only takes one argument. The following example shows how Python’s print() can take any number of arguments, and the each argument is separated by a space in the output:

>>>

In [1]: val_1 = 10
In [2]: val_2 = 20
In [3]: str_1 = "any number of arguments"
In [4]: print(val_1, val_2, str_1)
10 20 any number of arguments

In this code, the input lines 1, 2, and 3 define val_1, val_2, and str_1, where val_1 and val_1 are integers, and str_1 is a string of text. On input line 4, you are printing the three variables using print(). The output below this line the value of the three variables are shown in the console output, separated by spaces.

You can control the separator used in the output between arguments to print() by using the sep keyword argument:

>>>

In [5]: print(val_1, val_2, str_1, sep="; ")
10; 20; any number of arguments

In this code, you are printing the same three variables but setting the separator to be a semicolon followed by a space. This separator is printed between the first and second and the second and third arguments, but not after the third argument. To control the character printed after the last value, you can use the end keyword argument to print():

>>>

In [6]: print(val_1, val_2, str_1, sep="; ", end=";")
10; 20; any number of arguments;

In this code, you have added the end keyword argument to print(), setting it to print a semicolon after the last value. This is shown in the output on line below the input.

Like disp() from MATLAB, print() cannot directly control the output format of variables. If you want more control over the format of the output, you should use f-strings or format(). In these strings, you can use very similar formatting style codes as fprintf() in MATLAB to format numbers. You can read a lot more about f-strings and other string formatting in Python 3’s f-Strings or the official documentation:

>>>

In [7]: print(f"Python's print() can handle {str_1}")
Python's print() can handle any number of arguments

In [8]: print(f"The value of val_1 = {val_1:8.3f}")
The value of val_1 =   10.000

In [9]: # The following line will only work in Python 3.8

In [10]: print(f"The value of {val_1=} and {val_2=}")
The value of val_1 = 10, and val_2 = 20

In this code, input line 7 includes an f-string, indicated by the f to start the string. This means that Python will substitute the value of any variables it encounters between {}, or curly braces, within the string. You can see that in the output, Python has replaced {str_1} with any number of arguments, the value of str_1. On input line 8, you are using some of the formatting syntax to print the value of val_1 as a floating point number with 8 columns in the output and 3 digits of precision.

On input line 10, a new feature that is coming in Python 3.8 (due out in late 2019) is demonstrated. If a variable name is immediately followed by an equals sign inside curly braces, the name of the variable and the value will be printed automatically.

You can take a deep dive into Python’s print() by checking out The Ultimate Guide to Python Print.

You Will Probably See These, but You Can Learn Them When You Need To

In this section, you’ll find examples of code that you’ll probably see in the wild, but you can wait a little while to understand them if you want. These examples use some intermediate features in Python but are still in the core of how Python works. Just like in the last section, you’ll see comparisons of the MATLAB vs Python syntax differences.

Function Definitions Start With `def` and `return` Values in Python

In MATLAB, you can define a function by placing the function keyword at the start of a line. This is followed by the name of any output variables, an equals (=) sign, then the name of the function and any input arguments in parentheses. Within the the function you have to assign to any variables you specified in the definition line as outputs. A simple example MATLAB function is shown below:

 1 function [total] = addition(num_1,num_2)
 2 total = num_1 + num_2;
 3 end

In this code, you see the function definition on line 1. There is only one output variable, called total, for this function. The name of the function is addition and it takes two arguments, which will be assigned the names num_1 and num_2 in the function body. Line 2 is the implementation of the function. The value of total is set equal to the sum of num_1 and num_2. The last line of the function is the end keyword that tells the MATLAB interpreter the definition of the function has finished.

To use this function in MATLAB, you should save it in a file called addition.m, matching the name of the function. Alternatively, it can be placed in file with other commands provided that the function definition is the last thing in the file and the file is not named addition.m. Then, you can run the function by typing the following code in the MATLAB console:

>>>

>> var_1 = 20;
>> var_2 = 10;
>> sum_of_vars = addition(var_1,var_2)

sum_of_vars =

    30

In this code, you have defined two variables called var_1 and var_2 that hold the values 20 and 10, respectively. Then you created a third variable called sum_of_vars that stores the output from addition(). Check out the Variable explorer, and you’ll see that sum_of_vars has the value 30, as expected. Notice that the name sum_of_vars did not have to be the same name as the output variable used in the function definition, which was total.

MATLAB does not require a function to provide an output value. In this case, you would remove the output variable and the equals sign from the function definition. Modify your addition.m file so that the code looks like this:

 1 function addition(num_1,num_2)
 2 total = num_1 + num_2;
 3 end

The only change in this code from the earlier code is that you deleted the [total] = from line 1, the other lines are exactly the same. Now if you try to assign the result of calling this function to a variable, MATLAB will generate an error in the console:

>>>

>> var_1 = 20;
>> var_2 = 10;
>> sum_of_vars = addition(var_1,var_2);
Error using addition
Too many output arguments.

In this code, you defined the same two variables var_1 and var_2 as before and called addition() in the same way as before. However, since addition() no longer specifies an output variable, MATLAB generates an error message that there are too many output arguments. Clicking on the word addition will open the definition of the function for you to edit or view the source code to fix the problem.

In Python, the def keyword starts a function definition. The def keyword must be followed by the name of the function and any arguments to the function inside parentheses, similar to MATLAB. The line with def must be ended with a colon (:).

Starting on the next line, the code that should be executed as part of the function must be indented one level. In Python, the function definition ends when a line of code starts at the same indentation level as the def keyword on the first line.

If your function returns some output back to the caller, Python does not require that you specify a name for an output variable. Instead, you use the return statement to send an output value from the function.

An equivalent function in Python to your first addition() example with an output variable is shown below:

 1 def addition(num_1, num_2):
 2     total = num_1 + num_2
 3     return total

In this code, you see the def keyword followed by the function name and the two arguments num_1 and num_2 on line 1. On line 2 you can see the creation of a new variable total to store the sum of num_1 and num_2, and on line 3 the value of total is returned to the point where this function was called. Notice that lines 2 and 3 are indented by 4 spaces because they make up the body of the function.

The variable that stores the sum of num_1 and num_2 can have any name, it doesn’t have to be called total. In fact, you don’t need to create a variable there at all. You can simplify your previous function definition by eliminating total and simply returning the value of num_1 + num_2:

 1 def addition(num_1, num_2):
 2     return num_1 + num_1

Line 1 in this code is the same as it was before, you have only changed line 2 and deleted line 3. Line 2 now computes the value of num_1 + num_2 and returns that value back to the caller of the function. Line 2 is indented by 4 spaces because it makes up the body of the function.

To use this function in Python, you do not need to save it in a file with a special name. You can place the function definition in any Python file, at any point in the file. There is no restriction that the function definition has to be last. In fact, you can even define functions right from the console, which is not possible in MATLAB.

Open Spyder and in the Console pane type:

>>>

In [1]: def addition(num_1, num_2):

On this line of code you are creating the function definition. In the Spyder/IPython console, once you start a function definition and press Enter, the start of the line becomes three dots and the cursor is automatically indented. Now you can type the remainder of the function definition. You’ll have to press Enter twice to complete the definition:

>>>

In [1]: def addition(num_1, num_2):
   ...:     return num_1 + num_2
   ...:

In this code, you have the definition of the function on the first line and the body of the function on the second line. The console automatically adds the ...: at the start of the lines to indicate these are continuation lines that apply to the function definition.

Once you’ve completed the definition, you can execute the function from the console as well. You should type this code:

>>>

In [2]: var_1 = 20
In [3]: var_2 = 10
In [4]: sum_of_vars = addition(var_1, var_2)
In [5]: sum_of_vars
Out[5]: 30

In this code, you first create two variables var_1 and var_2 that store the values you want to add together. Then, on input line 4, you assign sum_of_vars to the result that is returned from addition(). On input line 5, you are outputting the value of sum_of_vars to the console screen. This displays 30, the sum of 10 and 20.

In Python, if you do not explicitly put a return statement, your function will implicitly return the special value None. You should change your Python definition of addition() to see how this works. In the Spyder/IPython console, type the following:

>>>

In [6]: def addition(num_1, num_2):
   ...:     total = num_1 + num_2
   ...:

In this code, you have the same def line on input line 6. You have changed the first continuation line to assign the result of the addition to total instead of returning. Now you should see what happens when we execute this modified function:

>>>

In [7]: sum_of_vars = addition(var_1, var_2)
In [8]: sum_of_vars
In [9]:

In this code, on input line 7 you are assigning sum_of_vars to be the returned value from addition(). Then, on input line 8, you are showing the value of sum_of_vars on the console screen, just like before. This time though, there is no output! By default, Python prints nothing when it outputs a variable whose value is None. You can double check the value of the sum_of_vars variable by looking at the Variable explorer. In the Type column, it should list NoneType, telling you that sum_of_vars is the special None value.

Functions Accept Positional and Keyword Arguments in Python

In MATLAB, functions have input arguments specified on the first line, in the function definition. When you call a function in MATLAB, you can pass from zero up to the number of arguments that are specified. In the body of the function, you can check the number of input arguments the caller actually passed to execute different code. This is useful when you want different arguments to have different meaning, like in the example below:

 1 function [result] = addOrSubtract(num_1,num_2,subtract)
 2 % ADDORSUBTRACT  Add or subtract two value
 3 %   RESULT = addOrSubtract(NUM_1,NUM_2) adds NUM_1 and NUM_2 together
 4 %
 5 %   RESULT = addOrSubtract(NUM_1,NUM_2,true) subtracts NUM_2 from NUM_1
 6 
 7     switch nargin
 8         case 2
 9             result = num_1 + num_2;
10         case 3
11             result = num_1 - num_2;
12         otherwise
13             result = 0;
14     end
15 end

In this code, you are defining a function with three possible input arguments. On line 7, you are starting a switch/case block that determines how many input arguments were passed to the function by using the special variable nargin. This variable stores the actual number of arguments the caller passed into the function.

In your code above, you are defining three cases:

If the number of input arguments is 2, you are adding num_1 and num_2 together.
If the number of input arguments is 3, you are subtracting num_2 from num_1.
If fewer than 2 arguments are passed, the output will be 0.

If more than 3 arguments are passed, MATLAB will raise an error.

Now you should experiment with this function. Save the code above into a file called addOrSubtract.m and then on the MATLAB console, try the version with two input arguments:

>>>

>> addOrSubtract(10,20)

ans =
    30

In this code, you are calling addOrSubtract() with two arguments, so the arguments are added together, resulting in an answer of 30. Next, try calling addOrSubtract() with three arguments:

>>>

>> addOrSubtract(10,20,true)

ans =
   -10

In this code, you used three input arguments, and found that the second argument was subtracted from the first, resulting in an answer of -10. Third, try calling addOrSubtract() with one argument:

>>>

>> addOrSubtract(10)

ans =
     0

In this code, you used one input argument and found the answer was 0, because MATLAB only found one argument to the function and used the otherwise case. Finally, try calling addOrSubtract() with four arguments:

>>>

>> addOrSubtract(10,20,true,30)
Error using addOrSubtract
Too many input arguments.

In this code, you find that MATLAB raises an error because there were more input arguments passed than were defined in the function line.

There are four key takeaways from this example with MATLAB:

There is only one kind of argument in a function definition.
The meaning of an argument in the code is determined by its position in the function definition.
The maximum number of arguments that can be passed to a function is determined by the number of arguments specified in the function definition.
Any number of arguments up to the maximum can be passed by the caller.

In Python, there are two kinds of arguments you can specify when defining a function. These are required and optional arguments. The key difference between these is that required arguments must be passed when a function is called, while optional are given a default value in the function definition.

You can see the differences between these two styles in the next example:

 1 def add_or_subtract(num_1, num_2, subtract=False):
 2     """Add or subtract two numbers, depending on the value of subtract."""
 3     if subtract:
 4         return num_1 - num_2
 5     else:
 6         return num_1 + num_2

In this code, you are defining a function called add_or_subtract() that has three arguments: num_1, num_2, and subtract. In the function definition, you can see the two types of arguments. The first two arguments, num_1 and num_2, are required arguments.

The third argument, subtract, has a default value assigned to it by specifying a value after an equals sign in the function definition. This means that when the function is called, passing a value for subtract is optional. If no value is passed, the default as defined in the function definition line will be used. In this case, the default value is False.

In the body of the function, you are testing the value of subtract with the if statement to determine whether addition or subtraction should be performed. If subtract is True, num_2 will be subtracted from num_1. Otherwise, if subtract is False, then num_1 will be added to num_2. In either case, the result of the arithmetic operation will be returned to the caller.

In addition to the two types of arguments you can use when defining a function, there are two kinds of arguments you can specify when calling a function. These are called positional and keyword arguments. You can see the difference between these in the following example. First, try passing only two arguments to the function:

>>>

In [1]: add_or_subtract(10, 20)
Out[1]: 30

In this code, you passed only two arguments to add_or_subtract(), 10 and 20. In this case, you passed these values as positional arguments, and the meaning of the arguments is defined by their position in the function call.

Since only the two required arguments were passed, subtract will take on the default value, which is False. Therefore, 10 and 20 will be added together, which you can see on the output line. Next, try passing a value for subtract:

>>>

In [2]: add_or_subtract(10, 20, False)
Out[2]: 30
In [3]: add_or_subtract(10, 20, True)
Out[3]: -10

In this code, you passed three arguments to add_or_subtract(), with two different values for the subtract argument. First, you passed False on input line 2. The result was the addition of 10 and 20. Then, you passed True on input line 3, resulting in the difference between 10 and 20, or -10.

In these examples, you saw that it is possible in Python to define default values for arguments to a function. This means when you call the function, any arguments with default values are optional and do not have to be passed. If no value is passed for any default arguments, the default value will be used. However, you must pass a value for every argument without a default value. Otherwise, Python will raise an error:

>>>

In [4]: add_or_subtract(10)
Traceback (most recent call last):

  File "<ipython-input-4-f9d1f2ae4494>", line 1, in <module>
    add_or_subtract(10)

TypeError: add_or_subtract() missing 1 required positional argument: 'num_2'

In this code, you have only passed one of the two required arguments to add_or_subtract(), so Python raises a TypeError. The error message tells you that you did not pass a value for num_2, because it does not have a default value.

In these last three example, you have used positional arguments, so which parameter is assigned to the variables in the function depends on the order they are passed. There is another method to pass arguments to functions in Python, called keyword arguments. To use keyword arguments, you specify the name of the argument in the function call:

>>>

In [5]: add_or_subtract(num_1=10, num_2=20, subtract=True)
Out[5]: -10

In this code, you have used keyword arguments for all three arguments to add_or_subtract(). Keyword arguments are specified by stating the argument name, then an equals sign, then the value that argument should have. One of the big advantages of keyword arguments is that they make your code more explicit. (As the Zen of Python says, explicit is better than implicit.) However, they make the code somewhat longer, so it’s up to your judgement when to use keyword arguments or not.

Another benefit of keyword arguments is that they can be specified in any order:

>>>

In [6]: add_or_subtract(subtract=True, num_2=20, num_1=10)
Out[6]: -10

In this code, you have specified the three arguments for add_or_subtract() as keyword arguments, but the order is different from in the function definition. Nonetheless, Python connects the right variables together because they are specified as keywords instead of positional arguments.

You can also mix positional and keyword arguments together in the same function call. If positional and keyword arguments are mixed together, the positional arguments must be specified first, before any keyword arguments:

>>>

In [7]: add_or_subtract(10, 20, subtract=True)
Out[7]: -10

In this code, you have specified the values for num_1 and num_2 using positional arguments, and the value for subtract using a keyword argument. This is probably the most common case of using keyword arguments, because it provides a good balance between being explicit and being concise.

Finally, there is one last benefit of using keyword arguments and default values. Spyder, and other IDEs, provide introspection of function definitions. This will tell you the names of all of the defined function arguments, which ones have default arguments, and the value of the default arguments. This can save you time and make your code easier and faster to read.

There Are No `switch`/`case` Blocks in Python

In MATLAB, you can use switch/case blocks to execute code by checking the value of a variable for equality with some constants. This type of syntax is quite useful when you know you want to handle a few discrete cases. Try out a switch/case block with this example:

num = 10;
switch num
    case 10
        disp("num is 10")
    case 20
        disp("num is 20")
    otherwise
        disp("num is neither 10 nor 20")
end

In this code, you start by defining num and setting it equal to 10 and on the following lines you test the value of num. This code will result in the output num is 10 being displayed on the console, since num is equal to 10.

This syntax is an interesting comparison of MATLAB vs Python because Python does not have a similar syntax. Instead, you should use an if/elif/else block:

num = 10
if num == 10:
    print("num is 10")
elif num == 20:
    print("num is 20")
else:
    print("num is neither 10 nor 20")

In this code, you start by defining num and setting it equal to 10. On the next several lines you are writing an if/elif/else block to check the different values that you are interested in.

Namespaces Are One Honking Great Idea in Python

In MATLAB, all functions are found in a single scope. MATLAB has a defined search order for finding functions within the current scope. If you define your own function for something that MATLAB already includes, you may get unexpected behavior.

As you saw in the Zen of Python, namespaces are one honking great idea. Namespaces are a way to provide different scopes for names of functions, classes, and variables. This means you have to tell Python which library has the function you want to use. This is a good thing, especially in cases where multiple libraries provide the same function.

For instance, the built-in math library provides a square root function, as does the more advanced NumPy library. Without namespaces, it would be more difficult to tell Python which square root function you wanted to use.

To tell Python where a function is located, you first have to import the library, which creates the namespace for that library’s code. Then, when you want to use a function from the library, you tell Python which namespace to look in:

>>>

In [1]: import math
In [2]: math.sqrt(4)
Out[2]: 2.0

In this code, on input line 1 you imported the math library that is built-in to Python. Then, input line 2 computes the square root of 4 using the square root function from within the math library. The math.sqrt() line should be read as “from within math, find sqrt().”

The import keyword searches for the named library and binds the namespace to the same name as the library by default. You can read more about how Python searches for libraries in Python Modules and Packages – An Introduction.

You can also tell Python what name it should use for a library. For instance, it is very common to see numpy shortened to np with the following code:

>>>

In [3]: import numpy as np
In [4]: np.sqrt(4)
Out[4]: 2.0

In this code, input line 3 imports NumPy and tells Python to put the library into the np namespace. Then, whenever you want to use a function from NumPy, you use the np abbreviation to find that function. On input line 4, you are computing the square root of 4 again, but this time, using np.sqrt(). np.sqrt() should be read as “from within NumPy, find sqrt().”

There are two main caveats to using namespaces where you should be careful:

You should not name a variable with the same name as one of the functions built into Python. You can find a complete list of these functions in the Python documentation. The most common variable names that are also built-in functions and should not be used are dir, id, input, list, max, min, sum, str, type, and vars.
You should not name a Python file (one with the extension .py) with the same name as a library that you have installed. In other words, you should not create a Python file called math.py. This is because Python searches the current working directory first when it tries to import a library. If you have a file called math.py, that file will be found before the built-in math library and you will probably see an AttributeError.

The Most Recent Unassigned Result Is Available as `_` in Python

The MATLAB console uses ans to store the result of the most recent calculation if that result was not assigned to a variable. This is really useful when you forgot to assign the result of a calculation to a variable or when you just want to chain a few simple calculations together. To see the differences between MATLAB vs Python, try out this example:

>>>

>> sum([10, 20, 30])

ans =

    60

>> ans + 10

ans =

    70

In this code, you use sum() to calculate the sum of the array. Since there is no equal sign with a variable name on the left, MATLAB assigns the output from sum() to ans. You can then use that variable in further calculations, as you do here by adding 10 to the last result. Note that this will only work in the MATLAB console, not in a script file.

In the Python console (including the IPython/Spyder console), the output from the most recent calculation is stored in _ (the underscore character). Try the following code:

>>>

In [1]: sum([10, 20, 30])
Out[1]: 60

In [2]: _ + 10
Out[2]: 70

In this code, you use sum() to calculate the sum of the list. Since there is no equal sign with a variable name on the left, Python assigns the output from sum() to the underscore (_), in addition to printing it on the output line. You can then use that variable in further calculations, as you do here by adding 10 to the last result. Note that this will only work in the Python console, not in a script file.

In the IPython console, there is one additional feature enabled. You can append a number after the underscore to retrieve the result of any previous line. Try the following code:

>>>

In [3]: _1 + 20
Out[3]: 80

In [4]: _2 + 20
Out[4]: 90

In this code on input line 3 you are using _1 to mean the value of output line 1, the sum() line. To that result (60) you are adding 20, producing a result of 80. On input line 4, you are adding 20 to the value of output line 2, accessed with _2, so the result is 90.

Notice that the Spyder Variable explorer does not show this variable by default, whereas ans is shown in the MATLAB Variable explorer. In a few sections, you’ll see why the underscore isn’t shown by default and how you can see it.

Anonymous Functions Are Created With the `lambda` Keyword in Python

MATLAB uses the the at-symbol (@) to indicate that what follows is the definition of an anonymous function. Anonymous functions are functions that are not defined in a program file and do not use the function keyword. A program file is a MATLAB file with a filename ending in .m. Anonymous functions are limited to a single statement so they are intended for simple computations.

You can try out the differences of anonymous functions in MATLAB vs Python with this example:

>>>

>> sayHello = @(x) fprintf("Hello, %s\n",x);
>> sayHello("Eleanor")
Hello, Eleanor

In this code, the first input line defines the anonymous function with one input parameter, x. The body of the function follows, which uses fprintf() to format the input into a string. This function is assigned to sayHello. On the second input line, sayHello() is executed and passed "Eleanor" as the value. The result the string Hello, Eleanor printed on the console.

Anonymous functions are most often used when you need to pass one function into another function. In these cases, it is often not necessary to assign the function definition to a variable:

>>>

>> integral(@(x) x.^2,0,9)

ans =

    243

In this code, the first input line executes integral(), a function that computes the definite integral of a given function. The first argument to integral() must be a function, so this is a perfect place to use an anonymous function. Here, your anonymous function squares whatever the input value is. The other two arguments to integral() are the limits of the integration, such that the result of integrating x.^2 from 0 to 9 is 243.

Python uses the lambda keyword to define anonymous functions. Other than this syntax difference, anonymous functions work the same way in Python as in MATLAB:

>>>

In [1]: say_hello = lambda x: print(f"Hello, {x:s}")
In [2]: say_hello("Eleanor")
Hello, Eleanor

In this code, input line 1 defines the lambda function with one parameter, x. You use print() within the function definition to show an f-string with the value of the input parameter. The function is then stored in say_hello(). Input line 2 evaluates say_hello() with the input string "Eleanor" and produces the Hello, Eleanor output.

In Python, the official style guide called PEP 8 specifically disrecommends assigning lambda expressions to variable names, as you saw in the last example. If you want to give a function a name to refer to it several times, the preference is to use the def syntax and define a full function, even for one-line functions.

However, lambda functions are still useful when they are passed as arguments into another function:

>>>

In [3]: from scipy import integrate
In [4]: integrate.quad(lambda x: x ** 2, 0, 9)
Out[4]: (243.0, 2.6978419498391304e-12)

In this code, input line 3 imports the scipy.integrate library and stores it in the integrate namespace. On input line 4, you are using quad() from scipy.integrate to compute the integral using quadrature, very similar to integral() in MATLAB. The first argument to quad() is the function to be integrated, and you use a lambda function to specify that x ** 2 should be integrated. The second and third arguments to quad() specify that the integral should be conducted from 0 to 9.

You can see that the result on output line 4 has two values, 243.0 and 2.6978419498391304e-12. The first value is the result of the integration, and is equal to the result from MATLAB. The second value is an estimate of the absolute error in the result. An error this small is approximately the precision of the numbers used to store the result, so the answer is about as accurate as it can be.

You can read more about lambda in How to Use Python lambda Functions or watch the videos in the How to Use Python Lambda Functions course.

You Will Only Need These in Specialized Situations

In this section, the examples are more advanced concepts that you will need as you become more advanced in Python development. Some of the examples here deal with developing applications or higher-level code than the other sections. When you see these concepts in other code, you can dig in to them when you feel comfortable.

Class Definitions Start With `class` in Python

MATLAB has two ways to define a class. With the first way, you can put all of the class definition in a single file with the name of the class as the filename. Then within the file, you can use the classdef keyword to define the properties and methods that belong to the class.

With the second way, you can create a folder that starts with @ and has the same name as the class. In that folder, you can create a single file that has the same name as the class. The function definition in that file will be used as the class initializer, and it should call class() to instantiate the class. Methods of the class can be defined in other files in the same folder, where the name of each file must be the same as the name of the method.

Python only has one way to define a class, using the class keyword. Since Python uses indentation to find the end of the class definition, the entire definition must be contained in a single file:

class MyClass:
    # The rest of the class definition goes here

In this code, the first line defines the name of the class. It starts with the class keyword, followed by the name of the class and a colon. Underneath this line, all of the code that is part of the class definition (methods and attributes) must be indented. Once a line of code starts in the same column as the c in class, the class definition will be ended.

The second line in this code is a comment to note that the rest of the class definition would follow the class line.

As in all object-oriented code, Python classes can inherit from superclasses. The superclass of a given class can be given as a parameter in the class definition, as shown below:

class MyClass(MySuperClass):
    # The rest of the class definition goes here

In this code, the only change is that the name of the superclass is listed inside round brackets before the colon.

There Are No Private Properties or Methods in Python

MATLAB allows class properties and method to be set as one of four Access options:

public: Access to the property or method is unrestricted.
protected: Access to the property or method is only allowed in this class or subclasses.
private: Access to the property or method is only allowed in this class.
meta.class or {meta.class}: Access to the property or method is only allowed in the listed class or classes.

This allows you to specifically control the ways that a property or class method can be accessed.

In Python, there is no way to set a class or instance attribute or method as protected or private. All classes and class instances can have their attributes and methods changed at runtime. Python’s convention is that attributes and methods that start with an underscore (_) are intended to be private, or at least non-public. However, this convention is not enforced by any checks in the language and all attributes and methods can be modified by the user at runtime.

As you saw in the section about using the underscore to retrieve values in the console, the underscore is not visible by default in the Spyder Variable explorer. This is because Spyder and other tools respect the convention that underscore indicates something should be non-public. However, underscores can be shown in the Spyder Variable explorer if you click the gear icon in the top right of that pane and uncheck the Exclude private variables item. This will also show other non-public variables as well.

Python uses several special methods that start with a double-underscore (__), called dunder methods, to implement specific behavior for classes. The most commonly used dunder method is __init__(), which is the class initializer or constructor. You can read a lot more about dunder methods in Enriching Your Python Classes With Dunder (Magic, Special) Methods.

If you want more information about Python’s classes, you can read Object-Oriented Programming in Python vs Java. Even though that article is about Java, Java is similar to the MATLAB OOP paradigm in terms of the nature of attributes and methods.

A Class Refers to Itself as `self` in Python

MATLAB uses the name obj when a class wants to refer to the current instance of itself. The obj should be the first argument passed to an ordinary method. MATLAB also defines static methods that have no reference to the class instance.

Python uses the name self when a class wants to refer to the current instance of itself, but this is actually only a convention. You can call the first argument to an instance method any name you want, but self is the most common convention. Python also defines static methods that don’t take an argument of the class instance and class methods that take an argument of the class object instead of the instance. You can read more about instance, static, and class methods in Python’s Instance, Class, and Static Methods Demystified.

There Is One String Type in Python

In MATLAB, strings of characters are stored in string arrays when you use double quotes (") or in character arrays if you use single quotes ('). If you use both single and double quotes in an array assignment, the array will be promoted to a string array.

In character arrays, each character in the string occupies one column in the array. For multidimensional character arrays, each row of the array must have the same number of characters, which is to say, the same number of columns. This is shown in the example below:

>>>

 1 >> charArray = ['Real'; 'Python'];
 2 Dimensions of arrays being concatenated are not consistent.
 3 
 4 >> charArray = ['MATLAB'; 'Python'];
 5 >> size(charArray)
 6 
 7 ans =
 8 
 9      2     6

In this example, line 1 shows an attempt to define a 2-row character array using single quotes. However, the number of characters in Real is not the same as in Python, so MATLAB shows an error message that the dimensions are not consistent.

On line 4, you successfully create a character array, and on the third input line you are checking the size of the array. The output shows that there are 2 rows, as expected, and 6 columns, since the length of both MATLAB and Python are 6 characters.

This is not the case for string arrays. In string arrays, each string occupies one column in the array and each row in a multidimensional array must have the same number of strings, although each string can have different length. This is shown in the example below:

>>>

 1 >> stringArray = ["Real", "Python"; "Real"];
 2 Error using vertcat
 3 Dimensions of arrays being concatenated are not consistent.
 4 
 5 >> stringArray = ["Real"; "Python"];
 6 >> size(stringArray)
 7 
 8 ans =
 9 
10      2     1

In this code, line 1 shows an attempt to define a 2-row string array using double quotes. However, the number of strings on the first row (2) does not match the number of strings in the second row (1), so MATLAB raises an error.

On line 5, you successfully create a string array. Notice that even though the number of characters is different between Real and Python, MATLAB is able to create the string array. On line 6 you are checking the size of the string array, which shows that there are 2 rows and 1 column, as expected.

In Python, there is only one string literal type, called str. You can create a string literal using single quotes (') or double quotes ("), there is no difference between the two definitions. However, there are some good arguments to prefer double quotes when defining string literals in Python, which are well expressed by the Black code formatting library.

There is one additional way to define strings in Python, using triple-single-quotes (''') or triple-double-quotes ("""). This method of creating strings allows the strings to be defined across multiple lines with the newline characters retained. You can see an example of this in the section about comments and docstrings.

You can read a lot more about defining strings in Basic Data Types in Python and Strings and Character Data in Python.

You can generate similar data structures to the string arrays and character arrays in MATLAB using NumPy in Python. NumPy has several data types, or dtypes, that are related to strings. In Python 3, the default string dtype for arrays is a fixed-width Unicode string:

>>>

In [1]: import numpy as np
In [2]: arr = np.array(("Real", "Python"))
In [3]: arr
Out[3]: array(['Real', 'Python'], dtype='<U6')

In this code, you are importing the NumPy library on input line 1 and assigning it to the np abbreviation. On input line 2, you are creating a NumPy array with 2 string elements, Real and Python, and assigning the array to arr.

On input line 3, you are showing the value of arr. The output from the third line shows that arr is storing an array that has 2 elements, 'Real' and 'Python', as expected. Notice that although you defined the array with double-quote strings, Python is displaying them with single-quote strings. Remember that there is no difference between single and double quotes in Python.

Output line 3 also shows the dtype of the data in the array. For this array, the dtype is <U6. The three characters here represent the aspects of how the strings are arranged in memory. The < means that the byte order of the array is little endian. The U means the string is of the Unicode type. Finally, the 6 means the maximum length of an element is 6 characters. This was chosen as the length of the longest string in the input.

Notice that the string Real only has 4 characters. In NumPy string dtype arrays, elements can have fewer than the maximum number of characters without problems, but assigning to elements with strings that are longer than the maximum length will truncate the input:

>>>

In [4]: arr[0] = "My favorite language is Python"
In [5]: arr
Out[5]: array(['My fav', 'Python'], dtype='<U6')

In this code, you are attempting to reassign the first element of the array with the string My favorite language is Python. Clearly, this string is longer than 6 characters, so it is truncated to only 6 when it is assigned, My fav. (The space counts as 1 character.)

If you want to create an array that can hold strings of any length, you should pass the object dtype when you create the array:

>>>

In [6]: arr_2 = np.array(("Real", "Python"), dtype=object)
In [7]: arr_2
Out[7]: array(['Real', 'Python'], dtype=object)

In this code, you are creating a new array, arr_2, with two elements again, but this time you specified the dtype to be object, which you confirmed by showing the output on the console. Now you should see how the object dtype affects assigning long strings to an element:

>>>

In [8]: arr_2[0] = "My favorite language is Python"
In [9]: arr_2
Out[9]: array(['My favorite language is Python', 'Python'], dtype=object)

In this code, you are again assigning the first element of the array to have the value My favorite language is Python. You can see from the output line that that string is stored as the first element of the array without truncation, because the dtype is object. The disadvantage of using the object dtype is that it is usually much slower than the more specific U dtype, because it has to create a a whole Python object for each element rather than just a NumPy-optimized Unicode object.

One other difference you will notice from MATLAB is how the shape or size of the array is determined:

>>>

In [10]: arr.shape
Out[10]: (2,)
In [11]: arr_2.shape
Out[11]: (2,)

In this code, we are printing the shape of arr and arr_2. Notice that they both have the same shape, two elements in a one-dimensional array. This is similar to the string array from MATLAB, where each string counts as one element in the array. However, the fact that NumPy arrays with the U dtype have a fixed maximum size behaves more like the character array from MATLAB. You’ll see more about differences in how MATLAB and NumPy compute the shape of arrays in a later section.

Libraries Are Not Automatically Reloaded in Python

When executing a function or script, MATLAB will always use the most up-to-date copy of the file on the disk. Therefore, as you’re developing a script, you can run it in the console many times and new changes that you make will automatically be picked up.

Python operates somewhat differently. Remember that when you want to access code from a file, you have to import it into a namespace. When Python imports a file or module, it only reads the code the first time it is imported. This saves quite a bit of time if you’re importing the same file several times. However, if you’re testing your code in the interactive console prompt as you work on it, Python will not pick up any changes if you import it again.

When you’re developing a module, you have a few options to have Python reload your code when it is imported. If you’re using the Spyder IDE, this is not a problem at all, since Spyder has an automatic User Module Reloading feature enabled by default.

Otherwise, if you’re using the IPython console outside of Spyder, or the Jupyter Notebook, you can use a magic command defined in those interpreters called autoreload:

>>>

In [1]: %load_ext autoreload
In [2]: %autoreload 2

In this code, you are using the load_ext magic command to load the autoreload extension. In IPython and Jupyter Notebooks, commands prefixed with the percent sign % are magic commands. The autoreload extension defines the autoreload magic function, which you use on input line 2. You are passing the parameter 2 to the autoreload magic function, which means that all modules should be reloaded every time a line of code is executed.

An Overview of Basic Array Operations

As you have seen, Python does not include a high-speed library for arrays in its standard library. However, the excellent NumPy library is easily available if you install Anaconda. NumPy functions as the de facto array and matrix library for Python.

NumPy has two array-like types:

numpy.ndarray, also known as numpy.array
numpy.matrix

The main difference between these two types is that the ndarray can be any number of dimensions, while the matrix is limited to exactly two dimensions. For ndarray, all operations such as addition, subtraction, multiplication, exponentiation, and division operate element-wise. However, for the matrix type, operations like multiplication and exponentiation are matrix operations.

When you’re converting from MATLAB, the matrix type may seem more familiar. It offers similar behavior that you may be used to from MATLAB in terms of operation syntax. However, NumPy strongly recommends that you use the ndarray type because it is more flexible and because matrix will eventually be removed.

In the rest of this section, you will get to know the major differences between MATLAB and NumPy arrays. You can go in-depth on how to use NumPy arrays by reading Look Ma, No For-Loops: Array Programming With NumPy.

Basic Mathematical Operators Work Element-Wise in NumPy

MATLAB, with its heritage as a matrix scripting language, assumes that all arithmetic operators will be operating on arrays. Therefore, MATLAB treats the multiplication of matrices or vectors as matrix multiplication. Consider this example:

>>>

>> arr_1 = [1,2,3];
>> arr_2 = [4,5,6];
>> arr_1 * arr_2
Error using  *
Incorrect dimensions for matrix multiplication. Check that the number of
columns in the first matrix matches the number of rows in the second
matrix. To perform elementwise multiplication, use '.*'.

In this code, you are creating two 1x3 matrices, arr_1, and arr_2. Then, you are attempting to multiply them together. For these 1xN arrays, this is equivalent to taking the dot or scalar product. However, the scalar product only works when the left operand is 1xN and the right is Nx1, so MATLAB produces an error message and suggests the dot-star operator (.*) as the proper syntax for element-wise multiplication:

>>>

>> arr_1 .* arr_2

ans =

     4    10    18

In this code, you are performing the element-wise multiplication of arr_1 and arr_2. This multiplies the first element of arr_1 with the first element of arr_2 (4*1 = 4), second with second (2*5 = 10), and third with third (3*6 = 18).

To perform the scalar product, you can take the transpose of arr_2 to convert it to a 3x1 array:

>>>

>> arr_1 * transpose(arr_2)

ans =

    32

In this code, you are performing matrix multiplication with arr_1 and the transpose of arr_2. Note that you can use either transpose() or the quote operator (') to take the transpose of arr_2. Since arr_1 is 1x3 and transpose(arr_2) is 3x1, this results in the scalar, or dot, product.

With NumPy arrays, operations like multiplication with the asterisk (*) operate element-wise by default:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.array([1, 2, 3])
In [3]: arr_2 = np.array([4, 5, 6])

In [4]: arr_1 * arr_2
Out[4]: np.array([ 4, 10, 18])

In this code, you are first importing the NumPy package and assigning it to the name np. Then you are creating two one-dimensional arrays. Notice the syntax for creating arrays in NumPy. It starts with np.array(), which should be read as “from within np, find array().” Then, you have to pass a Python list or tuple to the array constructor that contains the elements of the array. In this case, you are passing a Python list, denoted by the square brackets.

Finally, on input line 4, you are multiplying arr_1 and arr_2. Notice that the result on output line 4 is another array with the elements 4, 10, and 18, the same result as the element-wise multiplication in MATLAB.

If you want to perform the dot or scalar product for two arrays in NumPy, you have two options. The preferred option is to use the matrix multiplication operator (@) added in Python 3.5. You may see some older code also use dot() from the NumPy library and pass the two arrays:

>>>

In [5]: arr_1 @ arr_2
Out[5]: 32

In [6]: np.dot(arr_1, arr_2)
Out[6]: 32

In this code, input line 5 uses the matrix multiplication operator to find the scalar product of arr_1 and arr_2. As expected, the result is 32. Input line 5 uses dot() and should be read as “from within np, find dot() and pass arr_1 and arr_2.” You can see that the result is identical.

Notice that NumPy did not require you to transpose arr_2 before performing the scalar product. You’ll learn more about this feature in the next section.

One-Dimensional Arrays Are Vectors in NumPy

As you saw in the last section, MATLAB insists that the dimensions of arrays align when performing matrix multiplication, while NumPy is a little bit more flexible. This is because of how one-dimensional arrays are treated in MATLAB versus in NumPy.

In MATLAB, every array always has at least two dimensions, even if only implicitly. You can see this by checking the size() of a single number:

>>>

>> size(1)

ans =

     1     1

Here, you are finding the size of the integer 1. You can see that the result is an array with 1 row and 1 column.

You can create row vectors or column vectors in MATLAB, and switch between them with the transpose operator (') or transpose():

>>>

>> arr_1 = [1,2,3];
>> arr_2 = [1;2;3];
>> arr_1' == arr_2

ans =

  3×1 logical array

   1
   1
   1

>> arr_2' == arr_1

ans =

  1×3 logical array

   1   1   1

In this code, you are creating two vectors: arr_1 and arr_2. Arrays with values in one dimension only are called vectors. arr_1 is a row vector because the elements are arranged in one row and three columns, whereas arr_2 is a column vector because the elements are arranged in three rows and one column. In MATLAB, elements are put into different columns by separating them with a comma in the assignment and elements are put into different rows by separating them with a semicolon.

Then, you are checking the equality of the transpose of arr_1 with arr_2, and you find that all of the elements are equal and the result is a column vector of logical values. Finally, you are checking the equality of the transpose of arr_2 with arr_1, and you find that all of the elements are equal and the result is a row vector of logical values.

You can see that in MATLAB, even vectors have two dimensions associated with them: rows and columns. When the transpose is performed, the rows are switched with the columns, and the shape of the array is changed. This means there are two types of vectors in MATLAB: row-vectors and column-vectors.

In NumPy, there are three types of one-dimensional arrays or vectors. The default is an N-element vector with only one dimension. This is different from the default in MATLAB, where every array has at least 2 dimensions. This one-dimensional vector in NumPy does not have a sense of rows and columns, since for a one-dimensional structure, it does not matter in general whether the elements are stored in rows or in columns, only how many elements there are.

You can see an example of creating this kind of array in the following example. In the next few examples, there are extra spaces added before and after parentheses to clarify the syntax. These spaces are usually not considered good Python style, but they’re in the example to help you see what’s going on:

>>>

In [1]: import numpy as np
In [2]: arr_vec = np.array( [1, 2, 3] )
In [3]: arr_vec.shape
Out[3]: (3,)

In this code, you are creating a default 3-element vector in NumPy. On input line 1, you import NumPy and make it available under np. On input line 2 you are creating the array and storing it in arr_vec. You are passing the list [1, 2, 3] to array(), where the list has 3 elements and none of the elements are themselves lists. This creates the 3-element array with only one dimension.

You can verify that this is the case by displaying the shape of the array, as shown on input line 3. That line should be read as “from within arr_vec (an array), find the shape.” The shape of the array is equivalent to size() in MATLAB. In this case, the shape is (3,), indicating there are three elements and only one dimension, since there is not a second number after the comma.

You can also create row-vectors and column-vectors in NumPy, analogous to the row-vectors and column-vectors in MATLAB. NumPy’s array() takes a flat list or a nested list as input. Using the flat list gets you a one-dimensional, N-element vector. By using the nested list, you can create arrays of any dimension that you want. A nested list means that there are one or more lists contained within an outer list. Here’s an example of a nested list:

[[1, 2, 3], [4, 5, 6]]

In this code, you see an outer list that has 2 elements. Each of these 2 elements of the outer list is another, nested, list that has three elements, the integers 1-3 and 4-6. In terms of arrays, you can think of the number of elements of each inner list as the number of columns, and the number of nested lists is the number of rows. This is easier to see if you change the formatting:

[
    [1, 2, 3],
    [4, 5, 6]
]

This code is still valid Python syntax, but it emphasizes how the inner lists are each a row of the array, and the number of elements in each inner list is the number of columns. In this case, we would have an array with 2 rows and 3 columns. We can use these nested lists to create row-vectors and column-vectors in NumPy arrays:

>>>

In [4]: arr_row = np.array( [[1, 2, 3]] )
In [5]: arr_row.shape
Out[5]: (1, 3)

In this code, you are creating a row array or vector by using a nested list. Input line 4 is passing [[1, 2, 3]] to array(). You can break out the formatting of this nested list to see how it looks:

[
    [1, 2, 3]
]

As you can see, there is one row in this nested list with three columns. On input line 5, you are displaying the shape of this array. As expected, the shape is (1, 3), or one row with three columns.

Finally, you can create a column array by including three nested lists in the input:

>>>

In [6]: arr_col = np.array( [[1], [2], [3]] )
In [7]: arr_col.shape
Out[7]: (3, 1)

In this code, input line 6 is passing [[1], [2], [3]] to the array constructor. You can break out the formatting of this nested list to see how it looks:

[
    [1],
    [2],
    [3]
]

As you can see, there are three rows in this nested list with one column each. On input line 7, you are displaying the shape of this array. As expected, the shape is (3, 1), or three rows with one column.

Since the general N-element vector has no sense of rows and columns, NumPy is able to shape the vector in whatever way makes sense for the operation being performed. You saw this in the last section, where the NumPy array did not need to be transposed to perform the scalar product, whereas the MATLAB array did need to be transposed.

Trying to take the transpose of the N-element vector does not change the shape of the array. You can take the transpose using either np.transpose() or the .T attribute of the array:

>>>

In [8]: arr_vec_transp = np.transpose(arr_vec)
In [9]: arr_vec_transp.shape
Out[9]: (3,)

In this code, you are taking the transpose of the N-element vector arr_vec and printing its shape. Notice that the shape is the same as the shape of the original arr_vec.

However, if you are using row-vectors and column-vectors, you will need to ensure that the dimensions are appropriate for the particular operation. For instance, trying to take the scalar product of the row vector with itself will result in an error:

>>>

In [10]: arr_row @ arr_row
Traceback (most recent call last):

  File "<ipython-input-10-2b447c0bc8d5>", line 1, in <module>
    arr_row @ arr_row

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,
with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 3)

In this code, trying to find the scalar product of the row vector with itself results in a ValueError informing you that the dimensions of the arrays are not aligned. Using dot() gives the same error but a slightly different message:

>>>

In [11]: np.dot(arr_row, arr_row)
Traceback (most recent call last):

  File "<ipython-input-11-d6e191b317ae>", line 1, in <module>
    np.dot(arr_row, arr_row)

ValueError: shapes (1,3) and (1,3) not aligned: 3 (dim 1) != 1 (dim 0)

In this code, you are using dot() from the np namespace to attempt to find the scalar product of two 1x3 row-vectors. Since this operation is not permitted, NumPy raises a ValueError, similar to the matrix multiplication operator.

Instead, you need to take the transpose of one of the arguments:

>>>

In [12]: arr_row.T
Out[12]:
array([[1],
       [2],
       [3]])
In [13]: sc_prod = arr_row @ arr_row.T
In [14]: sc_prod
Out[14]: array([[14]])

On input line 12, you are taking the transpose of the row vector to turn it into a column vector using the transpose attribute (.T). This is shown in corresponding output line, where the elements are arranged to form a column for printing purposes. Then, you are taking the scalar product of the vector with its transpose, producing an array with a single value, 14. Notice that this is a 1x1 array, so to access just the value, you need to access the first element in each dimension:

>>>

In [15]: sc_prod.shape
Out[15]: (1, 1)
In [16]: sc_prod[0, 0]
Out[16]: 14

In this code, you are verifying that the shape is 1x1, and then accessing the first element in each dimension located at the 0th index. Remember that Python uses 0 as the first index, not 1.

You can use the nested lists to create arrays of any shape that you want. To create a three-by-three array (two-dimensional), simply include three elements in each of your three nested lists:

>>>

In [17]: arr_2d = np.array( [[1, 2, 3], [4, 5, 6], [7, 8, 9]] )
In [18]: arr_2d.shape
Out[18]: (3, 3)

In this code, you have nested three lists with three elements each into the constructor. As shown by the shape, this produces a 3x3 array with the elements 1 through 9.

Creating Arrays Is Very Flexible in NumPy

MATLAB and NumPy both allow you to explicitly specify the specific elements in an array, as you have seen in the previous section. In addition to this direct creation of arrays, both MATLAB and NumPy support a number of other methods to create arrays without explicitly specifying each element. The NumPy project maintains a detailed list of the equivalent functions between MATLAB and NumPy.

Many functions operate identically between MATLAB and NumPy. This includes commonly used functions like linspace() and logspace() to generate evenly spaced data and ones() and zeros() to generate arrays of a given shape filled with ones and zeros, respectively. The full list of ways to create arrays in NumPy is listed in the official documentation.

The one big difference between MATLAB and NumPy in terms of array creation routines is that MATLAB supports simply using the colon to create an array, while NumPy does not. Instead, NumPy uses arange() to create an array between specified values.

In MATLAB, you can use a colon to create an array specification range. In general, you can use up to 2 colons in a specification. The syntax is as follows:

start : stop
start : step : stop

In this syntax, the, first method only uses one colon and specifies the start and stop values. The second method includes a second colon, where the value before the first colon is the start, the middle value is the step, and the last value is the stop.

Try out these examples to experiment with this syntax:

>>>

>> arr_1 = 1:6

arr_1 =

     1     2     3     4     5     6

>> size(arr_1)

ans =

     1     6

In this example, you are using the single colon with the start and stop to generate an array with the values from 1 to 6. You can see that when the step is omitted, it defaults to a value of 1. Notice that MATLAB includes both the start and the stop values in the array, and that the size of the array is 6 elements long. Next, change the value of the step size to create a new array:

>>>

>> arr_2 = 1:2:6

arr_2 =

     1     3     5

In this example, you are using the two colons syntax with the start, step, and stop. The start value is 1, the step is 2, and the stop value is 6, so MATLAB starts with 1, increments to 3, and then to 5. The next step would exceed the stop value, so MATLAB does not include the stop value in the array. Next, change the starting value to create another new array:

>>>

>> arr_3 = 2:2:6

arr_3 =

     2     4     6

In this example, you are again using the two colon method, but you are specifying the start value as 2 instead of 1. In this case, MATLAB starts at 2, increments to 4, increments to 6, and then has reached the stop value so does not go further. Notice that in this case, the the stop value of 6 is included in the array.

With NumPy, you can use arange() to create an array with specific start, stop, and step values. However, arange() has one big difference from MATLAB, which is that the stop value is not included in the resulting array. The reason for this is so that the size of the array is equal to stop - start for the default case of a step size of 1. Notice in MATLAB that the size of the array of the integers from 1 to 6 is 6, but 6 - 1 = 5.

There are three ways to use arange():

import numpy as np
np.arange(stop)
np.arange(start, stop)
np.arange(start, stop, step)

If you only pass one argument to arange(), it will be interpreted as the stop value. The start value defaults to 0 and the step defaults to 1. If you pass two arguments to arange(), they are interpreted as the start and stop values. Finally, you can pass all three of start, stop, and step to arange().

Notice that the order of the arguments is different from MATLAB, going start, stop, step in Python. If you’re having trouble remembering the order that these arguments go, remember that you can use keyword arguments in Python to be explicit about what each argument means.

You can try out arange() with the following examples:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.arange(1, 7)
In [3]: arr_1
Out[3]: array([1, 2, 3, 4, 5, 6])
In [4]: arr_1.shape
Out[4]: (6,)

In this example, you are creating an array that contains the values from 1 to 6. As in MATLAB, if the step is omitted, it defaults to 1. Notice that you had to pass the stop value 7 so that the array stopped at 6. However, the size of the resulting array is 7 - 1 = 6 elements long. Next, you should see how to change the step size:

>>>

In [5]: arr_2 = np.arange(1, 7, 2)
In [6]: arr_2
Out[6]: array([1, 3, 5])

In this code, you are creating an array that contains the values from 1 to 6, incrementing by two between each element. The step is two, so NumPy starts with 1, increments to 3, and then to 5. The next step would equal the stop value, but NumPy does not include the stop value in the array. Notice that the formula to compute the size of the array is a little bit different, since the step size is not 1.

With step sizes other than 1, the size of the array can be computed by (stop - start)/step if this results in an integer value. In this case, the size of the array is (7 - 1)/2 = 3 elements, as expected. If (stop - start)/step results in a floating point number, the size of the array is equal to the next largest integer as demonstrated in the next example:

>>>

In [7]: arr_3 = np.arange(2, 7, 2)
In [8]: arr_3
Out[8]: array([2, 4, 6])

In this example, you are creating an array that contains the values from 2 to 6, incrementing by two between each element. The step is two, so NumPy starts with 2, increments to 4, and then to 6. The next step would exceed the stop value, so NumPy stops at 6. Notice that the size of the array is (7 - 2)/2 = 2.5, so the next highest integer is 3 elements, as expected.

Finally, you should usually use integer arguments to arange() in NumPy and the colon operator in MATLAB. If you use floating point values (numbers with decimals), especially for the step, the elements may not come out exactly as you expect. If you want to use floating point numbers, linspace() is a better choice in general.

The Colon Operator Is Very Powerful in NumPy

In MATLAB, the colon operator is used to perform a number of useful tasks. As you saw, it can be used to create arrays, and it can also be used to index or slice arrays. When indexing arrays, MATLAB supports the end keyword to extend the specified range to the end of that dimension, as you saw earlier:

>>>

>> arr_1 = 1:2:6;
>> arr_1(2:end)

ans =

     3     5

In this code, you are indexing arr_1 starting at the second index and going to the end of the array. You can also specify a specific index as the stop value:

>>>

>> arr_2 = 1:6;
>> arr_2(2:4)

ans =

     2     3     4

In this code, you are creating an array arr_2 with the numbers 1 through 6, inclusive. Then, you are specifying the second element as the start value and the fourth element as the stop value in the slice. MATLAB supports the two-colon increment syntax when indexing as well:

>>>

>> arr_2(2:2:end)

ans =

     2     4     6

In this code, you are indexing the array, starting at the second element, skipping every other element, until the end of the array. You can also use end as the starting point of the slice with a negative step:

>>>

>> arr_2(end:-1:4)

ans =

     6     5     4

In this code, you are indexing arr_2 starting from the last value, decrementing by 1, and ending at the 4th element. Finally, you can slice all of the element in a dimension by using just a bare colon:

>>>

>> arr_2(:)

ans =

     1     2     3     4     5     6

In this code, you are selecting all of the first dimension of the array using just the colon.

NumPy and Python in general also use the colon for the slice syntax, but the order of the values is slightly different. In Python, the order is start : stop : step, whereas in MATLAB, it is start : step : stop, as you saw earlier. In addition, in NumPy you can omit start or stop and they will have default a value of 0 (or the first element) for start and the last element for stop. In MATLAB, you must specify start and stop if you want to specify either of them. Thus, Python does not have the end keyword, since you can omit stop to achieve the same behavior.

Try out the following examples of the slice syntax in NumPy:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.arange(1, 7, 2)
In [3]: arr_1[1:]
Out[3]: array([3, 5])

In this code, you are creating an array with the integers from 1 to 6, inclusive, skipping every other number. Then, you are slicing the array taking the second element (index 1) until the end of the array. Notice that the stop value was omitted, so it defaulted to the last element in the array.

You can also specify a specific element as the stop value. You saw in using arange() that the array did not include the stop value. The same is true of the slice syntax in Python, the slice will include everything up to, but not including, the stop index:

>>>

In [4]: arr_2 = np.arange(1, 7)
In [5]: arr_2[1:4]
Out[5]: array([2, 3, 4])

In this code, you are creating an array with the integers from 1 to 6, inclusive. Then, you are slicing the array starting at the second element (index 1, value 2) until the fourth element (index 3, value 4). However, you specified the stop index as 4 (the fifth element in the array, value 5). The reason Python includes up to the (stop - 1) index is the same reason arange() does not include the stop value, so that the length of the resulting array is equal to stop - start. Next, try changing the step of the slice:

>>>

In [6]: arr_2[1::2]
Out[6]: array([2, 4, 6])

In this code, you are slicing the array starting at the second element (index 1), going until the end of the array, and taking every second element. This results in an array with the values 2, 4, and 6. Notice that the stop value was omitted in the slice syntax, so it defaulted to the last element in the array.

You can also use a negative step in the slicing syntax for Python:

>>>

In [7]: arr_2[:2:-1]
Out[7]: array([6, 5, 4])

In this code, you are not specifying the start index of the slice, you are specifying the stop value should be index 2, and the step should be -1. Since the start index is not specified and the step is negative, the start value is assumed to be the last element in the array (or the first element in the reversed array). For the stop value, index 2 has the value of 3 and one index before that (in the reversed array) is index 3 with the value of 4.

Finally, just like in MATLAB, a bare colon means to select all of the elements from that dimension:

>>>

In [8]: arr_2[:]
Out[8]: array([1, 2, 3, 4, 5, 6])

Array Slices Are Views of Arrays in NumPy

In MATLAB, when you access a slice of an array and assign it to a variable, MATLAB will make a copy of that portion of the array into your new variable. This means that when you assign values to the slice, the original array is not affected. Try out this example to help explain the differences of MATLAB vs Python:

>>>

>> arr_1 = [1,2,3;4,5,6;7,8,9];
>> arr_2 = arr_1(2:end,2:end);

arr_2 =

     5     6
     8     9

>> arr_2(1, 1) = 10

arr_2 =

    10     6
     8     9

>> arr_1

arr_1 =

     1     2     3
     4     5     6
     7     8     9

In this code, you have created a 3x3 array arr_1 storing the values from 1 through 9. Then, you create a 2x2 slice of the original array storing from the second value to the end in both dimensions, arr_2. On the third input line, you assign the value 10 to the upper left element in arr_2. Finally, you print arr_1 again to verify that none of the values in arr_1 have changed.

Technical detail: MATLAB employs a copy-on-write memory management system, where an array may only be copied to a new memory location when it is modified. You can read more about MATLAB memory management in Memory Management for Functions and Variables on the Mathworks blog and in Internal Matlab memory optimizations.

In NumPy, slices of arrays are views to the original array. This behavior saves memory and time, since the values in the array don’t have to be copied to a new location. However, it means that changes that you make to a slice from an array will change the original array. You should try the following code to see how this works:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8 ,9]])
In [3]: arr_2 = arr_1[1:, 1:]
In [4]: arr_2
Out[4]:
array([[5, 6],
       [8, 9]])

In this code, you are creating a 3x3 array arr_1 storing the values from 1 through 9. Then, you create a 2x2 slice of the original array storing from the second value to the end in both dimensions, arr_2. Notice that the Python indexing is 0-based, so the second element has the index 1. Finally, you are printing arr_2 to verify that it is a 2x2 array.

Now you should see what happens when you change a value in arr_2. Like in the MATLAB example, you should change the upper left element of arr_2:

>>>

In [5]: arr_2[0, 0] = 10
In [6]: arr_2
Out[6]:
array([[10,  6],
       [ 8,  9]])
In [7]: arr_1
Out[7]:
array([[ 1,  2,  3],
       [ 4, 10,  6],
       [ 7,  8,  9]])

In this code, you first assign the upper left element in arr_2, at index (0, 0) to have a value of 10. Then you print arr_2 to verify that the appropriate value has changed. Finally, you print arr_1 and see that the value in the middle of the array has changed from 5 to 10!

This is what is meant by arr_2 being a view of arr_1. Since it is a view, arr_2 points to the same memory location as arr_1, so updating arr_2 also updates arr_1 because the value stored in the memory location accessed by both arr_2 and arr_1 has been updated. This also goes the other direction, where changing values in arr_1 will update the value in arr_2:

>>>

In [8]: arr_1[-1, -1] = 42
In [9]: arr_1
Out[9]:
array([[ 1,  2,  3],
       [ 4, 10,  6],
       [ 7,  8, 42]])
In [10]: arr_2
Out[10]:
array([[10,  6],
       [ 8, 42]])

In this code, you are assigning the bottom right element of arr_1 to have the value 42. Remember that in Python, an index of -1 means the last value on that dimension. Then you are printing arr_1 to verify that the lower right value has changed from 9 to 42. Finally, you are printing arr_2, and you see for arr_2 as well, the bottom right value has changed from 9 to 42.

If you want to generate a copy of an array, you can use np.copy(). Copying an array creates a new place in memory for the copy to be stored, so changes to the copied array do not affect the original:

>>>

In [11]: arr_3 = np.copy(arr_2)
In [12]: arr_3[1, 0] = 37
In [13]: arr_3
Out[13]:
array([[10,  6],
       [37, 42]])
In [14]: arr_2
Out[14]:
array([[10,  6],
       [ 8, 42]])

In this code, you are creating arr_3 as a copy of arr_2. Then, you are changing the element in the second row, first column to have the value of 37. Then, you are printing arr_3 to verify that the specified change has been made. Finally, you are printing arr_2 to verify that no changes have occurred in arr_2, as expected.

Tips and Tricks to Make Your Code Pythonic

Like any other programming language, Python code written by experienced Python developers often has a particular look and feel to it. This is because they are able to take advantage of specific idioms in Python to work with Python rather than against Python. Developers coming from other languages often miss out on what makes code Pythonic in their first projects.

In this section, you’ll learn some tips and tricks to make your code Pythonic and level up your Python skills. There are many more tips and tricks than you can learn here, so feel free to check out Write More Pythonic Code.

You Should Not Use Semicolons to End Lines in Python

In MATLAB, ending a line of code with a semicolon ; suppresses the output from that line. For instance, assigning a variable will print the value of the variable after the assignment if the semicolon is omitted.

In Python, you should not end lines of code with semicolons. It is unnecessary, since Python does not change its behavior whether the line is ended with a semicolon or not. So you can save yourself a key stroke and not bother including the semicolon in your scripts and libraries.

There is one case in Python where the semicolon is useful. When you want to execute several statements, but you cannot include a newline character in the input, you can separate the statements with semicolons. This is mostly useful to execute very short scripts from the command prompt or terminal. For instance, to find the particular Python executable that is running, you can type the following:

$ python -c "import sys; print(sys.executable)"
/home/eleanor/anaconda3/bin/python

In this code, you are executing the Python interpreter in the python executable and passing the -c switch. This switch takes the next argument and executes it within the interpreter. Since the shell environment would execute if you pressed Enter to insert a new line, you can type the whole script on one line.

In this case, you have two logical statements that need to be separated by the semicolon. First you are importing the built-in sys library and then you are printing the value of sys.executable. In this example, the Python interpreter that the shell is running comes from the /home/eleanor/anaconda3/bin/python file.

You Should Not Import `*` From a Module in Python

In a previous section, you read about how namespaces are one honking great idea in Python. In MATLAB, all functions are part of the global namespace by default, so every function and class name has to be unique. Python solves this problem by using namespaces and requiring you to specify which module a function should come from.

You will find tutorials around the Web that suggest you write the following:

from a_module import *

In this code, you are using the * to indicate that Python should import everything that is contained in a_module and put it in the current scope without a prefix. This is slightly more convenient, because you no longer have to prefix functions and classes from a_module with anything, you can just use them directly. However, it is not a good practice because you don’t know what names are defined in a_module and whether or not they will override any existing names in your current scope.

Technical Note: When you from a_module import *, Python imports all of the names listed in a special variable called __all__ in a_module. However, if that variable is not defined, Python will import all of the variables, functions, and classes defined in a_module.

You Should Take Advantage of the Different Data Types in Python

MATLAB, with its heritage as a linear algebra and array focused language, treats most data types as arrays of some sort. This often makes it a little bit tricky to work with more advanced data types like structs, containers.Map, cell arrays, and more.

Python has several built-in data types that are very flexible and can be used to accomplish a number of useful tasks. The major ones that you’ll learn about in this section are lists and dictionaries.

Lists

Python lists are mutable sequences of values. Lists can contain heterogeneous data, which means that each element of the list can be of a different type. Because lists are mutable, you can change the value of any element in the list, or add or remove values from the list, without creating a new list object.

Since lists are sequences, you can create loops that iterate over them. In Python, you do not need to access each element of a list with an index in a for loop, as you would do in MATLAB:

>>>

>> arr_1 = 1:2:6;
>> for i = 1:length(arr_1)
    disp(arr_1(i))
end
1
3
5

In this code, you are creating an array arr_1 with the integers from 1 to 6, taking every other number. Then you are creating a for loop where the loop variable goes from 1 to the length of arr_1. Finally, you are displaying the value of the element of arr_1 at the loop variable on each step by using the loop variable i to index arr_1.

In Python, you should not use an index for the list when you loop over it. Instead, you should loop directly over the items in a list:

>>>

In [1]: lst_1 = [1, "b", 3.0]
In [2]: for item in lst_1:
   ...:     print(item)
   ...:
1
b
3.0

In this code, on input line 1 you are first creating a Python list with three elements:

The integer 1
The string "b"
The float 3.0

This list is assigned to lst_1. Then you are using a for loop to access each item in the list in turn. On each iteration, the next value in the list is put into the variable item that you specified on the for line. Then, you are printing the value of item on each iteration.

Notice in the previous example that you could loop over the value of each element in the list without using an index. Nonetheless, sometimes you want to access the index of each item in the list as you’re looping over it. For those cases, Python provides enumerate() that returns the index and the value of the item:

>>>

In [3]: for index, item in enumerate(lst_1):
   ...:     print(f"The index is {index} and the item is {item}")
   ...:
The index is 0 and the item is 1
The index is 1 and the item is b
The index is 2 and the item is 3.0

In this code, you are looping over lst_1 again, but this time, you are using enumerate() to get both the index and the item. Then you are printing the value of the index and item on each loop iteration. As you can see from the result, the index values start at 0 as expected, but you do not need to use the index to access the item from the list.

In summary, you should not write Python code like this:

for i in range(len(lst_1)):
    print(lst_1[i])

In this code, you are creating a range of integers from 0 to the length of lst_1 and then you are accessing each element in the list by its index. This can lead to off-by-one and fencepost errors. Instead, you should write code that loops over the list directly:

for item in lst_1:
    print(item)

You can read a lot more about lists in Lists and Tuples in Python and about for loops and iteration in Python “for” Loops (Definite Iteration). There is also a more advanced concept called list comprehensions that you can learn about in Using List Comprehensions Effectively.

Dictionaries

In MATLAB, you can create a map data type with containers.Map(). This kind of data structure is useful when you have two pieces of data that are always related to each other and you want to connect them together. For instance, you can map cities to their population with a containers.Map():

>>>

>> cities = containers.Map({'Cleveland';'Pittsburgh';'Hartford'},
                           [383793,301048,122587]);
>> cities('Cleveland')

ans =

     383793

In this code, you are creating a containers.Map() on the first line. The first argument is a cell array of character arrays with the city names. These are called the keys of the map. The second argument is an array of populations. These are called the values of the map. Then, you are accessing the value of the population in Cleveland by indexing the map with a character array.

You can assign new values into the map by assigning to an undefined key value:

>>>

>> cities('Providence') = 180393;

If you try to access a key that does not exist, you will receive an error message:

>>>

>> cities('New York City')
Error using containers.Map/subsref
The specified key is not present in this container.

Python has an equivalent data structure called a dictionary. To create a Python dictionary, you can use curly braces and specify the keys and values with each other:

>>>

In [1]: cities = {
   ...: "Cleveland": 383_793,
   ...: "Pittsburgh": 301_048,
   ...: "Hartford": 122_587}
In [2]: cities["Cleveland"]
Out[2]: 383793

In this code, on input line 1 you are creating the dictionary of cities using curly braces. Notice that the key and value are specified together, separated by a colon. The values are specified with _ in the numbers, a feature available since Python 3.6. This does not change the value of the number, it only makes it easier to read very large numbers. Then, you are accessing the value at the "Cleveland" key using square brackets, similar to the indexing syntax for lists and arrays.

You can add new keys to the dictionary by assigning to them:

>>>

In [3]: cities["Providence"] = 180_393

In this code, you assigned a new key "Providence" to the dictionary with a value of 180,393. If you try to access a key that is not in the dictionary, you will get a KeyError:

>>>

In [4]: cities["New York City"]
Traceback (most recent call last):

  File "<ipython-input-4-6ebe5b35f3ea>", line 1, in <module>
    cities["New York City"]

KeyError: 'New York City'

In this code, you are trying to access the dictionary using the "New York City" key. However, this key does not exist in the dictionary so Python raises a KeyError letting you know that "New York City" is not an option.

You can read a lot more about Python dictionaries in Dictionaries in Python and KeyError exceptions in Python KeyError Exceptions and How to Handle Them. You can also iterate through dictionaries and use dictionary comprehensions, similar to list comprehensions. You can read about these topics in How to Iterate Through a Dictionary in Python.

Exceptions Help You Control Program Flow in Python

MATLAB and Python both use errors and exceptions to let you know when something has gone wrong in the code. In this section, you will learn about common exceptions in Python and how you can handle them appropriately.

If you want an introduction to Python exceptions overall, you can read Python Exceptions: An Introduction. When a Python exception is raised, it produces a traceback. You can read about how to interpret the traceback in Understanding Python Tracebacks. Understanding tracebacks is very helpful to interpret and correct Python exceptions in general. There are a few specific cases that usually have the same resolution. You’ll see those described in the rest of this section.

NameError

Python NameError exceptions are usually the result of a variable being undefined. When you see a NameError, check your code for typos and misspelled variable names. You can use the debugging features and the variable explorer in Spyder to find out which variables are defined.

SyntaxError

Python SyntaxError exceptions mean that you have input some improper syntax. This is usually caused by mismatched brackets, when you have only the opening bracket or closing bracket but not the matching one. These exceptions usually point to the line after the place where the problem is located.

Another common SyntaxError is using only one equals sign in an if statement. In this case, you either meant not-equals (!=) or equals (==), so you can correct the line.

KeyError

Python KeyError exceptions occur when you try to access a key in a dictionary that does not exist. You can use .get() to retrieve a key from a dictionary if it exists, or return a default value if the key does not exist. You can read more about KeyError exceptions in Python KeyError Exceptions and How to Handle Them.

IndexError

Python IndexError exceptions occur when you are trying to access the index of an array or list that does not exist. This usually means the array or list you are trying to access has fewer elements than the index you are trying to access. You can use the debugging features and the variable explorer in Spyder to see the size of lists and arrays and make sure you’re only accessing indices that exist.

ImportError/ModuleNotFoundError

Python ImportError and ModuleNotFoundError exceptions occur when you try to import a module that Python cannot find. This might be because it is installed in a different conda environment or virtualenv, or it might be because you forgot to install the package.

The solution to this error is usually to conda install or pip install the package and make sure the correct environment is activated. If you’re not using the base environment in conda, you also need to make sure to install Spyder or Jupyter into your environment.

TypeError/ValueError

Python TypeError exceptions happen when an argument is of the wrong type. This happens most commonly when you pass an argument of the wrong type into a function. For instance, a function that works with numbers would raise a TypeError if a string were passed in.

A related exception is the ValueError. This exception happens when an argument is of the correct type, but has an incorrect value. For instance, a function that works only with positive numbers would raise a ValueError if a negative number were passed in.

AttributeError

Python AttributeError exceptions happen when you try to access an attribute of an object when the object does not have that attribute. You will often see this error associated with the message NoneType object has no attribute. This message most likely means that a function has returned None instead of the object you were expecting and you are trying to access an attribute that would be present on the real object, but is not defined for None.

How to Handle Exceptions in Python

MATLAB allows you to try a statement of code and catch any errors that are thrown by the code. Once you have caught an error, you can do further processing of the error and assign variables based on the type of error. The MATLAB documentation has several good examples of how this would look in MATLAB.

In Python, one big difference from MATLAB is that you can choose to catch only certain types of exceptions and handle them. This allows all other exceptions to continue to be displayed to the user. If you want to learn more about how to do this in Python, you can read The try and except Block: Handling Exceptions.

To see how this works, you can try the following example:

>>>

In [1]: import math
In [2]: def my_sqrt(number):
   ...:    print(f"You passed the argument: {number!r}")
   ...:    try:
   ...:        return math.sqrt(number)
   ...:    except ValueError:
   ...:        print("You passed a number that cannot be operated on")
   ...:    except TypeError:
   ...:        print("You passed an argument that was not a number")
   ...:

In [3]: my_sqrt(4.0)
You passed the argument: 4.0
Out[3]: 2.0

In [4]: my_sqrt(-1.0)
You passed the argument: -1.0
You passed a number that cannot be operated on

In [4]: my_sqrt("4.0")
You passed the argument: '4.0'
You passed an argument that was not a number

In this code, on input line 1 you are importing the built-in math library. Then, starting on input line 2, you are defining a function called my_sqrt() that will take one argument, called number. Inside the function definition, you first print the argument that the user passed.

Next, you enter the try/except block. First, you try to take the square root of the input argument and return the result. If taking the square root of the argument results in an error, Python will catch that error and check which type of error was raised.

You have defined code that handles two specific exceptions: ValueError and TypeError. If math.sqrt() raises a ValueError, your code will print a message that the number cannot be operated on. If math.sqrt() raises a TypeError, your code will print a message that the argument was not a number. If any other type of exception is raised by math.sqrt(), that error will be passed through without any processing, since there is no handler for any other error types.

More specifically, Python checks for any error that is raised by the code in the try block. In your case, you only defined one line of code in the try block, but this is not required, and you can have as many lines as you want there. However, it is usually a good practice to minimize the number of lines of code in the try block so you can be very specific about which code is raising any errors.

On input line 3, you are testing out my_sqrt(). First, you pass the value 4.0 to the function. The function prints the argument, and math.sqrt() has no problems taking the square root of 4, resulting in 2.0 on the output line.

On input line 4, you are passing -1.0 as the argument to my_sqrt(). As you may recall, taking the square root of negative numbers results in a complex number, which the math.sqrt() function is not equipped to handle. Taking the square root of a negative number using math.sqrt() raises a ValueError. You exception handler catches this ValueError and prints the message that the number cannot be operated on.

On input line 5, you are passing "4.0" as the argument to my_sqrt(). In this case, math.sqrt() does not know how to take the square root of a string, even though that string appears to represent a number. You can see that you have passed a string by the quotes in the statement giving the value of the argument: You passed the argument: '4.0'. Since math.sqrt() cannot take the square root of a string, it raises a TypeError, and your function prints the message that the argument was not a number.

There Is an Official Guide to Writing Good Code in Python

The Python community has developed a set of recommendations for how to style your Python code. These are codified in a document called PEP 8, which stands for Python Enhancement Proposal #8. PEP 8 can be found in full on the Python website. You can also learn more about good Python style in How to Write Beautiful Python Code With PEP 8 and Idiomatic Python 101.

Perhaps the most important principle in PEP 8 is the saying that “a foolish consistency is the hobgoblin of little minds.” This means that you should follow the recommendations in PEP 8 for almost all of your code, but there may be some limited cases where it is a good idea not to follow the PEP 8 recommendations. For instance, if you are working with an existing codebase that has its own style, you should follow that style where it diverges from PEP 8. You can see an excellent discussion of this principle from Raymond Hettinger, one of the core Python developers, in a talk from PyCon 2015.

Aside from reading PEP 8, you can use a few Python packages to automatically make sure that your code fits the style guidelines. Flake8 is a code linter that reads your code and makes suggestions for how you can improve it. This is similar to the feature in the MATLAB code editor that makes improvement suggestions. In addition, packages such as Black, yapf, and autopep8 will automatically format your code to be compliant with PEP 8 or your own style rules. Using these packages can help your code feel more Pythonic and help you learn good Python style!

Python Has a Fantastic and Supportive Community

Python is known for having a very supportive, open, and welcoming community. Whether you are a brand new developer or an experienced one, whether you’re brand new to Python or have been to a dozen conferences, the community is there to support you and what you want to do.

The community starts with the Python Package Index (called PyPI or the CheeseShop, a reference to the Monty Python sketch), which houses hundreds of thousands of different Python packages that you can download for free. These packages can be installed using pip, a package manager that comes bundled with Python. This means that adding the functionality you need to Python can be as simple as pip install package or if you’re using Anaconda, conda install package.

Since Python is used in so many different areas of software development, data science, science, and engineering, there are always people around who want to talk about Python. Most large cities around the world have Python meetup groups. You can go to these groups to learn about Python by hearing people speak about their work or work on some open-source code.

A few times a year, these groups coalesce into different PyCons which happen on every continent around the globe. PyCon North America is the largest of these, with several thousand attendees every year. You can read all about what it’s like to attend in How to Get the Most Out of PyCon.

Python also has a very strong online community. If you have a question about programming in Python, you can ask on StackOverflow and some of the world-leading Python experts will be able to help you out. Make sure to follow the instructions for how to ask a question on StackOverflow. Remember that the more effort you put into your question, the more likely you are to either find the answer yourself (hooray!) or get a good answer from someone else.

If you want to follow along with Python’s development, you can sign up for one of the mailing lists covering different aspects of Python’s community. The general mailing list for asking questions about writing programs in Python is called comp.lang.python. If you are interested in the development of Python itself, you can follow the python-dev mailing list.

If you’re interested in learning much, much more about Python development, you can check out the Real Python Learning Paths!

Areas Where You Should Still Use MATLAB®

In spite of the awesome community and terrific packages, there are still one or two areas where MATLAB works better than Python. The main place where Python can’t compete with MATLAB is the Simulink Toolbox. This toolbox offers advanced capabilities for signal processing and modeling in a convenient graphical interface.

Python does not have an equivalent graphical interface to these kinds of functions. However, to the extent that Simulink is a convenient interface to ordinary differential equation solvers, Python has equivalent solvers as in MATLAB and the underlying functionality of Simulink can certainly be replicated in Python.

Otherwise, you can do anything in Python that you can do in MATLAB! If you can think of work that you can do with MATLAB, but you’re not sure how to do it in Python, let us know in the comments and we’ll be able to help with suggestions.

Conclusion

Congratulations, you now have the knowledge you need to switch your MATLAB code to Python! In this article, you learned a little bit about what Python is, how to set up your computer to use Python, and how to convert your code from MATLAB to Python.

Python is a really huge language and community, with lots to learn and lots of people to learn from. Remember, you weren’t a MATLAB expert the first time you opened the MATLAB development environment, and the same is true about the first time you write some Python code. Come back to this article as often as you need to improve your skills and learn more about becoming a Python wizard!

Python args and kwargs: Demystified

2019-09-04T14:00:00+00:00

Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: *args and **kwargs. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in main(), then this article is for you. You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.

By the end of the article, you’ll know:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

This article assumes that you already know how to define Python functions and work with lists and dictionaries.

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Passing Multiple Arguments to a Function

*args and **kwargs allow you to pass multiple arguments or keyword arguments to a function. Consider the following example. This is a simple function that takes two arguments and returns their sum:

def my_sum(a, b):
    return a + b

This function works fine, but it’s limited to only two arguments. What if you need to sum a varying number of arguments, where the specific number of arguments passed is only determined at runtime? Wouldn’t it be great to create a function that could sum all the integers passed to it, no matter how many there are?

Using the Python args Variable in Function Definitions

There are a few ways you can pass a varying number of arguments to a function. The first way is often the most intuitive for people that have experience with collections. You simply pass a list or a set of all the arguments to your function. So for my_sum(), you could pass a list of all the integers you need to add:

# sum_integers_list.py
def my_sum(my_integers):
    result = 0
    for x in my_integers:
        result += x
    return result

list_of_integers = [1, 2, 3]
print(my_sum(list_of_integers))

This implementation works, but whenever you call this function you’ll also need to create a list of arguments to pass to it. This can be inconvenient, especially if you don’t know up front all the values that should go into the list.

This is where *args can be really useful, because it allows you to pass a varying number of positional arguments. Take the following example:

# sum_integers_args.py
def my_sum(*args):
    result = 0
    # Iterating over the Python args tuple
    for x in args:
        result += x
    return result

print(my_sum(1, 2, 3))

In this example, you’re no longer passing a list to my_sum(). Instead, you’re passing three different positional arguments. my_sum() takes all the parameters that are provided in the input and packs them all into a single iterable object named args.

Note that args is just a name. You’re not required to use the name args. You can choose any name that you prefer, such as integers:

# sum_integers_args_2.py
def my_sum(*integers):
    result = 0
    for x in integers:
        result += x
    return result

print(my_sum(1, 2, 3))

The function still works, even if you pass the iterable object as integers instead of args. All that matters here is that you use the unpacking operator (*).

Bear in mind that the iterable object you’ll get using the unpacking operator * is not a list but a tuple. A tuple is similar to a list in that they both support slicing and iteration. However, tuples are very different in at least one aspect: lists are mutable, while tuples are not. To test this, run the following code. This script tries to change a value of a list:

# change_list.py
my_list = [1, 2, 3]
my_list[0] = 9
print(my_list)

The value located at the very first index of the list should be updated to 9. If you execute this script, you will see that the list indeed gets modified:

$ python change_list.py
[9, 2, 3]

The first value is no longer 0, but the updated value 9. Now, try to do the same with a tuple:

# change_tuple.py
my_tuple = (1, 2, 3)
my_tuple[0] = 9
print(my_tuple)

Here, you see the same values, except they’re held together as a tuple. If you try to execute this script, you will see that the Python interpreter returns an error:

$ python change_tuple.py
Traceback (most recent call last):
  File "change_tuple.py", line 3, in <module>
    my_tuple[0] = 9
TypeError: 'tuple' object does not support item assignment

This is because a tuple is an immutable object, and its values cannot be changed after assignment. Keep this in mind when you’re working with tuples and *args.

Using the Python kwargs Variable in Function Definitions

Okay, now you’ve understood what *args is for, but what about **kwargs? **kwargs works just like *args, but instead of accepting positional arguments it accepts keyword (or named) arguments. Take the following example:

# concatenate.py
def concatenate(**kwargs):
    result = ""
    # Iterating over the Python kwargs dictionary
    for arg in kwargs.values():
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

When you execute the script above, concatenate() will iterate through the Python kwargs dictionary and concatenate all the values it finds:

$ python concatenate.py
RealPythonIsGreat!

Like args, kwargs is just a name that can be changed to whatever you want. Again, what is important here is the use of the unpacking operator (**).

So, the previous example could be written like this:

# concatenate_2.py
def concatenate(**words):
    result = ""
    for arg in words.values():
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

Note that in the example above the iterable object is a standard dict. If you iterate over the dictionary and want to return its values, like in the example shown, then you must use .values().

In fact, if you forget to use this method, you will find yourself iterating through the keys of your Python kwargs dictionary instead, like in the following example:

# concatenate_keys.py
def concatenate(**kwargs):
    result = ""
    # Iterating over the keys of the Python kwargs dictionary
    for arg in kwargs:
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

Now, if you try to execute this example, you’ll notice the following output:

$ python concatenate_keys.py
abcde

As you can see, if you don’t specify .values(), your function will iterate over the keys of your Python kwargs dictionary, returning the wrong result.

Ordering Arguments in a Function

Now that you have learned what *args and **kwargs are for, you are ready to start writing functions that take a varying number of input arguments. But what if you want to create a function that takes a changeable number of both positional and named arguments?

In this case, you have to bear in mind that order counts. Just as non-default arguments have to precede default arguments, so *args must come before **kwargs.

To recap, the correct order for your parameters is:

Standard arguments
*args arguments
**kwargs arguments

For example, this function definition is correct:

# correct_function_definition.py
def my_function(a, b, *args, **kwargs):
    pass

The *args variable is appropriately listed before **kwargs. But what if you try to modify the order of the arguments? For example, consider the following function:

# wrong_function_definition.py
def my_function(a, b, **kwargs, *args):
    pass

Now, **kwargs comes before *args in the function definition. If you try to run this example, you’ll receive an error from the interpreter:

$ python wrong_function_definition.py
  File "wrong_function_definition.py", line 2
    def my_function(a, b, **kwargs, *args):
                                    ^
SyntaxError: invalid syntax

In this case, since *args comes after **kwargs, the Python interpreter throws a SyntaxError.

Unpacking With the Asterisk Operators: `*` & `**`

You are now able to use *args and **kwargs to define Python functions that take a varying number of input arguments. Let’s go a little deeper to understand something more about the unpacking operators.

The single and double asterisk unpacking operators were introduced in Python 2. As of the 3.5 release, they have become even more powerful, thanks to PEP 448. In short, the unpacking operators are operators that unpack the values from iterable objects in Python. The single asterisk operator * can be used on any iterable that Python provides, while the double asterisk operator ** can only be used on dictionaries.

Let’s start with an example:

# print_list.py
my_list = [1, 2, 3]
print(my_list)

This code defines a list and then prints it to the standard output:

$ python print_list.py
[1, 2, 3]

Note how the list is printed, along with the corresponding brackets and commas.

Now, try to prepend the unpacking operator * to the name of your list:

# print_unpacked_list.py
my_list = [1, 2, 3]
print(*my_list)

Here, the * operator tells print() to unpack the list first.

In this case, the output is no longer the list itself, but rather the content of the list:

$ python print_unpacked_list.py
1 2 3

Can you see the difference between this execution and the one from print_list.py? Instead of a list, print() has taken three separate arguments as the input.

Another thing you’ll notice is that in print_unpacked_list.py, you used the unpacking operator * to call a function, instead of in a function definition. In this case, print() takes all the items of a list as though they were single arguments.

You can also use this method to call your own functions, but if your function requires a specific number of arguments, then the iterable you unpack must have the same number of arguments.

To test this behavior, consider this script:

# unpacking_call.py
def my_sum(a, b, c):
    print(a + b + c)

my_list = [1, 2, 3]
my_sum(*my_list)

Here, my_sum() explicitly states that a, b, and c are required arguments.

If you run this script, you’ll get the sum of the three numbers in my_list:

$ python unpacking_call.py
6

The 3 elements in my_list match up perfectly with the required arguments in my_sum().

Now look at the following script, where my_list has 4 arguments instead of 3:

# wrong_unpacking_call.py
def my_sum(a, b, c):
    print(a + b + c)

my_list = [1, 2, 3, 4]
my_sum(*my_list)

In this example, my_sum() still expects just three arguments, but the * operator gets 4 items from the list. If you try to execute this script, you’ll see that the Python interpreter is unable to run it:

$ python wrong_unpacking_call.py
Traceback (most recent call last):
  File "wrong_unpacking_call.py", line 6, in <module>
    my_sum(*my_list)
TypeError: my_sum() takes 3 positional arguments but 4 were given

When you use the * operator to unpack a list and pass arguments to a function, it’s exactly as though you’re passing every single argument alone. This means that you can use multiple unpacking operators to get values from several lists and pass them all to a single function.

To test this behavior, consider the following example:

# sum_integers_args_3.py
def my_sum(*args):
    result = 0
    for x in args:
        result += x
    return result

list1 = [1, 2, 3]
list2 = [4, 5]
list3 = [6, 7, 8, 9]

print(my_sum(*list1, *list2, *list3))

If you run this example, all three lists are unpacked. Each individual item is passed to my_sum(), resulting in the following output:

$ python sum_integers_args_3.py
45

There are other convenient uses of the unpacking operator. For example, say you need to split a list into three different parts. The output should show the first value, the last value, and all the values in between. With the unpacking operator, you can do this in just one line of code:

# extract_list_body.py
my_list = [1, 2, 3, 4, 5, 6]

a, *b, c = my_list

print(a)
print(b)
print(c)

In this example, my_list contains 6 items. The first variable is assigned to a, the last to c, and all other values are packed into a new list b. If you run the script, print() will show you that your three variables have the values you would expect:

$ python extract_list_body.py
1
[2, 3, 4, 5]
6

Another interesting thing you can do with the unpacking operator * is to split the items of any iterable object. This could be very useful if you need to merge two lists, for instance:

# merging_lists.py
my_first_list = [1, 2, 3]
my_second_list = [4, 5, 6]
my_merged_list = [*my_first_list, *my_second_list]

print(my_merged_list)

The unpacking operator * is prepended to both my_first_list and my_second_list.

If you run this script, you’ll see that the result is a merged list:

$ python merging_lists.py
[1, 2, 3, 4, 5, 6]

You can even merge two different dictionaries by using the unpacking operator **:

# merging_dicts.py
my_first_dict = {"A": 1, "B": 2}
my_second_dict = {"C": 3, "D": 4}
my_merged_dict = {**my_first_dict, **my_second_dict}

print(my_merged_dict)

Here, the iterables to merge are my_first_dict and my_second_dict.

Executing this code outputs a merged dictionary:

$ python merging_dicts.py
{'A': 1, 'B': 2, 'C': 3, 'D': 4}

Remember that the * operator works on any iterable object. It can also be used to unpack a string:

# string_to_list.py
a = [*"RealPython"]
print(a)

In Python, strings are iterable objects, so * will unpack it and place all individual values in a list a:

$ python string_to_list.py
['R', 'e', 'a', 'l', 'P', 'y', 't', 'h', 'o', 'n']

The previous example seems great, but when you work with these operators it’s important to keep in mind the seventh rule of The Zen of Python by Tim Peters: Readability counts.

To see why, consider the following example:

# mysterious_statement.py
*a, = "RealPython"
print(a)

There’s the unpacking operator *, followed by a variable, a comma, and an assignment. That’s a lot packed into one line! In fact, this code is no different from the previous example. It just takes the string RealPython and assigns all the items to the new list a, thanks to the unpacking operator *.

The comma after the a does the trick. When you use the unpacking operator with variable assignment, Python requires that your resulting variable is either a list or a tuple. With the trailing comma, you have actually defined a tuple with just one named variable a.

While this is a neat trick, many Pythonistas would not consider this code to be very readable. As such, it’s best to use these kinds of constructions sparingly.

Conclusion

You are now able to use *args and **kwargs to accept a changeable number of arguments in your functions. You have also learned something more about the unpacking operators.

You’ve learned:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

If you still have questions, don’t hesitate to reach out in the comments section below! To learn more about the use of the asterisks in Python, have a look at Trey Hunner’s article on the subject.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Lists and Tuples in Python

2019-09-03T14:00:00+00:00

In this course, you’ll learn about working with lists and tuples. Lists and tuples are arguably Python’s most versatile, useful data types. You’ll find them in virtually every non-trivial Python program.

Here’s what you’ll learn in this tutorial: You’ll cover the important characteristics of lists and tuples. You’ll learn how to define them and how to manipulate them. When you’re finished, you’ll have a good feel for when and how to use these object types in a Python program.

Take the Quiz: Test your knowledge with our interactive “Python Lists and Tuples” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Natural Language Processing With spaCy in Python

2019-09-02T14:00:00+00:00

spaCy is a free and open-source library for Natural Language Processing (NLP) in Python with a lot of in-built capabilities. It’s becoming increasingly popular for processing and analyzing data in NLP. Unstructured textual data is produced at a large scale, and it’s important to process and derive insights from unstructured data. To do that, you need to represent the data in a format that can be understood by computers. NLP can help you do that.

In this tutorial, you’ll learn:

What the foundational terms and concepts in NLP are
How to implement those concepts in spaCy
How to customize and extend built-in functionalities in spaCy
How to perform basic statistical analysis on a text
How to create a pipeline to process unstructured text
How to parse a sentence and extract meaningful insights from it

What Are NLP and spaCy?

NLP is a subfield of Artificial Intelligence and is concerned with interactions between computers and human languages. NLP is the process of analyzing, understanding, and deriving meaning from human languages for computers.

NLP helps you extract insights from unstructured text and has several use cases, such as:

spaCy is a free, open-source library for NLP in Python. It’s written in Cython and is designed to build information extraction or natural language understanding systems. It’s built for production use and provides a concise and user-friendly API.

Installation

In this section, you’ll install spaCy and then download data and models for the English language.

How to Install spaCy

spaCy can be installed using pip, a Python package manager. You can use a virtual environment to avoid depending on system-wide packages. To learn more about virtual environments and pip, check out What Is Pip? A Guide for New Pythonistas and Python Virtual Environments: A Primer.

Create a new virtual environment:

$ python3 -m venv env

Activate this virtual environment and install spaCy:

$ source ./env/bin/activate
$ pip install spacy

How to Download Models and Data

spaCy has different types of models. The default model for the English language is en_core_web_sm.

Activate the virtual environment created in the previous step and download models and data for the English language:

$ python -m spacy download en_core_web_sm

Verify if the download was successful or not by loading it:

>>>

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')

If the nlp object is created, then it means that spaCy was installed and that models and data were successfully downloaded.

Using spaCy

In this section, you’ll use spaCy for a given input string and a text file. Load the language model instance in spaCy:

>>>

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')

Here, the nlp object is a language model instance. You can assume that, throughout this tutorial, nlp refers to the language model loaded by en_core_web_sm. Now you can use spaCy to read a string or a text file.

How to Read a String

You can use spaCy to create a processed Doc object, which is a container for accessing linguistic annotations, for a given input string:

>>>

>>> introduction_text = ('This tutorial is about Natural'
...     ' Language Processing in Spacy.')
>>> introduction_doc = nlp(introduction_text)
>>> # Extract tokens for the given doc
>>> print ([token.text for token in introduction_doc])
['This', 'tutorial', 'is', 'about', 'Natural', 'Language',
'Processing', 'in', 'Spacy', '.']

In the above example, notice how the text is converted to an object that is understood by spaCy. You can use this method to convert any text into a processed Doc object and deduce attributes, which will be covered in the coming sections.

How to Read a Text File

In this section, you’ll create a processed Doc object for a text file:

>>>

>>> file_name = 'introduction.txt'
>>> introduction_file_text = open(file_name).read()
>>> introduction_file_doc = nlp(introduction_file_text)
>>> # Extract tokens for the given doc
>>> print ([token.text for token in introduction_file_doc])
['This', 'tutorial', 'is', 'about', 'Natural', 'Language',
'Processing', 'in', 'Spacy', '.', '\n']

This is how you can convert a text file into a processed Doc object.

Note:

You can assume that:

Variable names ending with the suffix _text are Unicode string objects.
Variable name ending with the suffix _doc are spaCy’s language model objects.

Sentence Detection

Sentence Detection is the process of locating the start and end of sentences in a given text. This allows you to you divide a text into linguistically meaningful units. You’ll use these units when you’re processing your text to perform tasks such as part of speech tagging and entity extraction.

In spaCy, the sents property is used to extract sentences. Here’s how you would extract the total number of sentences and the sentences for a given input text:

>>>

>>> about_text = ('Gus Proto is a Python developer currently'
...               ' working for a London-based Fintech'
...               ' company. He is interested in learning'
...               ' Natural Language Processing.')
>>> about_doc = nlp(about_text)
>>> sentences = list(about_doc.sents)
>>> len(sentences)
2
>>> for sentence in sentences:
...     print (sentence)
...
'Gus Proto is a Python developer currently working for a
London-based Fintech company.'
'He is interested in learning Natural Language Processing.'

In the above example, spaCy is correctly able to identify sentences in the English language, using a full stop(.) as the sentence delimiter. You can also customize the sentence detection to detect sentences on custom delimiters.

Here’s an example, where an ellipsis(...) is used as the delimiter:

>>>

>>> def set_custom_boundaries(doc):
...     # Adds support to use `...` as the delimiter for sentence detection
...     for token in doc[:-1]:
...         if token.text == '...':
...             doc[token.i+1].is_sent_start = True
...     return doc
...
>>> ellipsis_text = ('Gus, can you, ... never mind, I forgot'
...                  ' what I was saying. So, do you think'
...                  ' we should ...')
>>> # Load a new model instance
>>> custom_nlp = spacy.load('en_core_web_sm')
>>> custom_nlp.add_pipe(set_custom_boundaries, before='parser')
>>> custom_ellipsis_doc = custom_nlp(ellipsis_text)
>>> custom_ellipsis_sentences = list(custom_ellipsis_doc.sents)
>>> for sentence in custom_ellipsis_sentences:
...     print(sentence)
...
Gus, can you, ...
never mind, I forgot what I was saying.
So, do you think we should ...
>>> # Sentence Detection with no customization
>>> ellipsis_doc = nlp(ellipsis_text)
>>> ellipsis_sentences = list(ellipsis_doc.sents)
>>> for sentence in ellipsis_sentences:
...     print(sentence)
...
Gus, can you, ... never mind, I forgot what I was saying.
So, do you think we should ...

Note that custom_ellipsis_sentences contain three sentences, whereas ellipsis_sentences contains two sentences. These sentences are still obtained via the sents attribute, as you saw before.

Tokenization in spaCy

Tokenization is the next step after sentence detection. It allows you to identify the basic units in your text. These basic units are called tokens. Tokenization is useful because it breaks a text into meaningful units. These units are used for further analysis, like part of speech tagging.

In spaCy, you can print tokens by iterating on the Doc object:

>>>

>>> for token in about_doc:
...     print (token, token.idx)
...
Gus 0
Proto 4
is 10
a 13
Python 15
developer 22
currently 32
working 42
for 50
a 54
London 56
- 62
based 63
Fintech 69
company 77
. 84
He 86
is 89
interested 92
in 103
learning 106
Natural 115
Language 123
Processing 132
. 142

Note how spaCy preserves the starting index of the tokens. It’s useful for in-place word replacement. spaCy provides various attributes for the Token class:

>>>

>>> for token in about_doc:
...     print (token, token.idx, token.text_with_ws,
...            token.is_alpha, token.is_punct, token.is_space,
...            token.shape_, token.is_stop)
...
Gus 0 Gus  True False False Xxx False
Proto 4 Proto  True False False Xxxxx False
is 10 is  True False False xx True
a 13 a  True False False x True
Python 15 Python  True False False Xxxxx False
developer 22 developer  True False False xxxx False
currently 32 currently  True False False xxxx False
working 42 working  True False False xxxx False
for 50 for  True False False xxx True
a 54 a  True False False x True
London 56 London True False False Xxxxx False
- 62 - False True False - False
based 63 based  True False False xxxx False
Fintech 69 Fintech  True False False Xxxxx False
company 77 company True False False xxxx False
. 84 .  False True False . False
He 86 He  True False False Xx True
is 89 is  True False False xx True
interested 92 interested  True False False xxxx False
in 103 in  True False False xx True
learning 106 learning  True False False xxxx False
Natural 115 Natural  True False False Xxxxx False
Language 123 Language  True False False Xxxxx False
Processing 132 Processing True False False Xxxxx False
. 142 . False True False . False

In this example, some of the commonly required attributes are accessed:

text_with_ws prints token text with trailing space (if present).
is_alpha detects if the token consists of alphabetic characters or not.
is_punct detects if the token is a punctuation symbol or not.
is_space detects if the token is a space or not.
shape_ prints out the shape of the word.
is_stop detects if the token is a stop word or not.

Note: You’ll learn more about stop words in the next section.

You can also customize the tokenization process to detect tokens on custom characters. This is often used for hyphenated words, which are words joined with hyphen. For example, “London-based” is a hyphenated word.

spaCy allows you to customize tokenization by updating the tokenizer property on the nlp object:

>>>

>>> import re
>>> import spacy
>>> from spacy.tokenizer import Tokenizer
>>> custom_nlp = spacy.load('en_core_web_sm')
>>> prefix_re = spacy.util.compile_prefix_regex(custom_nlp.Defaults.prefixes)
>>> suffix_re = spacy.util.compile_suffix_regex(custom_nlp.Defaults.suffixes)
>>> infix_re = re.compile(r'''[-~]''')
>>> def customize_tokenizer(nlp):
...     # Adds support to use `-` as the delimiter for tokenization
...     return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
...                      suffix_search=suffix_re.search,
...                      infix_finditer=infix_re.finditer,
...                      token_match=None
...                      )
...

>>> custom_nlp.tokenizer = customize_tokenizer(custom_nlp)
>>> custom_tokenizer_about_doc = custom_nlp(about_text)
>>> print([token.text for token in custom_tokenizer_about_doc])
['Gus', 'Proto', 'is', 'a', 'Python', 'developer', 'currently',
'working', 'for', 'a', 'London', '-', 'based', 'Fintech',
'company', '.', 'He', 'is', 'interested', 'in', 'learning',
'Natural', 'Language', 'Processing', '.']

In order for you to customize, you can pass various parameters to the Tokenizer class:

nlp.vocab is a storage container for special cases and is used to handle cases like contractions and emoticons.
prefix_search is the function that is used to handle preceding punctuation, such as opening parentheses.
infix_finditer is the function that is used to handle non-whitespace separators, such as hyphens.
suffix_search is the function that is used to handle succeeding punctuation, such as closing parentheses.
token_match is an optional boolean function that is used to match strings that should never be split. It overrides the previous rules and is useful for entities like URLs or numbers.

Note: spaCy already detects hyphenated words as individual tokens. The above code is just an example to show how tokenization can be customized. It can be used for any other character.

Stop Words

Stop words are the most common words in a language. In the English language, some examples of stop words are the, are, but, and they. Most sentences need to contain stop words in order to be full sentences that make sense.

Generally, stop words are removed because they aren’t significant and distort the word frequency analysis. spaCy has a list of stop words for the English language:

>>>

>>> import spacy
>>> spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS
>>> len(spacy_stopwords)
326
>>> for stop_word in list(spacy_stopwords)[:10]:
...     print(stop_word)
...
using
becomes
had
itself
once
often
is
herein
who
too

You can remove stop words from the input text:

>>>

>>> for token in about_doc:
...     if not token.is_stop:
...         print (token)
...
Gus
Proto
Python
developer
currently
working
London
-
based
Fintech
company
.
interested
learning
Natural
Language
Processing
.

Stop words like is, a, for, the, and in are not printed in the output above. You can also create a list of tokens not containing stop words:

>>>

>>> about_no_stopword_doc = [token for token in about_doc if not token.is_stop]
>>> print (about_no_stopword_doc)
[Gus, Proto, Python, developer, currently, working, London,
-, based, Fintech, company, ., interested, learning, Natural,
Language, Processing, .]

about_no_stopword_doc can be joined with spaces to form a sentence with no stop words.

Lemmatization

Lemmatization is the process of reducing inflected forms of a word while still ensuring that the reduced form belongs to the language. This reduced form or root word is called a lemma.

For example, organizes, organized and organizing are all forms of organize. Here, organize is the lemma. The inflection of a word allows you to express different grammatical categories like tense (organized vs organize), number (trains vs train), and so on. Lemmatization is necessary because it helps you reduce the inflected forms of a word so that they can be analyzed as a single item. It can also help you normalize the text.

spaCy has the attribute lemma_ on the Token class. This attribute has the lemmatized form of a token:

>>>

>>> conference_help_text = ('Gus is helping organize a developer'
...     'conference on Applications of Natural Language'
...     ' Processing. He keeps organizing local Python meetups'
...     ' and several internal talks at his workplace.')
>>> conference_help_doc = nlp(conference_help_text)
>>> for token in conference_help_doc:
...     print (token, token.lemma_)
...
Gus Gus
is be
helping help
organize organize
a a
developer developer
conference conference
on on
Applications Applications
of of
Natural Natural
Language Language
Processing Processing
. .
He -PRON-
keeps keep
organizing organize
local local
Python Python
meetups meetup
and and
several several
internal internal
talks talk
at at
his -PRON-
workplace workplace
. .

In this example, organizing reduces to its lemma form organize. If you do not lemmatize the text, then organize and organizing will be counted as different tokens, even though they both have a similar meaning. Lemmatization helps you avoid duplicate words that have similar meanings.

Word Frequency

You can now convert a given text into tokens and perform statistical analysis over it. This analysis can give you various insights about word patterns, such as common words or unique words in the text:

>>>

>>> from collections import Counter
>>> complete_text = ('Gus Proto is a Python developer currently'
...     'working for a London-based Fintech company. He is'
...     ' interested in learning Natural Language Processing.'
...     ' There is a developer conference happening on 21 July'
...     ' 2019 in London. It is titled "Applications of Natural'
...     ' Language Processing". There is a helpline number '
...     ' available at +1-1234567891. Gus is helping organize it.'
...     ' He keeps organizing local Python meetups and several'
...     ' internal talks at his workplace. Gus is also presenting'
...     ' a talk. The talk will introduce the reader about "Use'
...     ' cases of Natural Language Processing in Fintech".'
...     ' Apart from his work, he is very passionate about music.'
...     ' Gus is learning to play the Piano. He has enrolled '
...     ' himself in the weekend batch of Great Piano Academy.'
...     ' Great Piano Academy is situated in Mayfair or the City'
...     ' of London and has world-class piano instructors.')
...
>>> complete_doc = nlp(complete_text)
>>> # Remove stop words and punctuation symbols
>>> words = [token.text for token in complete_doc
...          if not token.is_stop and not token.is_punct]
>>> word_freq = Counter(words)
>>> # 5 commonly occurring words with their frequencies
>>> common_words = word_freq.most_common(5)
>>> print (common_words)
[('Gus', 4), ('London', 3), ('Natural', 3), ('Language', 3), ('Processing', 3)]
>>> # Unique words
>>> unique_words = [word for (word, freq) in word_freq.items() if freq == 1]
>>> print (unique_words)
['Proto', 'currently', 'working', 'based', 'company',
'interested', 'conference', 'happening', '21', 'July',
'2019', 'titled', 'Applications', 'helpline', 'number',
'available', '+1', '1234567891', 'helping', 'organize',
'keeps', 'organizing', 'local', 'meetups', 'internal',
'talks', 'workplace', 'presenting', 'introduce', 'reader',
'Use', 'cases', 'Apart', 'work', 'passionate', 'music', 'play',
'enrolled', 'weekend', 'batch', 'situated', 'Mayfair', 'City',
'world', 'class', 'piano', 'instructors']

By looking at the common words, you can see that the text as a whole is probably about Gus, London, or Natural Language Processing. This way, you can take any unstructured text and perform statistical analysis to know what it’s about.

Here’s another example of the same text with stop words:

>>>

>>> words_all = [token.text for token in complete_doc if not token.is_punct]
>>> word_freq_all = Counter(words_all)
>>> # 5 commonly occurring words with their frequencies
>>> common_words_all = word_freq_all.most_common(5)
>>> print (common_words_all)
[('is', 10), ('a', 5), ('in', 5), ('Gus', 4), ('of', 4)]

Four out of five of the most common words are stop words, which don’t tell you much about the text. If you consider stop words while doing word frequency analysis, then you won’t be able to derive meaningful insights from the input text. This is why removing stop words is so important.

Part of Speech Tagging

Part of speech or POS is a grammatical role that explains how a particular word is used in a sentence. There are eight parts of speech:

Noun
Pronoun
Adjective
Verb
Adverb
Preposition
Conjunction
Interjection

Part of speech tagging is the process of assigning a POS tag to each token depending on its usage in the sentence. POS tags are useful for assigning a syntactic category like noun or verb to each word.

In spaCy, POS tags are available as an attribute on the Token object:

>>>

>>> for token in about_doc:
...     print (token, token.tag_, token.pos_, spacy.explain(token.tag_))
...
Gus NNP PROPN noun, proper singular
Proto NNP PROPN noun, proper singular
is VBZ VERB verb, 3rd person singular present
a DT DET determiner
Python NNP PROPN noun, proper singular
developer NN NOUN noun, singular or mass
currently RB ADV adverb
working VBG VERB verb, gerund or present participle
for IN ADP conjunction, subordinating or preposition
a DT DET determiner
London NNP PROPN noun, proper singular
- HYPH PUNCT punctuation mark, hyphen
based VBN VERB verb, past participle
Fintech NNP PROPN noun, proper singular
company NN NOUN noun, singular or mass
. . PUNCT punctuation mark, sentence closer
He PRP PRON pronoun, personal
is VBZ VERB verb, 3rd person singular present
interested JJ ADJ adjective
in IN ADP conjunction, subordinating or preposition
learning VBG VERB verb, gerund or present participle
Natural NNP PROPN noun, proper singular
Language NNP PROPN noun, proper singular
Processing NNP PROPN noun, proper singular
. . PUNCT punctuation mark, sentence closer

Here, two attributes of the Token class are accessed:

tag_ lists the fine-grained part of speech.
pos_ lists the coarse-grained part of speech.

spacy.explain gives descriptive details about a particular POS tag. spaCy provides a complete tag list along with an explanation for each tag.

Using POS tags, you can extract a particular category of words:

>>>

>>> nouns = []
>>> adjectives = []
>>> for token in about_doc:
...     if token.pos_ == 'NOUN':
...         nouns.append(token)
...     if token.pos_ == 'ADJ':
...         adjectives.append(token)
...
>>> nouns
[developer, company]
>>> adjectives
[interested]

You can use this to derive insights, remove the most common nouns, or see which adjectives are used for a particular noun.

Visualization: Using displaCy

spaCy comes with a built-in visualizer called displaCy. You can use it to visualize a dependency parse or named entities in a browser or a Jupyter notebook.

You can use displaCy to find POS tags for tokens:

>>>

>>> from spacy import displacy
>>> about_interest_text = ('He is interested in learning'
...     ' Natural Language Processing.')
>>> about_interest_doc = nlp(about_interest_text)
>>> displacy.serve(about_interest_doc, style='dep')

The above code will spin a simple web server. You can see the visualization by opening http://127.0.0.1:5000 in your browser:

displaCy: Part of Speech Tagging Demo

In the image above, each token is assigned a POS tag written just below the token.

Note: Here’s how you can use displaCy in a Jupyter notebook:

>>>

>>> displacy.render(about_interest_doc, style='dep', jupyter=True)

Preprocessing Functions

You can create a preprocessing function that takes text as input and applies the following operations:

Lowercases the text
Lemmatizes each token
Removes punctuation symbols
Removes stop words

A preprocessing function converts text to an analyzable format. It’s necessary for most NLP tasks. Here’s an example:

>>>

>>> def is_token_allowed(token):
...     '''
...         Only allow valid tokens which are not stop words
...         and punctuation symbols.
...     '''
...     if (not token or not token.string.strip() or
...         token.is_stop or token.is_punct):
...         return False
...     return True
...
>>> def preprocess_token(token):
...     # Reduce token to its lowercase lemma form
...     return token.lemma_.strip().lower()
...
>>> complete_filtered_tokens = [preprocess_token(token)
...     for token in complete_doc if is_token_allowed(token)]
>>> complete_filtered_tokens
['gus', 'proto', 'python', 'developer', 'currently', 'work',
'london', 'base', 'fintech', 'company', 'interested', 'learn',
'natural', 'language', 'processing', 'developer', 'conference',
'happen', '21', 'july', '2019', 'london', 'title',
'applications', 'natural', 'language', 'processing', 'helpline',
'number', 'available', '+1', '1234567891', 'gus', 'help',
'organize', 'keep', 'organize', 'local', 'python', 'meetup',
'internal', 'talk', 'workplace', 'gus', 'present', 'talk', 'talk',
'introduce', 'reader', 'use', 'case', 'natural', 'language',
'processing', 'fintech', 'apart', 'work', 'passionate', 'music',
'gus', 'learn', 'play', 'piano', 'enrol', 'weekend', 'batch',
'great', 'piano', 'academy', 'great', 'piano', 'academy',
'situate', 'mayfair', 'city', 'london', 'world', 'class',
'piano', 'instructor']

Note that the complete_filtered_tokens does not contain any stop word or punctuation symbols and consists of lemmatized lowercase tokens.

Rule-Based Matching Using spaCy

Rule-based matching is one of the steps in extracting information from unstructured text. It’s used to identify and extract tokens and phrases according to patterns (such as lowercase) and grammatical features (such as part of speech).

Rule-based matching can use regular expressions to extract entities (such as phone numbers) from an unstructured text. It’s different from extracting text using regular expressions only in the sense that regular expressions don’t consider the lexical and grammatical attributes of the text.

With rule-based matching, you can extract a first name and a last name, which are always proper nouns:

>>>

>>> from spacy.matcher import Matcher
>>> matcher = Matcher(nlp.vocab)
>>> def extract_full_name(nlp_doc):
...     pattern = [{'POS': 'PROPN'}, {'POS': 'PROPN'}]
...     matcher.add('FULL_NAME', None, pattern)
...     matches = matcher(nlp_doc)
...     for match_id, start, end in matches:
...         span = nlp_doc[start:end]
...         return span.text
...
>>> extract_full_name(about_doc)
'Gus Proto'

In this example, pattern is a list of objects that defines the combination of tokens to be matched. Both POS tags in it are PROPN (proper noun). So, the pattern consists of two objects in which the POS tags for both tokens should be PROPN. This pattern is then added to Matcher using FULL_NAME and the the match_id. Finally, matches are obtained with their starting and end indexes.

You can also use rule-based matching to extract phone numbers:

>>>

>>> from spacy.matcher import Matcher
>>> matcher = Matcher(nlp.vocab)
>>> conference_org_text = ('There is a developer conference'
...     'happening on 21 July 2019 in London. It is titled'
...     ' "Applications of Natural Language Processing".'
...     ' There is a helpline number available'
...     ' at (123) 456-789')
...
>>> def extract_phone_number(nlp_doc):
...     pattern = [{'ORTH': '('}, {'SHAPE': 'ddd'},
...                {'ORTH': ')'}, {'SHAPE': 'ddd'},
...                {'ORTH': '-', 'OP': '?'},
...                {'SHAPE': 'ddd'}]
...     matcher.add('PHONE_NUMBER', None, pattern)
...     matches = matcher(nlp_doc)
...     for match_id, start, end in matches:
...         span = nlp_doc[start:end]
...         return span.text
...
>>> conference_org_doc = nlp(conference_org_text)
>>> extract_phone_number(conference_org_doc)
'(123) 456-789'

In this example, only the pattern is updated in order to match phone numbers from the previous example. Here, some attributes of the token are also used:

ORTH gives the exact text of the token.
SHAPE transforms the token string to show orthographic features.
OP defines operators. Using ? as a value means that the pattern is optional, meaning it can match 0 or 1 times.

Note: For simplicity, phone numbers are assumed to be of a particular format: (123) 456-789. You can change this depending on your use case.

Rule-based matching helps you identify and extract tokens and phrases according to lexical patterns (such as lowercase) and grammatical features(such as part of speech).

Dependency Parsing Using spaCy

Dependency parsing is the process of extracting the dependency parse of a sentence to represent its grammatical structure. It defines the dependency relationship between headwords and their dependents. The head of a sentence has no dependency and is called the root of the sentence. The verb is usually the head of the sentence. All other words are linked to the headword.

The dependencies can be mapped in a directed graph representation:

Words are the nodes.
The grammatical relationships are the edges.

Dependency parsing helps you know what role a word plays in the text and how different words relate to each other. It’s also used in shallow parsing and named entity recognition.

Here’s how you can use dependency parsing to see the relationships between words:

>>>

>>> piano_text = 'Gus is learning piano'
>>> piano_doc = nlp(piano_text)
>>> for token in piano_doc:
...     print (token.text, token.tag_, token.head.text, token.dep_)
...
Gus NNP learning nsubj
is VBZ learning aux
learning VBG learning ROOT
piano NN learning dobj

In this example, the sentence contains three relationships:

nsubj is the subject of the word. Its headword is a verb.
aux is an auxiliary word. Its headword is a verb.
dobj is the direct object of the verb. Its headword is a verb.

There is a detailed list of relationships with descriptions. You can use displaCy to visualize the dependency tree:

>>>

>>> displacy.serve(piano_doc, style='dep')

This code will produce a visualization that can be accessed by opening http://127.0.0.1:5000 in your browser:

displaCy: Dependency Parse Demo

This image shows you that the subject of the sentence is the proper noun Gus and that it has a learn relationship with piano.

Navigating the Tree and Subtree

The dependency parse tree has all the properties of a tree. This tree contains information about sentence structure and grammar and can be traversed in different ways to extract relationships.

spaCy provides attributes like children, lefts, rights, and subtree to navigate the parse tree:

>>>

>>> one_line_about_text = ('Gus Proto is a Python developer'
...     ' currently working for a London-based Fintech company')
>>> one_line_about_doc = nlp(one_line_about_text)
>>> # Extract children of `developer`
>>> print([token.text for token in one_line_about_doc[5].children])
['a', 'Python', 'working']
>>> # Extract previous neighboring node of `developer`
>>> print (one_line_about_doc[5].nbor(-1))
Python
>>> # Extract next neighboring node of `developer`
>>> print (one_line_about_doc[5].nbor())
currently
>>> # Extract all tokens on the left of `developer`
>>> print([token.text for token in one_line_about_doc[5].lefts])
['a', 'Python']
>>> # Extract tokens on the right of `developer`
>>> print([token.text for token in one_line_about_doc[5].rights])
['working']
>>> # Print subtree of `developer`
>>> print (list(one_line_about_doc[5].subtree))
[a, Python, developer, currently, working, for, a, London, -,
based, Fintech, company]

You can construct a function that takes a subtree as an argument and returns a string by merging words in it:

>>>

>>> def flatten_tree(tree):
...     return ''.join([token.text_with_ws for token in list(tree)]).strip()
...
>>> # Print flattened subtree of `developer`
>>> print (flatten_tree(one_line_about_doc[5].subtree))
a Python developer currently working for a London-based Fintech company

You can use this function to print all the tokens in a subtree.

Shallow Parsing

Shallow parsing, or chunking, is the process of extracting phrases from unstructured text. Chunking groups adjacent tokens into phrases on the basis of their POS tags. There are some standard well-known chunks such as noun phrases, verb phrases, and prepositional phrases.

Noun Phrase Detection

A noun phrase is a phrase that has a noun as its head. It could also include other kinds of words, such as adjectives, ordinals, determiners. Noun phrases are useful for explaining the context of the sentence. They help you infer what is being talked about in the sentence.

spaCy has the property noun_chunks on Doc object. You can use it to extract noun phrases:

>>>

>>> conference_text = ('There is a developer conference'
...     ' happening on 21 July 2019 in London.')
>>> conference_doc = nlp(conference_text)
>>> # Extract Noun Phrases
>>> for chunk in conference_doc.noun_chunks:
...     print (chunk)
...
a developer conference
21 July
London

By looking at noun phrases, you can get information about your text. For example, a developer conference indicates that the text mentions a conference, while the date 21 July lets you know that conference is scheduled for 21 July. You can figure out whether the conference is in the past or the future. London tells you that the conference is in London.

Verb Phrase Detection

A verb phrase is a syntactic unit composed of at least one verb. This verb can be followed by other chunks, such as noun phrases. Verb phrases are useful for understanding the actions that nouns are involved in.

spaCy has no built-in functionality to extract verb phrases, so you’ll need a library called textacy:

Note:

You can use pip to install textacy:

$ pip install textacy

Now that you have textacy installed, you can use it to extract verb phrases based on grammar rules:

>>>

>>> import textacy
>>> about_talk_text = ('The talk will introduce reader about Use'
...                    ' cases of Natural Language Processing in'
...                    ' Fintech')
>>> pattern = r'(<VERB>?<ADV>*<VERB>+)'
>>> about_talk_doc = textacy.make_spacy_doc(about_talk_text,
...                                         lang='en_core_web_sm')
>>> verb_phrases = textacy.extract.pos_regex_matches(about_talk_doc, pattern)
>>> # Print all Verb Phrase
>>> for chunk in verb_phrases:
...     print(chunk.text)
...
will introduce
>>> # Extract Noun Phrase to explain what nouns are involved
>>> for chunk in about_talk_doc.noun_chunks:
...     print (chunk)
...
The talk
reader
Use cases
Natural Language Processing
Fintech

In this example, the verb phrase introduce indicates that something will be introduced. By looking at noun phrases, you can see that there is a talk that will introduce the reader to use cases of Natural Language Processing or Fintech.

The above code extracts all the verb phrases using a regular expression pattern of POS tags. You can tweak the pattern for verb phrases depending upon your use case.

Note: In the previous example, you could have also done dependency parsing to see what the relationships between the words were.

Named Entity Recognition

Named Entity Recognition (NER) is the process of locating named entities in unstructured text and then classifying them into pre-defined categories, such as person names, organizations, locations, monetary values, percentages, time expressions, and so on.

You can use NER to know more about the meaning of your text. For example, you could use it to populate tags for a set of documents in order to improve the keyword search. You could also use it to categorize customer support tickets into relevant categories.

spaCy has the property ents on Doc objects. You can use it to extract named entities:

>>>

>>> piano_class_text = ('Great Piano Academy is situated'
...     ' in Mayfair or the City of London and has'
...     ' world-class piano instructors.')
>>> piano_class_doc = nlp(piano_class_text)
>>> for ent in piano_class_doc.ents:
...     print(ent.text, ent.start_char, ent.end_char,
...           ent.label_, spacy.explain(ent.label_))
...
Great Piano Academy 0 19 ORG Companies, agencies, institutions, etc.
Mayfair 35 42 GPE Countries, cities, states
the City of London 46 64 GPE Countries, cities, states

In the above example, ent is a Span object with various attributes:

text gives the Unicode text representation of the entity.
start_char denotes the character offset for the start of the entity.
end_char denotes the character offset for the end of the entity.
label_ gives the label of the entity.

spacy.explain gives descriptive details about an entity label. The spaCy model has a pre-trained list of entity classes. You can use displaCy to visualize these entities:

>>>

>>> displacy.serve(piano_class_doc, style='ent')

If you open http://127.0.0.1:5000 in your browser, then you can see the visualization:

displaCy: Named Entity Recognition Demo

You can use NER to redact people’s names from a text. For example, you might want to do this in order to hide personal information collected in a survey. You can use spaCy to do that:

>>>

>>> survey_text = ('Out of 5 people surveyed, James Robert,'
...                ' Julie Fuller and Benjamin Brooks like'
...                ' apples. Kelly Cox and Matthew Evans'
...                ' like oranges.')
...
>>> def replace_person_names(token):
...     if token.ent_iob != 0 and token.ent_type_ == 'PERSON':
...         return '[REDACTED] '
...     return token.string
...
>>> def redact_names(nlp_doc):
...     for ent in nlp_doc.ents:
...         ent.merge()
...     tokens = map(replace_person_names, nlp_doc)
...     return ''.join(tokens)
...
>>> survey_doc = nlp(survey_text)
>>> redact_names(survey_doc)
'Out of 5 people surveyed, [REDACTED] , [REDACTED] and'
' [REDACTED] like apples. [REDACTED] and [REDACTED]'
' like oranges.'

In this example, replace_person_names() uses ent_iob. It gives the IOB code of the named entity tag using inside-outside-beginning (IOB) tagging. Here, it can assume a value other than zero, because zero means that no entity tag is set.

Conclusion

spaCy is a powerful and advanced library that is gaining huge popularity for NLP applications due to its speed, ease of use, accuracy, and extensibility. Congratulations! You now know:

What the foundational terms and concepts in NLP are
How to implement those concepts in spaCy
How to customize and extend built-in functionalities in spaCy
How to perform basic statistical analysis on a text
How to create a pipeline to process unstructured text
How to parse a sentence and extract meaningful insights from it

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyCharm for Productive Python Development (Guide)

2019-08-28T14:00:00+00:00

As a programmer, you should be focused on the business logic and creating useful applications for your users. In doing that, PyCharm by JetBrains saves you a lot of time by taking care of the routine and by making a number of other tasks such as debugging and visualization easy.

In this article, you’ll learn about:

Installing PyCharm
Writing code in PyCharm
Running your code in PyCharm
Debugging and testing your code in PyCharm
Editing an existing project in PyCharm
Searching and navigating in PyCharm
Using Version Control in PyCharm
Using Plugins and External Tools in PyCharm
Using PyCharm Professional features, such as Django support and Scientific mode

This article assumes that you’re familiar with Python development and already have some form of Python installed on your system. Python 3.6 will be used for this tutorial. Screenshots and demos provided are for macOS. Because PyCharm runs on all major platforms, you may see slightly different UI elements and may need to modify certain commands.

Note:

PyCharm comes in three editions:

PyCharm Edu is free and for educational purposes.
PyCharm Community is free as well and intended for pure Python development.
PyCharm Professional is paid, has everything the Community edition has and also is very well suited for Web and Scientific development with support for such frameworks as Django and Flask, Database and SQL, and scientific tools such as Jupyter.

For more details on their differences, check out the PyCharm Editions Comparison Matrix by JetBrains. The company also has special offers for students, teachers, open source projects, and other cases.

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

Installing PyCharm

This article will use PyCharm Community Edition 2019.1 as it’s free and available on every major platform. Only the section about the professional features will use PyCharm Professional Edition 2019.1.

The recommended way of installing PyCharm is with the JetBrains Toolbox App. With its help, you’ll be able to install different JetBrains products or several versions of the same product, update, rollback, and easily remove any tool when necessary. You’ll also be able to quickly open any project in the right IDE and version.

To install the Toolbox App, refer to the documentation by JetBrains. It will automatically give you the right instructions depending on your OS. In case it didn’t recognize your OS correctly, you can always find it from the drop down list on the top right section:

After installing, launch the app and accept the user agreement. Under the Tools tab, you’ll see a list of available products. Find PyCharm Community there and click Install:

Voilà! You have PyCharm available on your machine. If you don’t want to use the Toolbox app, then you can also do a stand-alone installation of PyCharm.

Launch PyCharm, and you’ll see the import settings popup:

PyCharm will automatically detect that this is a fresh install and choose Do not import settings for you. Click OK, and PyCharm will ask you to select a keymap scheme. Leave the default and click Next: UI Themes on the bottom right:

PyCharm will then ask you to choose a dark theme called Darcula or a light theme. Choose whichever you prefer and click Next: Launcher Script:

I’ll be using the dark theme Darcula throughout this tutorial. You can find and install other themes as plugins, or you can also import them.

On the next page, leave the defaults and click Next: Featured plugins. There, PyCharm will show you a list of plugins you may want to install because most users like to use them. Click Start using PyCharm, and now you are ready to write some code!

Writing Code in PyCharm

In PyCharm, you do everything in the context of a project. Thus, the first thing you need to do is create one.

After installing and opening PyCharm, you are on the welcome screen. Click Create New Project, and you’ll see the New Project popup:

Specify the project location and expand the Project Interpreter drop down. Here, you have options to create a new project interpreter or reuse an existing one. Choose New environment using. Right next to it, you have a drop down list to select one of Virtualenv, Pipenv, or Conda, which are the tools that help to keep dependencies required by different projects separate by creating isolated Python environments for them.

You are free to select whichever you like, but Virtualenv is used for this tutorial. If you choose to, you can specify the environment location and choose the base interpreter from the list, which is a list of Python interpreters (such as Python2.7 and Python3.6) installed on your system. Usually, the defaults are fine. Then you have to select boxes to inherit global site-packages to your new environment and make it available to all other projects. Leave them unselected.

Click Create on the bottom right and you will see the new project created:

You will also see a small Tip of the Day popup where PyCharm gives you one trick to learn at each startup. Go ahead and close this popup.

It is now time to start a new Python program. Type Cmd+N if you are on Mac or Alt+Ins if you are on Windows or Linux. Then, choose Python File. You can also select File → New from the menu. Name the new file guess_game.py and click OK. You will see a PyCharm window similar to the following:

For our test code, let’s quickly code up a simple guessing game in which the program chooses a number that the user has to guess. For every guess, the program will tell if the user’s guess was smaller or bigger than the secret number. The game ends when the user guesses the number. Here’s the code for the game:

 1 from random import randint
 2 
 3 def play():
 4     random_int = randint(0, 100)
 5 
 6     while True:
 7         user_guess = int(input("What number did we guess (0-100)?"))
 8 
 9         if user_guess == randint:
10             print(f"You found the number ({random_int}). Congrats!")
11             break
12 
13         if user_guess < random_int:
14             print("Your number is less than the number we guessed.")
15             continue
16 
17         if user_guess > random_int:
18             print("Your number is more than the number we guessed.")
19             continue
20 
21 
22 if __name__ == '__main__':
23     play()

Type this code directly rather than copying and pasting. You’ll see something like this:

As you can see, PyCharm provides Intelligent Coding Assistance with code completion, code inspections, on-the-fly error highlighting, and quick-fix suggestions. In particular, note how when you typed main and then hit tab, PyCharm auto-completed the whole main clause for you.

Also note how, if you forget to type if before the condition, append .if, and then hit Tab, PyCharm fixes the if clause for you. The same is true with True.while. That’s PyCharm’s Postfix completions working for you to help reduce backward caret jumps.

Running Code in PyCharm

Now that you’ve coded up the game, it’s time for you to run it.

You have three ways of running this program:

Use the shortcut Ctrl+Shift+R on Mac or Ctrl+Shift+F10 on Windows or Linux.
Right-click the background and choose Run ‘guess_game’ from the menu.
Since this program has the __main__ clause, you can click on the little green arrow to the left of the __main__ clause and choose Run ‘guess_game’ from there.

Use any one of the options above to run the program, and you’ll see the Run Tool pane appear at the bottom of the window, with your code output showing:

Play the game for a little bit to see if you can find the number guessed. Pro tip: start with 50.

Debugging in PyCharm

Did you find the number? If so, you may have seen something weird after you found the number. Instead of printing the congratulations message and exiting, the program seems to start over. That’s a bug right there. To discover why the program starts over, you’ll now debug the program.

First, place a breakpoint by clicking on the blank space to the left of line number 8:

This will be the point where the program will be suspended, and you can start exploring what went wrong from there on. Next, choose one of the following three ways to start debugging:

Press Ctrl+Shift+D on Mac or Shift+Alt+F9 on Windows or Linux.
Right-click the background and choose Debug ‘guess_game’.
Click on the little green arrow to the left of the __main__ clause and choose Debug ‘guess_game from there.

Afterwards, you’ll see a Debug window open at the bottom:

Follow the steps below to debug the program:

Notice that the current line is highlighted in blue.
See that random_int and its value are listed in the Debug window. Make a note of this number. (In the picture, the number is 85.)
Hit F8 to execute the current line and step over to the next one. You can also use F7 to step into the function in the current line, if necessary. As you continue executing the statements, the changes in the variables will be automatically reflected in the Debugger window.
Notice that there is the Console tab right next to the Debugger tab that opened. This Console tab and the Debugger tab are mutually exclusive. In the Console tab, you will be interacting with your program, and in the Debugger tab you will do the debugging actions.
Switch to the Console tab to enter your guess.
Type the number shown, and then hit Enter.
Switch back to the Debugger tab.
Hit F8 again to evaluate the if statement. Notice that you are now on line 14. But wait a minute! Why didn’t it go to the line 11? The reason is that the if statement on line 10 evaluated to False. But why did it evaluate to False when you entered the number that was chosen?
Look carefully at line 10 and notice that we are comparing user_guess with the wrong thing. Instead of comparing it with random_int, we are comparing it with randint, the function that was imported from the random package.
Change it to random_int, restart the debugging, and follow the same steps again. You will see that, this time, it will go to line 11, and line 10 will evaluate to True:

Congratulations! You fixed the bug.

Testing in PyCharm

No application is reliable without unit tests. PyCharm helps you write and run them very quickly and comfortably. By default, unittest is used as the test runner, but PyCharm also supports other testing frameworks such as pytest, nose, doctest, tox, and trial. You can, for example, enable pytest for your project like this:

Open the Settings/Preferences → Tools → Python Integrated Tools settings dialog.
Select pytest in the Default test runner field.
Click OK to save the settings.

For this example, we’ll be using the default test runner unittest.

In the same project, create a file called calculator.py and put the following Calculator class in it:

 1 class Calculator:
 2     def add(self, a, b):
 3         return a + b
 4 
 5     def multiply(self, a, b):
 6         return a * b

PyCharm makes it very easy to create tests for your existing code. With the calculator.py file open, execute any one of the following that you like:

Press Shift+Cmd+T on Mac or Ctrl+Shift+T on Windows or Linux.
Right-click in the background of the class and then choose Go To and Test.
On the main menu, choose Navigate → Test.

Choose Create New Test…, and you will see the following window:

Leave the defaults of Target directory, Test file name, and Test class name. Select both of the methods and click OK. Voila! PyCharm automatically created a file called test_calculator.py and created the following stub tests for you in it:

 1 from unittest import TestCase
 2 
 3 class TestCalculator(TestCase):
 4     def test_add(self):
 5         self.fail()
 6 
 7     def test_multiply(self):
 8         self.fail()

Run the tests using one of the methods below:

Press Ctrl+R on Mac or Shift+F10 on Windows or Linux.
Right-click the background and choose Run ‘Unittests for test_calculator.py’.
Click on the little green arrow to the left of the test class name and choose Run ‘Unittests for test_calculator.py’.

You’ll see the tests window open on the bottom with all the tests failing:

Notice that you have the hierarchy of the test results on the left and the output of the terminal on the right.

Now, implement test_add by changing the code to the following:

 1 from unittest import TestCase
 2 
 3 from calculator import Calculator
 4 
 5 class TestCalculator(TestCase):
 6     def test_add(self):
 7         self.calculator = Calculator()
 8         self.assertEqual(self.calculator.add(3, 4), 7)
 9 
10     def test_multiply(self):
11         self.fail()

Run the tests again, and you’ll see that one test passed and the other failed. Explore the options to show passed tests, to show ignored tests, to sort tests alphabetically, and to sort tests by duration:

Note that the sleep(0.1) method that you see in the GIF above is intentionally used to make one of the tests slower so that sorting by duration works.

Editing an Existing Project in PyCharm

These single file projects are great for examples, but you’ll often work on much larger projects over a longer period of time. In this section, you’ll take a look at how PyCharm works with a larger project.

To explore the project-focused features of PyCharm, you’ll use the Alcazar web framework that was built for learning purposes. To continue following along, clone the repo locally:

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

Once you have a project locally, open it in PyCharm using one of the following methods:

Click File → Open on the main menu.
Click Open on the Welcome Screen if you are there.

After either of these steps, find the folder containing the project on your computer and open it.

If this project contains a virtual environment, then PyCharm will automatically use this virtual environment and make it the project interpreter.

If you need to configure a different virtualenv, then open Preferences on Mac by pressing Cmd+, or Settings on Windows or Linux by pressing Ctrl+Alt+S and find the Project: ProjectName section. Open the drop-down and choose Project Interpreter:

Choose the virtualenv from the drop-down list. If it’s not there, then click on the settings button to the right of the drop-down list and then choose Add…. The rest of the steps should be the same as when we were creating a new project.

Searching and Navigating in PyCharm

In a big project where it’s difficult for a single person to remember where everything is located, it’s very important to be able to quickly navigate and find what you looking for. PyCharm has you covered here as well. Use the project you opened in the section above to practice these shortcuts:

Searching for a fragment in the current file: Press Cmd+F on Mac or Ctrl+F on Windows or Linux.
Searching for a fragment in the entire project: Press Cmd+Shift+F on Mac or Ctrl+Shift+F on Windows or Linux.
Searching for a class: Press Cmd+O on Mac or Ctrl+N on Windows or Linux.
Searching for a file: Press Cmd+Shift+O on Mac or Ctrl+Shift+N on Windows or Linux.
Searching all if you don’t know whether it’s a file, class, or a code fragment that you are looking for: Press Shift twice.

As for the navigation, the following shortcuts may save you a lot of time:

Going to the declaration of a variable: Press Cmd on Mac or Ctrl on Windows or Linux, and click on the variable.
Finding usages of a class, a method, or any symbol: Press Alt+F7.
Seeing your recent changes: Press Shift+Alt+C or go to View → Recent Changes on the main menu.
Seeing your recent files: Press Cmd+E on Mac or Ctrl+E on Windows or Linux, or go to View → Recent Files on the main menu.
Going backward and forward through your history of navigation after you jumped around: Press Cmd+[ / Cmd+] on Mac or Ctrl+Alt+Left / Ctrl+Alt+Right on Windows or Linux.

For more details, see the official documentation.

Using Version Control in PyCharm

Version control systems such as Git and Mercurial are some of the most important tools in the modern software development world. So, it is essential for an IDE to support them. PyCharm does that very well by integrating with a lot of popular VC systems such as Git (and Github), Mercurial, Perforce and, Subversion.

Note: Git is used for the following examples.

Configuring VCS

To enable VCS integration. Go to VCS → VCS Operations Popup… from the menu on the top or press Ctrl+V on Mac or Alt+` on Windows or Linux. Choose Enable Version Control Integration…. You’ll see the following window open:

Choose Git from the drop down list, click OK, and you have VCS enabled for your project. Note that if you opened an existing project that has version control enabled, then PyCharm will see that and automatically enable it.

Now, if you go to the VCS Operations Popup…, you’ll see a different popup with the options to do git add, git stash, git branch, git commit, git push and more:

If you can’t find what you need, you can most probably find it by going to VCS from the top menu and choosing Git, where you can even create and view pull requests.

Committing and Conflict Resolution

These are two features of VCS integration in PyCharm that I personally use and enjoy a lot! Let’s say you have finished your work and want to commit it. Go to VCS → VCS Operations Popup… → Commit… or press Cmd+K on Mac or Ctrl+K on Windows or Linux. You’ll see the following window open:

In this window, you can do the following:

Choose which files to commit
Write your commit message
Do all kinds of checks and cleanup before commit
See the difference of changes
Commit and push at once by pressing the arrow to the right of the Commit button on the right bottom and choosing Commit and Push…

It can feel magical and fast, especially if you’re used to doing everything manually on the command line.

When you work in a team, merge conflicts do happen. When somebody commits changes to a file that you’re working on, but their changes overlap with yours because both of you changed the same lines, then VCS will not be able to figure out if it should choose your changes or those of your teammate. So you’ll get these unfortunate arrows and symbols:

This looks strange, and it’s difficult to figure out which changes should be deleted and which ones should stay. PyCharm to the rescue! It has a much nicer and cleaner way of resolving conflicts. Go to VCS in the top menu, choose Git and then Resolve conflicts…. Choose the file whose conflicts you want to resolve and click on Merge. You will see the following window open:

On the left column, you will see your changes. On the right one, the changes made by your teammate. Finally, in the middle column, you will see the result. The conflicting lines are highlighted, and you can see a little X and >>/<< right beside those lines. Press the arrows to accept the changes and the X to decline. After you resolve all those conflicts, click the Apply button:

In the GIF above, for the first conflicting line, the author declined his own changes and accepted those of his teammates. Conversely, the author accepted his own changes and declined his teammates’ for the second conflicting line.

There’s a lot more that you can do with the VCS integration in PyCharm. For more details, see this documentation.

Using Plugins and External Tools in PyCharm

You can find almost everything you need for development in PyCharm. If you can’t, there is most probably a plugin that adds that functionality you need to PyCharm. For example, they can:

Add support for various languages and frameworks
Boost your productivity with shortcut hints, file watchers, and so on
Help you learn a new programming language with coding exercises

For instance, IdeaVim adds Vim emulation to PyCharm. If you like Vim, this can be a pretty good combination.

Material Theme UI changes the appearance of PyCharm to a Material Design look and feel:

Vue.js adds support for Vue.js projects. Markdown provides the capability to edit Markdown files within the IDE and see the rendered HTML in a live preview. You can find and install all of the available plugins by going to the Preferences → Plugins on Mac or Settings → Plugins on Windows or Linux, under the Marketplace tab:

If you can’t find what you need, you can even develop your own plugin.

If you can’t find the right plugin and don’t want to develop your own because there’s already a package in PyPI, then you can add it to PyCharm as an external tool. Take Flake8, the code analyzer, as an example.

First, install flake8 in your virtualenv with pip install flake8 in the Terminal app of your choice. You can also use the one integrated into PyCharm:

Then, go to Preferences → Tools on Mac or Settings → Tools on Windows/Linux, and then choose External Tools. Then click on the little + button at the bottom (1). In the new popup window, insert the details as shown below and click OK for both windows:

Here, Program (2) refers to the Flake8 executable that can be found in the folder /bin of your virtual environment. Arguments (3) refers to which file you want to analyze with the help of Flake8. Working directory is the directory of your project.

You could hardcode the absolute paths for everything here, but that would mean that you couldn’t use this external tool in other projects. You would be able to use it only inside one project for one file.

So you need to use something called Macros. Macros are basically variables in the format of $name$ that change according to your context. For example, $FileName$ is first.py when you’re editing first.py, and it is second.py when you’re editing second.py. You can see their list and insert any of them by clicking on the Insert Macro… buttons. Because you used macros here, the values will change according to the project you’re currently working on, and Flake8 will continue to do its job properly.

In order to use it, create a file example.py and put the following code in it:

 1 CONSTANT_VAR = 1
 2 
 3 
 4 
 5 def add(a, b):
 6     c = "hello"
 7     return a + b

It deliberately breaks some of the Flake8 rules. Right-click the background of this file. Choose External Tools and then Flake8. Voilà! The output of the Flake8 analysis will appear at the bottom:

In order to make it even better, you can add a shortcut for it. Go to Preferences on Mac or to Settings on Windows or Linux. Then, go to Keymap → External Tools → External Tools. Double-click Flake8 and choose Add Keyboard Shortcut. You’ll see this window:

In the image above, the shortcut is Ctrl+Alt+A for this tool. Add your preferred shortcut in the textbox and click OK for both windows. Now you can now use that shortcut to analyze the file you’re currently working on with Flake8.

PyCharm Professional Features

PyCharm Professional is a paid version of PyCharm with more out-of-the-box features and integrations. In this section, you’ll mainly be presented with overviews of its main features and links to the official documentation, where each feature is discussed in detail. Remember that none of the following features is available in the Community edition.

Django Support

PyCharm has extensive support for Django, one of the most popular and beloved Python web frameworks. To make sure that it’s enabled, do the following:

Open Preferences on Mac or Settings on Windows or Linux.
Choose Languages and Frameworks.
Choose Django.
Check the checkbox Enable Django support.
Apply changes.

Now that you’ve enabled Django support, your Django development journey will be a lot easier in PyCharm:

When creating a project, you’ll have a dedicated Django project type. This means that, when you choose this type, you’ll have all the necessary files and settings. This is the equivalent of using django-admin startproject mysite.
You can run manage.py commands directly inside PyCharm.
Django templates are supported, including:
- Syntax and error highlighting
- Code completion
- Navigation
- Completion for block names
- Completion for custom tags and filters
- Quick documentation for tags and filters
- Capability to debug them
Code completion in all other Django parts such as views, URLs and models, and code insight support for Django ORM.
Model dependency diagrams for Django models.

For more details on Django support, see the official documentation.

Database Support

Modern database development is a complex task with many supporting systems and workflows. That’s why JetBrains, the company behind PyCharm, developed a standalone IDE called DataGrip for that. It’s a separate product from PyCharm with a separate license.

Luckily, PyCharm supports all the features that are available in DataGrip through a plugin called Database tools and SQL, which is enabled by default. With the help of it, you can query, create and manage databases whether they’re working locally, on a server, or in the cloud. The plugin supports MySQL, PostgreSQL, Microsoft SQL Server, SQLite, MariaDB, Oracle, Apache Cassandra, and others. For more information on what you can do with this plugin, check out the comprehensive documentation on the database support.

Thread Concurrency Visualization

Django Channels, asyncio, and the recent frameworks like Starlette are examples of a growing trend in asynchronous Python programming. While it’s true that asynchronous programs do bring a lot of benefits to the table, it’s also notoriously hard to write and debug them. In such cases, Thread Concurrency Visualization can be just what the doctor ordered because it helps you take full control over your multi-threaded applications and optimize them.

Check out the comprehensive documentation of this feature for more details.

Profiler

Speaking of optimization, profiling is another technique that you can use to optimize your code. With its help, you can see which parts of your code are taking most of the execution time. A profiler runs in the following order of priority:

If you don’t have vmprof or yappi installed, then it’ll fall back to the standard cProfile. It’s well-documented, so I won’t rehash it here.

Scientific Mode

Python is not only a language for general and web programming. It also emerged as the best tool for data science and machine learning over these last years thanks to libraries and tools like NumPy, SciPy, scikit-learn, Matplotlib, Jupyter, and more. With such powerful libraries available, you need a powerful IDE to support all the functions such as graphing and analyzing those libraries have. PyCharm provides everything you need as thoroughly documented here.

Remote Development

One common cause of bugs in many applications is that development and production environments differ. Although, in most cases, it’s not possible to provide an exact copy of the production environment for development, pursuing it is a worthy goal.

With PyCharm, you can debug your application using an interpreter that is located on the other computer, such as a Linux VM. As a result, you can have the same interpreter as your production environment to fix and avoid many bugs resulting from the difference between development and production environments. Make sure to check out the official documentation to learn more.

Conclusion

PyCharm is one of best, if not the best, full-featured, dedicated, and versatile IDEs for Python development. It offers a ton of benefits, saving you a lot of time by helping you with routine tasks. Now you know how to be productive with it!

In this article, you learned about a lot, including:

Installing PyCharm
Writing code in PyCharm
Running your code in PyCharm
Debugging and testing your code in PyCharm
Editing an existing project in PyCharm
Searching and navigating in PyCharm
Using Version Control in PyCharm
Using Plugins and External Tools in PyCharm
Using PyCharm Professional features, such as Django support and Scientific mode

If there’s anything you’d like to ask or share, please reach out in the comments below. There’s also a lot more information at the PyCharm website for you to explore.

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use Python Lambda Functions

2019-08-27T14:00:00+00:00

Python and other languages like Java, C#, and even C++ have had lambda functions added to their syntax, whereas languages like LISP or the ML family of languages, Haskell, OCaml, and F#, use lambdas as a core concept. Python lambdas are little, anonymous functions, subject to a more restrictive but more concise syntax than regular Python functions.

By the end of this course, you’ll know:

How Python lambdas came to be
How lambdas compare with regular function objects
How to write lambda functions
Which functions in the Python standard library leverage lambdas
When to use or avoid Python lambda functions

This course is mainly for intermediate to experienced Python programmers, but it is accessible to any curious minds with interest in programming. All the examples included in this tutorial have been tested with Python 3.7.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Guide to Excel Spreadsheets in Python With openpyxl

2019-08-26T14:00:00+00:00

Excel spreadsheets are one of those things you might have to deal with at some point. Either it’s because your boss loves them or because marketing needs them, you might have to learn how to work with spreadsheets, and that’s when knowing openpyxl comes in handy!

Spreadsheets are a very intuitive and user-friendly way to manipulate large datasets without any prior technical background. That’s why they’re still so commonly used today.

In this article, you’ll learn how to use openpyxl to:

Manipulate Excel spreadsheets with confidence
Extract information from spreadsheets
Create simple or more complex spreadsheets, including adding styles, charts, and so on

This article is written for intermediate developers who have a pretty good knowledge of Python data structures, such as dicts and lists, but also feel comfortable around OOP and more intermediate level topics.

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

Before You Begin

If you ever get asked to extract some data from a database or log file into an Excel spreadsheet, or if you often have to convert an Excel spreadsheet into some more usable programmatic form, then this tutorial is perfect for you. Let’s jump into the openpyxl caravan!

Practical Use Cases

First things first, when would you need to use a package like openpyxl in a real-world scenario? You’ll see a few examples below, but really, there are hundreds of possible scenarios where this knowledge could come in handy.

Importing New Products Into a Database

You are responsible for tech in an online store company, and your boss doesn’t want to pay for a cool and expensive CMS system.

Every time they want to add new products to the online store, they come to you with an Excel spreadsheet with a few hundred rows and, for each of them, you have the product name, description, price, and so forth.

Now, to import the data, you’ll have to iterate over each spreadsheet row and add each product to the online store.

Exporting Database Data Into a Spreadsheet

Say you have a Database table where you record all your users’ information, including name, phone number, email address, and so forth.

Now, the Marketing team wants to contact all users to give them some discounted offer or promotion. However, they don’t have access to the Database, or they don’t know how to use SQL to extract that information easily.

What can you do to help? Well, you can make a quick script using openpyxl that iterates over every single User record and puts all the essential information into an Excel spreadsheet.

That’s gonna earn you an extra slice of cake at your company’s next birthday party!

Appending Information to an Existing Spreadsheet

You may also have to open a spreadsheet, read the information in it and, according to some business logic, append more data to it.

For example, using the online store scenario again, say you get an Excel spreadsheet with a list of users and you need to append to each row the total amount they’ve spent in your store.

This data is in the Database and, in order to do this, you have to read the spreadsheet, iterate through each row, fetch the total amount spent from the Database and then write back to the spreadsheet.

Not a problem for openpyxl!

Learning Some Basic Excel Terminology

Here’s a quick list of basic terms you’ll see when you’re working with Excel spreadsheets:

Term	Explanation
Spreadsheet or Workbook	A Spreadsheet is the main file you are creating or working with.
Worksheet or Sheet	A Sheet is used to split different kinds of content within the same spreadsheet. A Spreadsheet can have one or more Sheets.
Column	A Column is a vertical line, and it’s represented by an uppercase letter: A.
Row	A Row is a horizontal line, and it’s represented by a number: 1.
Cell	A Cell is a combination of Column and Row, represented by both an uppercase letter and a number: A1.

Getting Started With openpyxl

Now that you’re aware of the benefits of a tool like openpyxl, let’s get down to it and start by installing the package. For this tutorial, you should use Python 3.7 and openpyxl 2.6.2. To install the package, you can do the following:

$ pip install openpyxl

After you install the package, you should be able to create a super simple spreadsheet with the following code:

from openpyxl import Workbook

workbook = Workbook()
sheet = workbook.active

sheet["A1"] = "hello"
sheet["B1"] = "world!"

workbook.save(filename="hello_world.xlsx")

The code above should create a file called hello_world.xlsx in the folder you are using to run the code. If you open that file with Excel you should see something like this:

Woohoo, your first spreadsheet created!

Reading Excel Spreadsheets With openpyxl

Let’s start with the most essential thing one can do with a spreadsheet: read it.

You’ll go from a straightforward approach to reading a spreadsheet to more complex examples where you read the data and convert it into more useful Python structures.

Dataset for This Tutorial

Before you dive deep into some code examples, you should download this sample dataset and store it somewhere as sample.xlsx:

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

This is one of the datasets you’ll be using throughout this tutorial, and it’s a spreadsheet with a sample of real data from Amazon’s online product reviews. This dataset is only a tiny fraction of what Amazon provides, but for testing purposes, it’s more than enough.

A Simple Approach to Reading an Excel Spreadsheet

Finally, let’s start reading some spreadsheets! To begin with, open our sample spreadsheet:

>>>

>>> from openpyxl import load_workbook
>>> workbook = load_workbook(filename="sample.xlsx")
>>> workbook.sheetnames
['Sheet 1']

>>> sheet = workbook.active
>>> sheet
<Worksheet "Sheet 1">

>>> sheet.title
'Sheet 1'

In the code above, you first open the spreadsheet sample.xlsx using load_workbook(), and then you can use workbook.sheetnames to see all the sheets you have available to work with. After that, workbook.active selects the first available sheet and, in this case, you can see that it selects Sheet 1 automatically. Using these methods is the default way of opening a spreadsheet, and you’ll see it many times during this tutorial.

Now, after opening a spreadsheet, you can easily retrieve data from it like this:

>>>

>>> sheet["A1"]
<Cell 'Sheet 1'.A1>

>>> sheet["A1"].value
'marketplace'

>>> sheet["F10"].value
"G-Shock Men's Grey Sport Watch"

To return the actual value of a cell, you need to do .value. Otherwise, you’ll get the main Cell object. You can also use the method .cell() to retrieve a cell using index notation. Remember to add .value to get the actual value and not a Cell object:

>>>

>>> sheet.cell(row=10, column=6)
<Cell 'Sheet 1'.F10>

>>> sheet.cell(row=10, column=6).value
"G-Shock Men's Grey Sport Watch"

You can see that the results returned are the same, no matter which way you decide to go with. However, in this tutorial, you’ll be mostly using the first approach: ["A1"].

Note: Even though in Python you’re used to a zero-indexed notation, with spreadsheets you’ll always use a one-indexed notation where the first row or column always has index 1.

The above shows you the quickest way to open a spreadsheet. However, you can pass additional parameters to change the way a spreadsheet is loaded.

Additional Reading Options

There are a few arguments you can pass to load_workbook() that change the way a spreadsheet is loaded. The most important ones are the following two Booleans:

read_only loads a spreadsheet in read-only mode allowing you to open very large Excel files.
data_only ignores loading formulas and instead loads only the resulting values.

Importing Data From a Spreadsheet

Now that you’ve learned the basics about loading a spreadsheet, it’s about time you get to the fun part: the iteration and actual usage of the values within the spreadsheet.

This section is where you’ll learn all the different ways you can iterate through the data, but also how to convert that data into something usable and, more importantly, how to do it in a Pythonic way.

Iterating Through the Data

There are a few different ways you can iterate through the data depending on your needs.

You can slice the data with a combination of columns and rows:

>>>

>>> sheet["A1:C2"]
((<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>),
 (<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>))

You can get ranges of rows or columns:

>>>

>>> # Get all cells from column A
>>> sheet["A"]
(<Cell 'Sheet 1'.A1>,
 <Cell 'Sheet 1'.A2>,
 ...
 <Cell 'Sheet 1'.A99>,
 <Cell 'Sheet 1'.A100>)

>>> # Get all cells for a range of columns
>>> sheet["A:B"]
((<Cell 'Sheet 1'.A1>,
  <Cell 'Sheet 1'.A2>,
  ...
  <Cell 'Sheet 1'.A99>,
  <Cell 'Sheet 1'.A100>),
 (<Cell 'Sheet 1'.B1>,
  <Cell 'Sheet 1'.B2>,
  ...
  <Cell 'Sheet 1'.B99>,
  <Cell 'Sheet 1'.B100>))

>>> # Get all cells from row 5
>>> sheet[5]
(<Cell 'Sheet 1'.A5>,
 <Cell 'Sheet 1'.B5>,
 ...
 <Cell 'Sheet 1'.N5>,
 <Cell 'Sheet 1'.O5>)

>>> # Get all cells for a range of rows
>>> sheet[5:6]
((<Cell 'Sheet 1'.A5>,
  <Cell 'Sheet 1'.B5>,
  ...
  <Cell 'Sheet 1'.N5>,
  <Cell 'Sheet 1'.O5>),
 (<Cell 'Sheet 1'.A6>,
  <Cell 'Sheet 1'.B6>,
  ...
  <Cell 'Sheet 1'.N6>,
  <Cell 'Sheet 1'.O6>))

You’ll notice that all of the above examples return a tuple. If you want to refresh your memory on how to handle tuples in Python, check out the article on Lists and Tuples in Python.

There are also multiple ways of using normal Python generators to go through the data. The main methods you can use to achieve this are:

.iter_rows()
.iter_cols()

Both methods can receive the following arguments:

min_row
max_row
min_col
max_col

These arguments are used to set boundaries for the iteration:

>>>

>>> for row in sheet.iter_rows(min_row=1,
...                            max_row=2,
...                            min_col=1,
...                            max_col=3):
...     print(row)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>)
(<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>)


>>> for column in sheet.iter_cols(min_row=1,
...                               max_row=2,
...                               min_col=1,
...                               max_col=3):
...     print(column)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.A2>)
(<Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.B2>)
(<Cell 'Sheet 1'.C1>, <Cell 'Sheet 1'.C2>)

You’ll notice that in the first example, when iterating through the rows using .iter_rows(), you get one tuple element per row selected. While when using .iter_cols() and iterating through columns, you’ll get one tuple per column instead.

One additional argument you can pass to both methods is the Boolean values_only. When it’s set to True, the values of the cell are returned, instead of the Cell object:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=2,
...                              min_col=1,
...                              max_col=3,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id')
('US', 3653882, 'R3O9SGZBVQBV76')

If you want to iterate through the whole dataset, then you can also use the attributes .rows or .columns directly, which are shortcuts to using .iter_rows() and .iter_cols() without any arguments:

>>>

>>> for row in sheet.rows:
...     print(row)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>
...
<Cell 'Sheet 1'.M100>, <Cell 'Sheet 1'.N100>, <Cell 'Sheet 1'.O100>)

These shortcuts are very useful when you’re iterating through the whole dataset.

Manipulate Data Using Python’s Default Data Structures

Now that you know the basics of iterating through the data in a workbook, let’s look at smart ways of converting that data into Python structures.

As you saw earlier, the result from all iterations comes in the form of tuples. However, since a tuple is nothing more than an immutable list, you can easily access its data and transform it into other structures.

For example, say you want to extract product information from the sample.xlsx spreadsheet and into a dictionary where each key is a product ID.

A straightforward way to do this is to iterate over all the rows, pick the columns you know are related to product information, and then store that in a dictionary. Let’s code this out!

First of all, have a look at the headers and see what information you care most about:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=1,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id', 'product_id', ...)

This code returns a list of all the column names you have in the spreadsheet. To start, grab the columns with names:

product_id
product_parent
product_title
product_category

Lucky for you, the columns you need are all next to each other so you can use the min_column and max_column to easily get the data you want:

>>>

>>> for value in sheet.iter_rows(min_row=2,
...                              min_col=4,
...                              max_col=7,
...                              values_only=True):
...     print(value)
('B00FALQ1ZC', 937001370, 'Invicta Women\'s 15150 "Angel" 18k Yellow...)
('B00D3RGO20', 484010722, "Kenneth Cole New York Women's KC4944...)
...

Nice! Now that you know how to get all the important product information you need, let’s put that data into a dictionary:

import json
from openpyxl import load_workbook

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

products = {}

# Using the values_only because you want to return the cells' values
for row in sheet.iter_rows(min_row=2,
                           min_col=4,
                           max_col=7,
                           values_only=True):
    product_id = row[0]
    product = {
        "parent": row[1],
        "title": row[2],
        "category": row[3]
    }
    products[product_id] = product

# Using json here to be able to format the output for displaying later
print(json.dumps(products))

The code above returns a JSON similar to this:

{
  "B00FALQ1ZC": {
    "parent": 937001370,
    "title": "Invicta Women's 15150 ...",
    "category": "Watches"
  },
  "B00D3RGO20": {
    "parent": 484010722,
    "title": "Kenneth Cole New York ...",
    "category": "Watches"
  }
}

Here you can see that the output is trimmed to 2 products only, but if you run the script as it is, then you should get 98 products.

Convert Data Into Python Classes

To finalize the reading section of this tutorial, let’s dive into Python classes and see how you could improve on the example above and better structure the data.

For this, you’ll be using the new Python Data Classes that are available from Python 3.7. If you’re using an older version of Python, then you can use the default Classes instead.

So, first things first, let’s look at the data you have and decide what you want to store and how you want to store it.

As you saw right at the start, this data comes from Amazon, and it’s a list of product reviews. You can check the list of all the columns and their meaning on Amazon.

There are two significant elements you can extract from the data available:

Products
Reviews

A Product has:

ID
Title
Parent
Category

The Review has a few more fields:

ID
Customer ID
Stars
Headline
Body
Date

You can ignore a few of the review fields to make things a bit simpler.

So, a straightforward implementation of these two classes could be written in a separate file classes.py:

import datetime
from dataclasses import dataclass

@dataclass
class Product:
    id: str
    parent: str
    title: str
    category: str

@dataclass
class Review:
    id: str
    customer_id: str
    stars: int
    headline: str
    body: str
    date: datetime.datetime

After defining your data classes, you need to convert the data from the spreadsheet into these new structures.

Before doing the conversion, it’s worth looking at our header again and creating a mapping between columns and the fields you need:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=1,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id', 'product_id', ...)

>>> # Or an alternative
>>> for cell in sheet[1]:
...     print(cell.value)
marketplace
customer_id
review_id
product_id
product_parent
...

Let’s create a file mapping.py where you have a list of all the field names and their column location (zero-indexed) on the spreadsheet:

# Product fields
PRODUCT_ID = 3
PRODUCT_PARENT = 4
PRODUCT_TITLE = 5
PRODUCT_CATEGORY = 6

# Review fields
REVIEW_ID = 2
REVIEW_CUSTOMER = 1
REVIEW_STARS = 7
REVIEW_HEADLINE = 12
REVIEW_BODY = 13
REVIEW_DATE = 14

You don’t necessarily have to do the mapping above. It’s more for readability when parsing the row data, so you don’t end up with a lot of magic numbers lying around.

Finally, let’s look at the code needed to parse the spreadsheet data into a list of product and review objects:

from datetime import datetime
from openpyxl import load_workbook
from classes import Product, Review
from mapping import PRODUCT_ID, PRODUCT_PARENT, PRODUCT_TITLE, \
    PRODUCT_CATEGORY, REVIEW_DATE, REVIEW_ID, REVIEW_CUSTOMER, \
    REVIEW_STARS, REVIEW_HEADLINE, REVIEW_BODY

# Using the read_only method since you're not gonna be editing the spreadsheet
workbook = load_workbook(filename="sample.xlsx", read_only=True)
sheet = workbook.active

products = []
reviews = []

# Using the values_only because you just want to return the cell value
for row in sheet.iter_rows(min_row=2, values_only=True):
    product = Product(id=row[PRODUCT_ID],
                      parent=row[PRODUCT_PARENT],
                      title=row[PRODUCT_TITLE],
                      category=row[PRODUCT_CATEGORY])
    products.append(product)

    # You need to parse the date from the spreadsheet into a datetime format
    spread_date = row[REVIEW_DATE]
    parsed_date = datetime.strptime(spread_date, "%Y-%m-%d")

    review = Review(id=row[REVIEW_ID],
                    customer_id=row[REVIEW_CUSTOMER],
                    stars=row[REVIEW_STARS],
                    headline=row[REVIEW_HEADLINE],
                    body=row[REVIEW_BODY],
                    date=parsed_date)
    reviews.append(review)

print(products[0])
print(reviews[0])

After you run the code above, you should get some output like this:

Product(id='B00FALQ1ZC', parent=937001370, ...)
Review(id='R3O9SGZBVQBV76', customer_id=3653882, ...)

That’s it! Now you should have the data in a very simple and digestible class format, and you can start thinking of storing this in a Database or any other type of data storage you like.

Using this kind of OOP strategy to parse spreadsheets makes handling the data much simpler later on.

Appending New Data

Before you start creating very complex spreadsheets, have a quick look at an example of how to append data to an existing spreadsheet.

Go back to the first example spreadsheet you created (hello_world.xlsx) and try opening it and appending some data to it, like this:

from openpyxl import load_workbook

# Start by opening the spreadsheet and selecting the main sheet
workbook = load_workbook(filename="hello_world.xlsx")
sheet = workbook.active

# Write what you want into a specific cell
sheet["C1"] = "writing ;)"

# Save the spreadsheet
workbook.save(filename="hello_world_append.xlsx"

Et voilà, if you open the new hello_world_append.xlsx spreadsheet, you’ll see the following change:

Notice the additional writing ;) on cell C1.

Writing Excel Spreadsheets With openpyxl

There are a lot of different things you can write to a spreadsheet, from simple text or number values to complex formulas, charts, or even images.

Let’s start creating some spreadsheets!

Creating a Simple Spreadsheet

Previously, you saw a very quick example of how to write “Hello world!” into a spreadsheet, so you can start with that:

 1 from openpyxl import Workbook
 2 
 3 filename = "hello_world.xlsx"
 4 
 5 workbook = Workbook()
 6 sheet = workbook.active
 7 
 8 sheet["A1"] = "hello"
 9 sheet["B1"] = "world!"
10 
11 workbook.save(filename=filename)

The highlighted lines in the code above are the most important ones for writing. In the code, you can see that:

Line 5 shows you how to create a new empty workbook.
Lines 8 and 9 show you how to add data to specific cells.
Line 11 shows you how to save the spreadsheet when you’re done.

Even though these lines above can be straightforward, it’s still good to know them well for when things get a bit more complicated.

Note: You’ll be using the hello_world.xlsx spreadsheet for some of the upcoming examples, so keep it handy.

One thing you can do to help with coming code examples is add the following method to your Python file or console:

>>>

>>> def print_rows():
...     for row in sheet.iter_rows(values_only=True):
...         print(row)

It makes it easier to print all of your spreadsheet values by just calling print_rows().

Basic Spreadsheet Operations

Before you get into the more advanced topics, it’s good for you to know how to manage the most simple elements of a spreadsheet.

Adding and Updating Cell Values

You already learned how to add values to a spreadsheet like this:

>>>

>>> sheet["A1"] = "value"

There’s another way you can do this, by first selecting a cell and then changing its value:

>>>

>>> cell = sheet["A1"]
>>> cell
<Cell 'Sheet'.A1>

>>> cell.value
'hello'

>>> cell.value = "hey"
>>> cell.value
'hey'

The new value is only stored into the spreadsheet once you call workbook.save().

The openpyxl creates a cell when adding a value, if that cell didn’t exist before:

>>>

>>> # Before, our spreadsheet has only 1 row
>>> print_rows()
('hello', 'world!')

>>> # Try adding a value to row 10
>>> sheet["B10"] = "test"
>>> print_rows()
('hello', 'world!')
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, 'test')

As you can see, when trying to add a value to cell B10, you end up with a tuple with 10 rows, just so you can have that test value.

Managing Rows and Columns

One of the most common things you have to do when manipulating spreadsheets is adding or removing rows and columns. The openpyxl package allows you to do that in a very straightforward way by using the methods:

.insert_rows()
.delete_rows()
.insert_cols()
.delete_cols()

Every single one of those methods can receive two arguments:

idx
amount

Using our basic hello_world.xlsx example again, let’s see how these methods work:

>>>

>>> print_rows()
('hello', 'world!')

>>> # Insert a column before the existing column 1 ("A")
>>> sheet.insert_cols(idx=1)
>>> print_rows()
(None, 'hello', 'world!')

>>> # Insert 5 columns between column 2 ("B") and 3 ("C")
>>> sheet.insert_cols(idx=3, amount=5)
>>> print_rows()
(None, 'hello', None, None, None, None, None, 'world!')

>>> # Delete the created columns
>>> sheet.delete_cols(idx=3, amount=5)
>>> sheet.delete_cols(idx=1)
>>> print_rows()
('hello', 'world!')

>>> # Insert a new row in the beginning
>>> sheet.insert_rows(idx=1)
>>> print_rows()
(None, None)
('hello', 'world!')

>>> # Insert 3 new rows in the beginning
>>> sheet.insert_rows(idx=1, amount=3)
>>> print_rows()
(None, None)
(None, None)
(None, None)
(None, None)
('hello', 'world!')

>>> # Delete the first 4 rows
>>> sheet.delete_rows(idx=1, amount=4)
>>> print_rows()
('hello', 'world!')

The only thing you need to remember is that when inserting new data (rows or columns), the insertion happens before the idx parameter.

So, if you do insert_rows(1), it inserts a new row before the existing first row.

It’s the same for columns: when you call insert_cols(2), it inserts a new column right before the already existing second column (B).

However, when deleting rows or columns, .delete_... deletes data starting from the index passed as an argument.

For example, when doing delete_rows(2) it deletes row 2, and when doing delete_cols(3) it deletes the third column (C).

Managing Sheets

Sheet management is also one of those things you might need to know, even though it might be something that you don’t use that often.

If you look back at the code examples from this tutorial, you’ll notice the following recurring piece of code:

sheet = workbook.active

This is the way to select the default sheet from a spreadsheet. However, if you’re opening a spreadsheet with multiple sheets, then you can always select a specific one like this:

>>>

>>> # Let's say you have two sheets: "Products" and "Company Sales"
>>> workbook.sheetnames
['Products', 'Company Sales']

>>> # You can select a sheet using its title
>>> products_sheet = workbook["Products"]
>>> sales_sheet = workbook["Company Sales"]

You can also change a sheet title very easily:

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> products_sheet = workbook["Products"]
>>> products_sheet.title = "New Products"

>>> workbook.sheetnames
['New Products', 'Company Sales']

If you want to create or delete sheets, then you can also do that with .create_sheet() and .remove():

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> operations_sheet = workbook.create_sheet("Operations")
>>> workbook.sheetnames
['Products', 'Company Sales', 'Operations']

>>> # You can also define the position to create the sheet at
>>> hr_sheet = workbook.create_sheet("HR", 0)
>>> workbook.sheetnames
['HR', 'Products', 'Company Sales', 'Operations']

>>> # To remove them, just pass the sheet as an argument to the .remove()
>>> workbook.remove(operations_sheet)
>>> workbook.sheetnames
['HR', 'Products', 'Company Sales']

>>> workbook.remove(hr_sheet)
>>> workbook.sheetnames
['Products', 'Company Sales']

One other thing you can do is make duplicates of a sheet using copy_worksheet():

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> products_sheet = workbook["Products"]
>>> workbook.copy_worksheet(products_sheet)
<Worksheet "Products Copy">

>>> workbook.sheetnames
['Products', 'Company Sales', 'Products Copy']

If you open your spreadsheet after saving the above code, you’ll notice that the sheet Products Copy is a duplicate of the sheet Products.

Freezing Rows and Columns

Something that you might want to do when working with big spreadsheets is to freeze a few rows or columns, so they remain visible when you scroll right or down.

Freezing data allows you to keep an eye on important rows or columns, regardless of where you scroll in the spreadsheet.

Again, openpyxl also has a way to accomplish this by using the worksheet freeze_panes attribute. For this example, go back to our sample.xlsx spreadsheet and try doing the following:

>>>

>>> workbook = load_workbook(filename="sample.xlsx")
>>> sheet = workbook.active
>>> sheet.freeze_panes = "C2"
>>> workbook.save("sample_frozen.xlsx")

If you open the sample_frozen.xlsx spreadsheet in your favorite spreadsheet editor, you’ll notice that row 1 and columns A and B are frozen and are always visible no matter where you navigate within the spreadsheet.

This feature is handy, for example, to keep headers within sight, so you always know what each column represents.

Here’s how it looks in the editor:

Notice how you’re at the end of the spreadsheet, and yet, you can see both row 1 and columns A and B.

Adding Filters

You can use openpyxl to add filters and sorts to your spreadsheet. However, when you open the spreadsheet, the data won’t be rearranged according to these sorts and filters.

At first, this might seem like a pretty useless feature, but when you’re programmatically creating a spreadsheet that is going to be sent and used by somebody else, it’s still nice to at least create the filters and allow people to use it afterward.

The code below is an example of how you would add some filters to our existing sample.xlsx spreadsheet:

>>>

>>> # Check the used spreadsheet space using the attribute "dimensions"
>>> sheet.dimensions
'A1:O100'

>>> sheet.auto_filter.ref = "A1:O100"
>>> workbook.save(filename="sample_with_filters.xlsx")

You should now see the filters created when opening the spreadsheet in your editor:

You don’t have to use sheet.dimensions if you know precisely which part of the spreadsheet you want to apply filters to.

Adding Formulas

Formulas (or formulae) are one of the most powerful features of spreadsheets.

They gives you the power to apply specific mathematical equations to a range of cells. Using formulas with openpyxl is as simple as editing the value of a cell.

You can see the list of formulas supported by openpyxl:

>>>

>>> from openpyxl.utils import FORMULAE
>>> FORMULAE
frozenset({'ABS',
           'ACCRINT',
           'ACCRINTM',
           'ACOS',
           'ACOSH',
           'AMORDEGRC',
           'AMORLINC',
           'AND',
           ...
           'YEARFRAC',
           'YIELD',
           'YIELDDISC',
           'YIELDMAT',
           'ZTEST'})

Let’s add some formulas to our sample.xlsx spreadsheet.

Starting with something easy, let’s check the average star rating for the 99 reviews within the spreadsheet:

>>>

>>> # Star rating is column "H"
>>> sheet["P2"] = "=AVERAGE(H2:H100)"
>>> workbook.save(filename="sample_formulas.xlsx")

If you open the spreadsheet now and go to cell P2, you should see that its value is: 4.18181818181818. Have a look in the editor:

You can use the same methodology to add any formulas to your spreadsheet. For example, let’s count the number of reviews that had helpful votes:

>>>

>>> # The helpful votes are counted on column "I"
>>> sheet["P3"] = '=COUNTIF(I2:I100, ">0")'
>>> workbook.save(filename="sample_formulas.xlsx")

You should get the number 21 on your P3 spreadsheet cell like so:

You’ll have to make sure that the strings within a formula are always in double quotes, so you either have to use single quotes around the formula like in the example above or you’ll have to escape the double quotes inside the formula: "=COUNTIF(I2:I100, \">0\")".

There are a ton of other formulas you can add to your spreadsheet using the same procedure you tried above. Give it a go yourself!

Adding Styles

Even though styling a spreadsheet might not be something you would do every day, it’s still good to know how to do it.

Using openpyxl, you can apply multiple styling options to your spreadsheet, including fonts, borders, colors, and so on. Have a look at the openpyxl documentation to learn more.

You can also choose to either apply a style directly to a cell or create a template and reuse it to apply styles to multiple cells.

Let’s start by having a look at simple cell styling, using our sample.xlsx again as the base spreadsheet:

>>>

>>> # Import necessary style classes
>>> from openpyxl.styles import Font, Color, Alignment, Border, Side, colors

>>> # Create a few styles
>>> bold_font = Font(bold=True)
>>> big_red_text = Font(color=colors.RED, size=20)
>>> center_aligned_text = Alignment(horizontal="center")
>>> double_border_side = Side(border_style="double")
>>> square_border = Border(top=double_border_side,
...                        right=double_border_side,
...                        bottom=double_border_side,
...                        left=double_border_side)

>>> # Style some cells!
>>> sheet["A2"].font = bold_font
>>> sheet["A3"].font = big_red_text
>>> sheet["A4"].alignment = center_aligned_text
>>> sheet["A5"].border = square_border
>>> workbook.save(filename="sample_styles.xlsx")

If you open your spreadsheet now, you should see quite a few different styles on the first 5 cells of column A:

There you go. You got:

A2 with the text in bold
A3 with the text in red and bigger font size
A4 with the text centered
A5 with a square border around the text

Note: For the colors, you can also use HEX codes instead by doing Font(color="C70E0F").

You can also combine styles by simply adding them to the cell at the same time:

>>>

>>> # Reusing the same styles from the example above
>>> sheet["A6"].alignment = center_aligned_text
>>> sheet["A6"].font = big_red_text
>>> sheet["A6"].border = square_border
>>> workbook.save(filename="sample_styles.xlsx")

Have a look at cell A6 here:

When you want to apply multiple styles to one or several cells, you can use a NamedStyle class instead, which is like a style template that you can use over and over again. Have a look at the example below:

>>>

>>> from openpyxl.styles import NamedStyle

>>> # Let's create a style template for the header row
>>> header = NamedStyle(name="header")
>>> header.font = Font(bold=True)
>>> header.border = Border(bottom=Side(border_style="thin"))
>>> header.alignment = Alignment(horizontal="center", vertical="center")

>>> # Now let's apply this to all first row (header) cells
>>> header_row = sheet[1]
>>> for cell in header_row:
...     cell.style = header

>>> workbook.save(filename="sample_styles.xlsx")

If you open the spreadsheet now, you should see that its first row is bold, the text is aligned to the center, and there’s a small bottom border! Have a look below:

As you saw above, there are many options when it comes to styling, and it depends on the use case, so feel free to check openpyxl documentation and see what other things you can do.

Conditional Formatting

This feature is one of my personal favorites when it comes to adding styles to a spreadsheet.

It’s a much more powerful approach to styling because it dynamically applies styles according to how the data in the spreadsheet changes.

In a nutshell, conditional formatting allows you to specify a list of styles to apply to a cell (or cell range) according to specific conditions.

For example, a widespread use case is to have a balance sheet where all the negative totals are in red, and the positive ones are in green. This formatting makes it much more efficient to spot good vs bad periods.

Without further ado, let’s pick our favorite spreadsheet—sample.xlsx—and add some conditional formatting.

You can start by adding a simple one that adds a red background to all reviews with less than 3 stars:

>>>

>>> from openpyxl.styles import PatternFill, colors
>>> from openpyxl.styles.differential import DifferentialStyle
>>> from openpyxl.formatting.rule import Rule

>>> red_background = PatternFill(bgColor=colors.RED)
>>> diff_style = DifferentialStyle(fill=red_background)
>>> rule = Rule(type="expression", dxf=diff_style)
>>> rule.formula = ["$H1<3"]
>>> sheet.conditional_formatting.add("A1:O100", rule)
>>> workbook.save("sample_conditional_formatting.xlsx")

Now you’ll see all the reviews with a star rating below 3 marked with a red background:

Code-wise, the only things that are new here are the objects DifferentialStyle and Rule:

DifferentialStyle is quite similar to NamedStyle, which you already saw above, and it’s used to aggregate multiple styles such as fonts, borders, alignment, and so forth.
Rule is responsible for selecting the cells and applying the styles if the cells match the rule’s logic.

Using a Rule object, you can create numerous conditional formatting scenarios.

However, for simplicity sake, the openpyxl package offers 3 built-in formats that make it easier to create a few common conditional formatting patterns. These built-ins are:

ColorScale
IconSet
DataBar

The ColorScale gives you the ability to create color gradients:

>>>

>>> from openpyxl.formatting.rule import ColorScaleRule
>>> color_scale_rule = ColorScaleRule(start_type="min",
...                                   start_color=colors.RED,
...                                   end_type="max",
...                                   end_color=colors.GREEN)

>>> # Again, let's add this gradient to the star ratings, column "H"
>>> sheet.conditional_formatting.add("H2:H100", color_scale_rule)
>>> workbook.save(filename="sample_conditional_formatting_color_scale.xlsx")

Now you should see a color gradient on column H, from red to green, according to the star rating:

You can also add a third color and make two gradients instead:

>>>

>>> from openpyxl.formatting.rule import ColorScaleRule
>>> color_scale_rule = ColorScaleRule(start_type="num",
...                                   start_value=1,
...                                   start_color=colors.RED,
...                                   mid_type="num",
...                                   mid_value=3,
...                                   mid_color=colors.YELLOW,
...                                   end_type="num",
...                                   end_value=5,
...                                   end_color=colors.GREEN)

>>> # Again, let's add this gradient to the star ratings, column "H"
>>> sheet.conditional_formatting.add("H2:H100", color_scale_rule)
>>> workbook.save(filename="sample_conditional_formatting_color_scale_3.xlsx")

This time, you’ll notice that star ratings between 1 and 3 have a gradient from red to yellow, and star ratings between 3 and 5 have a gradient from yellow to green:

The IconSet allows you to add an icon to the cell according to its value:

>>>

>>> from openpyxl.formatting.rule import IconSetRule

>>> icon_set_rule = IconSetRule("5Arrows", "num", [1, 2, 3, 4, 5])
>>> sheet.conditional_formatting.add("H2:H100", icon_set_rule)
>>> workbook.save("sample_conditional_formatting_icon_set.xlsx")

You’ll see a colored arrow next to the star rating. This arrow is red and points down when the value of the cell is 1 and, as the rating gets better, the arrow starts pointing up and becomes green:

The openpyxl package has a full list of other icons you can use, besides the arrow.

Finally, the DataBar allows you to create progress bars:

>>>

>>> from openpyxl.formatting.rule import DataBarRule

>>> data_bar_rule = DataBarRule(start_type="num",
...                             start_value=1,
...                             end_type="num",
...                             end_value="5",
...                             color=colors.GREEN)
>>> sheet.conditional_formatting.add("H2:H100", data_bar_rule)
>>> workbook.save("sample_conditional_formatting_data_bar.xlsx")

You’ll now see a green progress bar that gets fuller the closer the star rating is to the number 5:

As you can see, there are a lot of cool things you can do with conditional formatting.

Here, you saw only a few examples of what you can achieve with it, but check the openpyxl documentation to see a bunch of other options.

Adding Images

Even though images are not something that you’ll often see in a spreadsheet, it’s quite cool to be able to add them. Maybe you can use it for branding purposes or to make spreadsheets more personal.

To be able to load images to a spreadsheet using openpyxl, you’ll have to install Pillow:

$ pip install Pillow

Apart from that, you’ll also need an image. For this example, you can grab the Real Python logo below and convert it from .webp to .png using an online converter such as cloudconvert.com, save the final file as logo.png, and copy it to the root folder where you’re running your examples:

Afterward, this is the code you need to import that image into the hello_word.xlsx spreadsheet:

from openpyxl import load_workbook
from openpyxl.drawing.image import Image

# Let's use the hello_world spreadsheet since it has less data
workbook = load_workbook(filename="hello_world.xlsx")
sheet = workbook.active

logo = Image("logo.png")

# A bit of resizing to not fill the whole spreadsheet with the logo
logo.height = 150
logo.width = 150

sheet.add_image(logo, "A3")
workbook.save(filename="hello_world_logo.xlsx")

You have an image on your spreadsheet! Here it is:

The image’s left top corner is on the cell you chose, in this case, A3.

Adding Pretty Charts

Another powerful thing you can do with spreadsheets is create an incredible variety of charts.

Charts are a great way to visualize and understand loads of data quickly. There are a lot of different chart types: bar chart, pie chart, line chart, and so on. openpyxl has support for a lot of them.

Here, you’ll see only a couple of examples of charts because the theory behind it is the same for every single chart type:

Note: A few of the chart types that openpyxl currently doesn’t have support for are Funnel, Gantt, Pareto, Treemap, Waterfall, Map, and Sunburst.

For any chart you want to build, you’ll need to define the chart type: BarChart, LineChart, and so forth, plus the data to be used for the chart, which is called Reference.

Before you can build your chart, you need to define what data you want to see represented in it. Sometimes, you can use the dataset as is, but other times you need to massage the data a bit to get additional information.

Let’s start by building a new workbook with some sample data:

 1 from openpyxl import Workbook
 2 from openpyxl.chart import BarChart, Reference
 3 
 4 workbook = Workbook()
 5 sheet = workbook.active
 6 
 7 # Let's create some sample sales data
 8 rows = [
 9     ["Product", "Online", "Store"],
10     [1, 30, 45],
11     [2, 40, 30],
12     [3, 40, 25],
13     [4, 50, 30],
14     [5, 30, 25],
15     [6, 25, 35],
16     [7, 20, 40],
17 ]
18 
19 for row in rows:
20     sheet.append(row)

Now you’re going to start by creating a bar chart that displays the total number of sales per product:

22 chart = BarChart()
23 data = Reference(worksheet=sheet,
24                  min_row=1,
25                  max_row=8,
26                  min_col=2,
27                  max_col=3)
28 
29 chart.add_data(data, titles_from_data=True)
30 sheet.add_chart(chart, "E2")
31 
32 workbook.save("chart.xlsx")

There you have it. Below, you can see a very straightforward bar chart showing the difference between online product sales online and in-store product sales:

Like with images, the top left corner of the chart is on the cell you added the chart to. In your case, it was on cell E2.

Note: Depending on whether you’re using Microsoft Excel or an open-source alternative (LibreOffice or OpenOffice), the chart might look slightly different.

Try creating a line chart instead, changing the data a bit:

 1 import random
 2 from openpyxl import Workbook
 3 from openpyxl.chart import LineChart, Reference
 4 
 5 workbook = Workbook()
 6 sheet = workbook.active
 7 
 8 # Let's create some sample sales data
 9 rows = [
10     ["", "January", "February", "March", "April",
11     "May", "June", "July", "August", "September",
12      "October", "November", "December"],
13     [1, ],
14     [2, ],
15     [3, ],
16 ]
17 
18 for row in rows:
19     sheet.append(row)
20 
21 for row in sheet.iter_rows(min_row=2,
22                            max_row=4,
23                            min_col=2,
24                            max_col=13):
25     for cell in row:
26         cell.value = random.randrange(5, 100)

With the above code, you’ll be able to generate some random data regarding the sales of 3 different products across a whole year.

Once that’s done, you can very easily create a line chart with the following code:

28 chart = LineChart()
29 data = Reference(worksheet=sheet,
30                  min_row=2,
31                  max_row=4,
32                  min_col=1,
33                  max_col=13)
34 
35 chart.add_data(data, from_rows=True, titles_from_data=True)
36 sheet.add_chart(chart, "C6")
37 
38 workbook.save("line_chart.xlsx")

Here’s the outcome of the above piece of code:

One thing to keep in mind here is the fact that you’re using from_rows=True when adding the data. This argument makes the chart plot row by row instead of column by column.

In your sample data, you see that each product has a row with 12 values (1 column per month). That’s why you use from_rows. If you don’t pass that argument, by default, the chart tries to plot by column, and you’ll get a month-by-month comparison of sales.

Another difference that has to do with the above argument change is the fact that our Reference now starts from the first column, min_col=1, instead of the second one. This change is needed because the chart now expects the first column to have the titles.

There are a couple of other things you can also change regarding the style of the chart. For example, you can add specific categories to the chart:

cats = Reference(worksheet=sheet,
                 min_row=1,
                 max_row=1,
                 min_col=2,
                 max_col=13)
chart.set_categories(cats)

Add this piece of code before saving the workbook, and you should see the month names appearing instead of numbers:

Code-wise, this is a minimal change. But in terms of the readability of the spreadsheet, this makes it much easier for someone to open the spreadsheet and understand the chart straight away.

Another thing you can do to improve the chart readability is to add an axis. You can do it using the attributes x_axis and y_axis:

chart.x_axis.title = "Months"
chart.y_axis.title = "Sales (per unit)"

This will generate a spreadsheet like the below one:

As you can see, small changes like the above make reading your chart a much easier and quicker task.

There is also a way to style your chart by using Excel’s default ChartStyle property. In this case, you have to choose a number between 1 and 48. Depending on your choice, the colors of your chart change as well:

# You can play with this by choosing any number between 1 and 48
chart.style = 24

With the style selected above, all lines have some shade of orange:

There is no clear documentation on what each style number looks like, but this spreadsheet has a few examples of the styles available.

Here’s the full code used to generate the line chart with categories, axis titles, and style:

import random
from openpyxl import Workbook
from openpyxl.chart import LineChart, Reference

workbook = Workbook()
sheet = workbook.active

# Let's create some sample sales data
rows = [
    ["", "January", "February", "March", "April",
    "May", "June", "July", "August", "September",
     "October", "November", "December"],
    [1, ],
    [2, ],
    [3, ],
]

for row in rows:
    sheet.append(row)

for row in sheet.iter_rows(min_row=2,
                           max_row=4,
                           min_col=2,
                           max_col=13):
    for cell in row:
        cell.value = random.randrange(5, 100)

# Create a LineChart and add the main data
chart = LineChart()
data = Reference(worksheet=sheet,
                           min_row=2,
                           max_row=4,
                           min_col=1,
                           max_col=13)
chart.add_data(data, titles_from_data=True, from_rows=True)

# Add categories to the chart
cats = Reference(worksheet=sheet,
                 min_row=1,
                 max_row=1,
                 min_col=2,
                 max_col=13)
chart.set_categories(cats)

# Rename the X and Y Axis
chart.x_axis.title = "Months"
chart.y_axis.title = "Sales (per unit)"

# Apply a specific Style
chart.style = 24

# Save!
sheet.add_chart(chart, "C6")
workbook.save("line_chart.xlsx")

There are a lot more chart types and customization you can apply, so be sure to check out the package documentation on this if you need some specific formatting.

Convert Python Classes to Excel Spreadsheet

You already saw how to convert an Excel spreadsheet’s data into Python classes, but now let’s do the opposite.

Let’s imagine you have a database and are using some Object-Relational Mapping (ORM) to map DB objects into Python classes. Now, you want to export those same objects into a spreadsheet.

Let’s assume the following data classes to represent the data coming from your database regarding product sales:

from dataclasses import dataclass
from typing import List

@dataclass
class Sale:
    id: str
    quantity: int

@dataclass
class Product:
    id: str
    name: str
    sales: List[Sale]

Now, let’s generate some random data, assuming the above classes are stored in a db_classes.py file:

 1 import random
 2 
 3 # Ignore these for now. You'll use them in a sec ;)
 4 from openpyxl import Workbook
 5 from openpyxl.chart import LineChart, Reference
 6 
 7 from db_classes import Product, Sale
 8 
 9 products = []
10 
11 # Let's create 5 products
12 for idx in range(1, 6):
13     sales = []
14 
15     # Create 5 months of sales
16     for _ in range(5):
17         sale = Sale(quantity=random.randrange(5, 100))
18         sales.append(sale)
19 
20     product = Product(id=str(idx),
21                       name="Product %s" % idx,
22                       sales=sales)
23     products.append(product)

By running this piece of code, you should get 5 products with 5 months of sales with a random quantity of sales for each month.

Now, to convert this into a spreadsheet, you need to iterate over the data and append it to the spreadsheet:

25 workbook = Workbook()
26 sheet = workbook.active
27 
28 # Append column names first
29 sheet.append(["Product ID", "Product Name", "Month 1",
30               "Month 2", "Month 3", "Month 4", "Month 5"])
31 
32 # Append the data
33 for product in products:
34     data = [product.id, product.name]
35     for sale in product.sales:
36         data.append(sale.quantity)
37     sheet.append(data)

That’s it. That should allow you to create a spreadsheet with some data coming from your database.

However, why not use some of that cool knowledge you gained recently to add a chart as well to display that data more visually?

All right, then you could probably do something like this:

38 chart = LineChart()
39 data = Reference(worksheet=sheet,
40                  min_row=2,
41                  max_row=6,
42                  min_col=2,
43                  max_col=7)
44 
45 chart.add_data(data, titles_from_data=True, from_rows=True)
46 sheet.add_chart(chart, "B8")
47 
48 cats = Reference(worksheet=sheet,
49                  min_row=1,
50                  max_row=1,
51                  min_col=3,
52                  max_col=7)
53 chart.set_categories(cats)
54 
55 chart.x_axis.title = "Months"
56 chart.y_axis.title = "Sales (per unit)"
57 
58 workbook.save(filename="oop_sample.xlsx")

Now we’re talking! Here’s a spreadsheet generated from database objects and with a chart and everything:

That’s a great way for you to wrap up your new knowledge of charts!

Bonus: Working With Pandas

Even though you can use Pandas to handle Excel files, there are few things that you either can’t accomplish with Pandas or that you’d be better off just using openpyxl directly.

For example, some of the advantages of using openpyxl are the ability to easily customize your spreadsheet with styles, conditional formatting, and such.

But guess what, you don’t have to worry about picking. In fact, openpyxl has support for both converting data from a Pandas DataFrame into a workbook or the opposite, converting an openpyxl workbook into a Pandas DataFrame.

Note: If you’re new to Pandas, check our course on Pandas DataFrames beforehand.

First things first, remember to install the pandas package:

$ pip install pandas

Then, let’s create a sample DataFrame:

 1 import pandas as pd
 2 
 3 data = {
 4     "Product Name": ["Product 1", "Product 2"],
 5     "Sales Month 1": [10, 20],
 6     "Sales Month 2": [5, 35],
 7 }
 8 df = pd.DataFrame(data)

Now that you have some data, you can use .dataframe_to_rows() to convert it from a DataFrame into a worksheet:

10 from openpyxl import Workbook
11 from openpyxl.utils.dataframe import dataframe_to_rows
12 
13 workbook = Workbook()
14 sheet = workbook.active
15 
16 for row in dataframe_to_rows(df, index=False, header=True):
17     sheet.append(row)
18 
19 workbook.save("pandas.xlsx")

You should see a spreadsheet that looks like this:

If you want to add the DataFrame’s index, you can change index=True, and it adds each row’s index into your spreadsheet.

On the other hand, if you want to convert a spreadsheet into a DataFrame, you can also do it in a very straightforward way like so:

import pandas as pd
from openpyxl import load_workbook

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

values = sheet.values
df = pd.DataFrame(values)

Alternatively, if you want to add the correct headers and use the review ID as the index, for example, then you can also do it like this instead:

import pandas as pd
from openpyxl import load_workbook
from mapping import REVIEW_ID

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

data = sheet.values

# Set the first row as the columns for the DataFrame
cols = next(data)
data = list(data)

# Set the field "review_id" as the indexes for each row
idx = [row[REVIEW_ID] for row in data]

df = pd.DataFrame(data, index=idx, columns=cols)

Using indexes and columns allows you to access data from your DataFrame easily:

>>>

>>> df.columns
Index(['marketplace', 'customer_id', 'review_id', 'product_id',
       'product_parent', 'product_title', 'product_category', 'star_rating',
       'helpful_votes', 'total_votes', 'vine', 'verified_purchase',
       'review_headline', 'review_body', 'review_date'],
      dtype='object')

>>> # Get first 10 reviews' star rating
>>> df["star_rating"][:10]
R3O9SGZBVQBV76    5
RKH8BNC3L5DLF     5
R2HLE8WKZSU3NL    2
R31U3UH5AZ42LL    5
R2SV659OUJ945Y    4
RA51CP8TR5A2L     5
RB2Q7DLDN6TH6     5
R2RHFJV0UYBK3Y    1
R2Z6JOQ94LFHEP    5
RX27XIIWY5JPB     4
Name: star_rating, dtype: int64

>>> # Grab review with id "R2EQL1V1L6E0C9", using the index
>>> df.loc["R2EQL1V1L6E0C9"]
marketplace               US
customer_id         15305006
review_id     R2EQL1V1L6E0C9
product_id        B004LURNO6
product_parent     892860326
review_headline   Five Stars
review_body          Love it
review_date       2015-08-31
Name: R2EQL1V1L6E0C9, dtype: object

There you go, whether you want to use openpyxl to prettify your Pandas dataset or use Pandas to do some hardcore algebra, you now know how to switch between both packages.

Conclusion

Phew, after that long read, you now know how to work with spreadsheets in Python! You can rely on openpyxl, your trustworthy companion, to:

Extract valuable information from spreadsheets in a Pythonic manner
Create your own spreadsheets, no matter the complexity level
Add cool features such as conditional formatting or charts to your spreadsheets

There are a few other things you can do with openpyxl that might not have been covered in this tutorial, but you can always check the package’s official documentation website to learn more about it. You can even venture into checking its source code and improving the package further.

Feel free to leave any comments below if you have any questions, or if there’s any section you’d love to hear more about.

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Histogram Plotting: NumPy, Matplotlib, Pandas & Seaborn

2019-08-20T14:00:00+00:00

In this course, you’ll be equipped to make production-quality, presentation-ready Python histogram plots with a range of choices and features.

If you have introductory to intermediate knowledge in Python and statistics, then you can use this article as a one-stop shop for building and plotting histograms in Python using libraries from its scientific stack, including NumPy, Matplotlib, Pandas, and Seaborn.

A histogram is a great tool for quickly assessing a probability distribution that is intuitively understood by almost any audience. Python offers a handful of different options for building and plotting histograms. Most people know a histogram by its graphical representation, which is similar to a bar graph:

This course will guide you through creating plots like the one above as well as more complex ones. Here’s what you’ll cover:

Building histograms in pure Python, without use of third party libraries
Constructing histograms with NumPy to summarize the underlying data
Plotting the resulting histogram with Matplotlib, Pandas, and Seaborn

Free Bonus: Short on time? Click here to get access to a free two-page Python histograms cheat sheet that summarizes the techniques explained in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Traditional Face Detection With Python

2019-08-13T14:00:00+00:00

Computer vision is an exciting and growing field. There are tons of interesting problems to solve! One of them is face detection: the ability of a computer to recognize that a photograph contains a human face, and tell you where it is located. In this course, you’ll learn about face detection with Python.

To detect any object in an image, it is necessary to understand how images are represented inside a computer, and how that object differs visually from any other object.

Once that is done, the process of scanning an image and looking for those visual cues needs to be automated and optimized. All these steps come together to form a fast and reliable computer vision algorithm.

In this course, you’ll learn:

What face detection is
How computers understand features in images
How to quickly analyze many different features to reach a decision
How to use a minimal Python solution for detecting human faces in images

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

11 Beginner Tips for Learning Python

2019-08-06T14:00:00+00:00

We are so excited that you have decided to embark on the journey of learning Python! One of the most common questions we receive from our readers is “What’s the best way to learn Python?”

The first step in learning any programming language is making sure that you understand how to learn. Learning how to learn is arguably the most critical skill involved in computer programming.

Why is knowing how to learn so important? Languages evolve, libraries are created, and tools are upgraded. Knowing how to learn will be essential to keeping up with these changes and becoming a successful programmer.

In this course, you’ll see several learning strategies that will help you jumpstart your journey towards becoming a rockstar Python programmer!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Dictionaries in Python

2019-07-30T14:00:00+00:00

Python provides a composite data type called a dictionary, which is similar to a list in that it is a collection of objects.

Here’s what you’ll learn in this course: You’ll cover the basic characteristics of Python dictionaries and learn how to access and manage dictionary data. Once you’ve finished this course, you’ll have a good sense of when a dictionary is the appropriate data type to use and know how to use it.

Dictionaries and lists share the following characteristics:

Both are mutable.
Both are dynamic. They can grow and shrink as needed.
Both can be nested. A list can contain another list. A dictionary can contain another dictionary. A dictionary can also contain a list, and vice versa.

Dictionaries differ from lists primarily in how elements are accessed:

List elements are accessed by their position in the list, via indexing.
Dictionary elements are accessed via keys.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Logging in Python

2019-07-23T14:00:00+00:00

Logging is a very useful tool in a programmer’s toolbox. It can help you develop a better understanding of the flow of a program and discover scenarios that you might not even have thought of while developing.

Logs provide developers with an extra set of eyes that are constantly looking at the flow that an application is going through. They can store information, like which user or IP accessed the application. If an error occurs, then they can provide more insights than a stack trace by telling you what the state of the program was before it arrived at the line of code where the error occurred.

By logging useful data from the right places, you can not only debug errors easily but also use the data to analyze the performance of the application to plan for scaling or look at usage patterns to plan for marketing.

Python provides a logging system as a part of its standard library, so you can quickly add logging to your application. In this course, you’ll learn why using this module is the best way to add logging to your application as well as how to get started quickly, and you will get an introduction to some of the advanced features available.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

How to Use Generators and yield in Python

Using Generators

Example 1: Reading Large Files

Example 2: Generating an Infinite Sequence

Example 3: Detecting Palindromes

Understanding Generators

Building Generators With Generator Expressions

Profiling Generator Performance

Understanding the Python Yield Statement

Using Advanced Generator Methods

How to Use .send()

How to Use .throw()

How to Use .close()

Creating Data Pipelines With Generators

Conclusion

Thonny: The Beginner-Friendly Python Editor

Getting Started With Async Features in Python

Understanding Asynchronous Programming

Building a Synchronous Web Server

Thinking Differently About Programming

Programming Parents: Not as Easy as It Looks!

Thought Experiment #1: The Synchronous Parent

Thought Experiment #2: The Polling Parent

Thought Experiment #3: The Threading Parent

Using Python Async Features in Practice

Synchronous Programming

Simple Cooperative Concurrency

Cooperative Concurrency With Blocking Calls

Cooperative Concurrency With Non-Blocking Calls

Synchronous (Blocking) HTTP Calls

Asynchronous (Non-Blocking) HTTP Calls

Conclusion

How to Convert a Python String to int

Representing Integers in Python

Converting a Python String to an int

Converting a Python int to a String

Conclusion

Python Debugging With pdb

PyGame: A Primer on Game Programming in Python

Background and Setup

Basic PyGame Program

PyGame Concepts

Initialization and Modules

Displays and Surfaces

Images and Rects

Basic Game Design

Importing and Initializing PyGame

Setting Up the Display

Setting Up the Game Loop

Processing Events

Drawing on the Screen

Using .blit() and .flip()

Sprites

Players

User Input

Enemies

Sprite Groups

Custom Events

Collision Detection

Sprite Images

Altering the Object Constructors

Adding Background Images

Game Speed

Sound Effects

A Note on Sources

Conclusion

Python vs C++: Selecting the Right Tool for the Job

Comparing Languages: Python vs C++

Compilation vs Virtual Machine

Syntax Differences

Whitespace

Boolean Expressions

Variables and Pointers

Comprehensions

Python’s std::algorithms

Static vs Dynamic Typing

Static Typing

Duck Typing

Templates

How to Use `.send()`

How to Use `.throw()`

How to Use `.close()`

Converting a Python String to an `int`

Converting a Python `int` to a String

Using `.blit()` and `.flip()`

Python’s `std::algorithms`

Comments Start With `#` in Python

Conditional Statements Use `elif` in Python

The Last Element of a Sequence Has Index `-1` in Python

Exponentiation Is Done With `**` in Python

The Length of a Sequence Is Found With `len()` in Python

Console Output Is Shown With `print()` in Python

Function Definitions Start With `def` and `return` Values in Python

There Are No `switch`/`case` Blocks in Python

The Most Recent Unassigned Result Is Available as `_` in Python

Anonymous Functions Are Created With the `lambda` Keyword in Python

Class Definitions Start With `class` in Python

A Class Refers to Itself as `self` in Python

You Should Not Import `*` From a Module in Python