Real Python

Python vs C++: Selecting the Right Tool for the Job

2019-09-11T14:00:00+00:00

Are you a C++ developer comparing Python vs C++? Are you looking at Python and wondering what all the fuss is about? Do you wonder how Python compares to the concepts you already know? Or perhaps you have a bet on who would win if you locked C++ and Python in a cage and let them battle it out? Then this article is for you!

In this article, you’ll learn about:

Differences and similarities when you’re comparing Python vs C++
Times when Python might be a better choice for a problem and vice versa
Resources to turn to as you have questions while learning Python

This article is aimed at C++ developers who are learning Python. It assumes a basic knowledge of both languages and will use concepts from Python 3.6 and up, as well as C++11 or later.

Let’s dive into looking at Python vs C++!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Comparing Languages: Python vs C++

Frequently, you’ll find articles that extoll the virtues of one programming language over another. Quite often, they devolve into efforts to promote one language by degrading the other. This isn’t that type of article.

When you’re comparing Python vs C++, remember that they’re both tools, and they both have uses for different problems. Think about comparing a hammer and a screwdriver. You could use a screwdriver to drive in nails, and you could use a hammer to force in screws, but neither experience will be all that effective.

Using the right tool for the job is important. In this article, you’ll learn about the features of Python and C++ that make each of them the right choice for certain types of problems. So, don’t view the “vs” in Python vs C++ as meaning “against.” Rather, think of it as a comparison.

Compilation vs Virtual Machine

Let’s start with the biggest difference when you’re comparing Python vs C++. In C++, you use a compiler that converts your source code into machine code and produces an executable. The executable is a separate file that can then be run as a stand-alone program:

This process outputs actual machine instructions for the specific processor and operating system it’s built for. In this drawing, it’s a Windows program. This means you’d have to recompile your program separately for Windows, Mac, and Linux:

You’ll likely need to modify your C++ code to run on those different systems as well.

Python, on the other hand, uses a different process. Now, remember that you’ll be looking at CPython which is the standard implementation for the language. Unless you’re doing something special, this is the Python you’re running.

Python runs each time you execute your program. It compiles your source just like the C++ compiler. The difference is that Python compiles to bytecode instead of native machine code. Bytecode is the native instruction code for the Python virtual machine. To speed up subsequent runs of your program, Python stores the bytecode in .pyc files:

If you’re using Python 2, then you’ll find these files next to the .py files. For Python 3, you’ll find them in a __pycache__ directory.

The generated bytecode doesn’t run natively on your processor. Instead, it’s run by the Python virtual machine. This is similar to the Java virtual machine or the .NET Common Runtime Environment. The initial run of your code will result in a compilation step. Then, the bytecode will be interpreted to run on your specific hardware:

As long as the program hasn’t been changed, each subsequent run will skip the compilation step and use the previously compiled bytecode to interpret:

Interpreting code is going to be slower than running native code directly on the hardware. So why does Python work that way? Well, interpreting the code in a virtual machine means that only the virtual machine needs to be compiled for a specific operating system on a specific processor. All of the Python code it runs will run on any machine that has Python.

Note: CPython is written in C, so it can run on most systems that have a C compiler.

Another feature of this cross-platform support is that Python’s extensive standard library is written to work on all operating systems.

Using pathlib, for example, will manage path separators for you whether you’re on Windows, Mac, or Linux. The developers of those libraries spent a lot of time making it portable so you don’t need to worry about it in your Python program!

Before you move on, let’s start keeping track of a Python vs C++ comparison chart. As you cover new comparisons, they’ll be added in italics:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x

Now that you’ve seen the differences in run time when you’re comparing Python vs C++, let’s dig into the specifics of the languages’ syntax.

Syntax Differences

Python and C++ share many syntactical similarities, but there are a few areas worth discussing:

Whitespace
Boolean expressions
Variables and pointers
Comprehensions

Let’s start with the most contentious one first: whitespace.

Whitespace

The first thing most developers notice when comparing Python vs C++ is the “whitespace issue.” Python uses leading whitespace to mark scope. This means that the body of an if block or other similar structure is indicated by the level of indentation. C++ uses curly braces ({}) to indicate the same idea.

While the Python lexer will accept any whitespace as long as you’re consistent, PEP8 (the official style guide for Python) specifies 4 spaces for each level of indentation. Most editors can be configured to do this automatically.

There has been an enormous amount of writing, shouting, and ranting about Python’s whitespace rules already, so let’s just jump past that issue and on to other matters.

Instead of relying on a lexical marker like ; to end each statement, Python uses the end of the line. If you need to extend a statement over a single line, then you can use the backslash (\) to indicate that. (Note that if you’re inside a set of parentheses, then the continuation character is not needed.)

There are people who are unhappy on both sides of the whitespace issue. Some Python developers love that you don’t have to type out braces and semicolons. Some C++ developers hate the reliance on formatting. Learning to be comfortable with both is your best bet.

Now that you’ve looked at the whitespace issue, let’s move on to one that’s a bit less contentious: Boolean expressions.

Boolean Expressions

The way you’ll use Boolean expressions changes slightly in Python vs C++. In C++, you can use numeric values to indicate true or false, in addition to the built-in values. Anything that evaluates to 0 is considered false, while every other numeric value is true.

Python has a similar concept but extends it to include other cases. The basics are quite similar. The Python documentation states that the following items evaluate to False:

Constants defined as false:
- None
- False
Zeros of any numeric type:
- 0
- 0.0
- 0j
- Decimal(0)
- Fraction(0, 1)
Empty sequences and collections:
- ''
- ()
- []
- {}
- set()
- range(0)

All other items are True. This means that an empty list [] is False, while a list containing only zero [0] is still True.

Most objects will evaluate to True, unless the object has __bool__() which returns False or __len__() which returns 0. This allows you to extend your custom classes to act as Boolean expressions.

Python has a few slight changes from C++ in the Boolean operators as well. For starters, if and while statements do not require the surrounding parentheses as they do in C++. Parentheses can aid in readability, however, so use your best judgment.

Most C++ Boolean operators have similar operators in Python:

C++ Operator	Python Operator
`&&`	`and`
`\|\|`	`or`
`!`	`not`
`&`	`&`
`\|`	`\|`

Most of the operators are similar to C++, but if you want to brush up you can read Operators and Expressions in Python.

Variables and Pointers

When you first start using Python after writing in C++, you might not give variables much thought. They seem to generally work as they do in C++. However, they’re not the same. Whereas in C++ you use variables to reference values, in Python you use names.

Note: For this section, where you’re looking at variables and names in Python vs C++, you’ll use variables for C++ and names for Python. Elsewhere, they will both be called variables.

First, let’s back up a bit and take a broader look at Python’s object model.

In Python, everything is an object. Numbers are held in objects. Modules are held in objects. Both the object of a class and the class itself are objects. Functions are also objects:

>>>

>>> a_list_object = list()
>>> a_list_object
[]
>>> a_class_object = list
>>> a_class_object
<class 'list'>
>>> def sayHi(name):
...      print(f'Hello, {name}')
...
>>> a_function_object = sayHi
>>> a_function_object
<function sayHi at 0x7faa326ac048>

Calling list() creates a new list object, which you assign to a_list_object. Using the name of the class list by itself places a label on the class object. You can place a new label on a function as well. This is a powerful tool and, like all powerful tools, it can be dangerous. (I’m looking at you, Mr. Chainsaw.)

Note: The code above is shown running in a REPL, which stands for “Read, Eval, Print Loop.” This interactive environment is used frequently to try out ideas in Python and other interpreted languages.

If you type python at a command prompt, then it will bring up a REPL where you can start typing in code and trying things out for yourself!

Moving back to the Python vs C++ discussion, note that this is behavior is different from what you’ll see in C++. Unlike Python, C++ has variables that are assigned to a memory location, and you must indicate how much memory that variable will use:

int an_int;
float a_big_array_of_floats[REALLY_BIG_NUMBER];

In Python, all objects are created in memory, and you apply labels to them. The labels themselves don’t have types, and they can be put on any type of object:

>>>

>>> my_flexible_name = 1
>>> my_flexible_name
1
>>> my_flexible_name = 'This is a string'
>>> my_flexible_name
'This is a string'
>>> my_flexible_name = [3, 'more info', 3.26]
>>> my_flexible_name
[3, 'more info', 3.26]
>>> my_flexible_name = print
>>> my_flexible_name
<built-in function print>

You can assign my_flexible_name to any type of object, and Python will just roll with it.

When you’re comparing Python vs C++, the difference in variables vs names can be a bit confusing, but it comes with some excellent benefits. One is that in Python you don’t have pointers, and you never need to think about heap vs stack issues. You’ll dive into memory management a bit later in this article.

Comprehensions

Python has a language feature called list comprehensions. While it’s possible to emulate list comprehensions in C++, it’s fairly tricky. In Python, they’re a basic tool that’s taught to beginning programmers.

One way of thinking about list comprehensions is that they’re like a super-charged initializer for lists, dicts, or sets. Given one iterable object, you can create a list, and filter or modify the original as you do so:

>>>

>>> [x**2 for x in range(5)]
[0, 1, 4, 9, 16]

This script starts with the iterable range(5) and creates a list that contains the square for each item in the iterable.

It’s possible to add conditions to the values in the first iterable:

>>>

>>> odd_squares = [x**2 for x in range(5) if x % 2]
>>> odd_squares
[1, 9]

The if x % 2 at the end of this comprehension limits the numbers used from range(5) to only the odd ones.

At this point you might be having two thoughts:

That’s a powerful syntax trick that will simplify some parts of my code.
You can do the same thing in C++.

While it’s true that you can create a vector of the squares of the odd numbers in C++, doing so usually means a little more code:

std::vector<int> odd_squares;
for (int ii = 0; ii < 10; ++ii) {
    if (ii % 2) {
        odd_squares.push_back(ii);
    }
}

For developers coming from C-style languages, list comprehensions are one of the first noticeable ways they can write more Pythonic code. Many developers start writing Python with C++ structure:

odd_squares = []
for ii in range(5):
    if (ii % 2):
        odd_squares.append(ii)

This is perfectly valid Python. It will likely run more slowly, however, and it’s not as clear and concise as the list comprehension. Learning to use list comprehensions will not only speed up your code, but it will also make your code more Pythonic and easier to read!

Note: When you’re reading about Python, you’ll frequently see the word Pythonic used to describe something. This is just a term the community uses to describe code that is clean, elegant and looks like it was written by a Python Jedi.

Python’s `std::algorithms`

C++ has a rich set of algorithms built into the standard library. Python has a similar set of built-in functions that cover the same ground.

The first and most powerful of these is the in operator, which provides a quite readable test to see if an item is included in a list, set, or dictionary:

>>>

>>> x = [1, 3, 6, 193]
>>> 6 in x
True
>>> 7 in x
False
>>> y = { 'Jim' : 'gray', 'Zoe' : 'blond', 'David' : 'brown' }
>>> 'Jim' in y
True
>>> 'Fred' in y
False
>>> 'gray' in y
False

Note that the in operator, when used on dictionaries, only tests for keys, not values. This is shown by the final test, 'gray' in y.

in can be combined with not for quite readable syntax:

if name not in y:
    print(f"{name} not found")

Next up in your parade of Python built-in operators is any. This is a boolean function that returns True if any element of the given iterable evaluates to True. This can seem a little silly until you remember your list comprehensions! Combining these two can produce powerful, clear syntax for many situations:

>>>

>>> my_big_list = [10, 23, 875]
>>> my_small_list = [1, 2, 8]
>>> any([x < 3 for x in my_big_list])
False
>>> any([x < 3 for x in my_small_list])
True

Finally, you have all, which is similar to any. This returns True only if—you guessed it—all of the elements in the iterable are True. Again, combining this with list comprehensions produces a powerful language feature:

>>>

>>> list_a = [1, 2, 9]
>>> list_b = [1, 3, 9]
>>> all([x % 2 for x in list_a])
False
>>> all([x % 2 for x in list_b])
True

any and all can cover much of the same ground where C++ developers would look to std::find or std::find_if.

Note: In the any and all examples above, you can remove the brackets ([]) without any loss of functionality. (for example: all(x % 2 for x in list_a)) This makes use of generator expressions which, while quite handy, are beyond the scope of this article.

Before moving on to variable typing, let’s update your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x

Okay, now you’re ready to look at variable and parameter typing. Let’s go!

Static vs Dynamic Typing

Another large topic when you’re comparing Python vs C++ is the use of data types. C++ is a statically typed language, while Python is dynamically typed. Let’s explore what that means.

Static Typing

C++ is statically typed, which means that each variable you use in your code must have a specific data type like int, char, float, and so forth. You can only assign values of the correct type to a variable, unless you jump through some hoops.

This has some advantages for both the developer and the compiler. The developer gains the advantage of knowing what the type of a particular variable is ahead of time, and therefore which operations are allowed. The compiler can use the type information to optimize the code, making it smaller, faster, or both.

This advance knowledge comes at a cost, however. The parameters passed into a function must match the type expected by the function, which can reduce the flexibility and potential usefulness of the code.

Duck Typing

Dynamic typing is frequently referred to as duck typing. It’s an odd name, and you’ll read more about that in just a minute! But first, let’s start with an example. This function takes a file object and reads the first ten lines:

def read_ten(file_like_object):
    for line_number in range(10):
        x = file_like_object.readline()
        print(f"{line_number} = {x.strip()}")

To use this function, you’ll create a file object and pass it in:

with open("types.py") as f:
    read_ten(f)

This shows how the basic design of the function works. While this function was described as “reading the first ten lines from a file object,” there is nothing in Python that requires that file_like_object be a file. As long as the object passed in supports .readline(), the object can be of any type:

class Duck():
    def readline(self):
        return "quack"

my_duck = Duck()
read_ten(my_duck)

Calling read_ten() with a Duck object produces:

0 = quack
1 = quack
2 = quack
3 = quack
4 = quack
5 = quack
6 = quack
7 = quack
8 = quack
9 = quack

This is the essence of duck typing. The saying goes, “If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.”

In other words, if the object has the needed methods, then it’s acceptable to pass it in, regardless of the object’s type. Duck or dynamic typing gives you an enormous amount of flexibility, as it allows any type to be used where it meets the required interfaces.

However, there is a problem here. What happens if you pass in an object that doesn’t meet the required interface? For example, what if you pass in a number to read_ten(), like this: read_ten(3)?

This results in an exception being thrown. Unless you catch the exception, your program will blow up with a traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "duck_test.py", line 4, in read_ten
    x = file_like_object.readline()
AttributeError: 'int' object has no attribute 'readline'

Dynamic typing can be quite a powerful tool, but as you can see, you must use caution when employing it.

Note: Python and C++ are both considered strongly typed languages. Although C++ has a stronger type system, the details of this are generally not significant to someone learning Python.

Let’s move on to a feature that benefits from Python’s dynamic typing: templates.

Templates

Python doesn’t have templates like C++, but it generally doesn’t need them. In Python, everything is a subclass of a single base type. This is what allows you to create duck typing functions like the ones above.

The templating system in C++ allows you to create functions or algorithms that operate on multiple different types. This is quite powerful and can save you significant time and effort. However, it can also be a source of confusion and frustration, as compiler errors in templates can leave you baffled.

Being able to use duck typing instead of templates makes some things much easier. But this, too, can cause hard-to-detect issues. As in all complex decisions, there are trade-offs when you’re comparing Python vs C++.

Type Checking

There’s been a lot of interest and discussion in the Python community lately about static type checking in Python. Projects like mypy have raised the possibility of adding pre-runtime type checking to specific spots in the language. This can be quite useful in managing interfaces between portions of large packages or specific APIs.

It helps to address one of the downsides of duck typing. For developers using a function, it helps if they can fully understand what each parameter needs to be. This can be useful on large project teams where many developers need to communicate through APIs.

Once again, let’s take a look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x

Now you’re ready to move on to differences in object-oriented programming.

Object-Oriented Programming

Like C++, Python supports an object-oriented programming model. Many of the same concepts you learned in C++ carry over into Python. You’ll still need to make decisions about inheritance, composition, and multiple inheritance.

Similarities

Inheritance between classes works similarly in Python vs C++. A new class can inherit methods and attributes from one or more base classes, just like you’ve seen in C++. Some of the details are a bit different, however.

Base classes in Python do not have their constructor called automatically like they do in C++. This can be confusing when you’re switching languages.

Multiple inheritance also works in Python, and it has just as many quirks and strange rules as it does in C++.

Similarly, you can also use composition to build classes, where you have objects of one type hold other types. Considering everything is an object in Python, this means that classes can hold anything else in the language.

Differences

There are some differences, however, when you’re comparing Python vs C++. The first two are related.

The first difference is that Python has no concept of access modifiers for classes. Everything in a class object is public. The Python community has developed a convention that any member of a class starting with a single underscore is treated as private. This is in no way enforced by the language, but it seems to work out pretty well.

The fact that every class member and method is public in Python leads to the second difference: Python has far weaker encapsulation support than C++.

As mentioned, the single underscore convention makes this far less of an issue in practical codebases than it is in a theoretical sense. In general, any user that breaks this rule and depends on the internal workings of a class is asking for trouble.

Operator Overloads vs Dunder Methods

In C++, you can add operator overloads. These allow you to define the behavior of specific syntactical operators (like ==) for certain data types. Usually, this is used to add more natural usage of your classes. For the == operator, you can define exactly what it means for two objects of a class to be equal.

One difference that takes some developers a long time to grasp is how to work around the lack of operator overloads in Python. It’s great that Python’s objects all work in any of the standard containers, but what if you want the == operator to do a deep comparison between two objects of your new class? In C++, you would create an operator==() in your class and do the comparison.

Python has a similar structure that’s used quite consistently across the language: dunder methods. Dunder methods get their name because they all start and end with a double underscore, or “d-under.”

Many of the built-in functions that operate on objects in Python are handled by calls to that object’s dunder methods. For your example above, you can add __eq__() to your class to do whatever fancy comparison you like:

class MyFancyComparisonClass():
    def __eq__(self, other):
        return True

This produces a class that compares the same way as any other instance of its class. Not particularly useful, but it demonstrates the point.

There are a large number of dunder methods used in Python, and the built-in functions make use of them extensively. For example, adding __lt__() will allow Python to compare the relative order of two of your objects. This means that not only will the < operator now work, but that >, <=, and >= will also work as well.

Even better, if you have several objects of your new class in a list, then you can use sorted() on the list and they’ll be sorted using __lt__().

Once again, let’s take a look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x

Now that you’ve seen object-oriented coding across both languages, let’s look at how Python and C++ manage those objects in memory.

Memory Management

One of the biggest differences, when you’re comparing Python vs C++, is how they handle memory. As you saw in the section about variables in C++ and Python’s names, Python does not have pointers, nor does it easily let you manipulate memory directly. While there are times when you want to have that level of control, most of the time it’s not necessary.

Giving up direct control of memory locations brings a few benefits. You don’t need to worry about memory ownership, or making sure that memory is freed once (and only once) after it’s been allocated. You also never have to worry about whether or not an object was allocated on the stack or the heap, which tends to trip up beginning C++ developers.

Python manages all of these issues for you. To do this everything in Python is a derived class from Python’s object. This allows the Python interpreter to implement reference counting as a means of keeping track of which objects are still in use and which can be freed.

This convenience comes at a price, of course. To free allocated memory objects for you, Python will occasionally need to run what is called a garbage collector, which finds unused memory objects and frees them.

Note: CPython has a complex memory management scheme, which means that freeing memory doesn’t necessarily mean the memory gets returned to the operating system.

Python uses two tools to free memory:

The reference counting collector
The generational collector

Let’s look at each of these individually.

Reference Counting Collector

The reference counting collector is fundamental to the standard Python interpreter and is always running. It works by keeping track of how many times a given block of memory (which is always a Python object) has a name attached to it while your program is running. Many rules describe when the reference count is incremented or decremented, but an example of one case might clarify:

>>>

 1 >>> x = 'A long string'
 2 >>> y = x
 3 >>> del x
 4 >>> del y

In the above example, line 1 creates a new object containing the string "A long string". It then places the name x on this object, increasing the object’s reference count to 1:

On line 2 it assigns y to name the same object, which will increase the reference count to 2:

When you call del with x in line 3, you’re removing one of the references to the object, dropping the count back to 1:

Finally, when you remove y, the final reference to the object, its reference count drops to zero and it can be freed by the reference counting garbage collector. It may or may not be freed immediately at this point, but generally, that shouldn’t matter to the developer:

While this will take care of finding and freeing many of the objects that need to be freed, there are a few situations it will not catch. For that, you need the generational garbage collector.

Generational Garbage Collector

One of the big holes in the reference counting scheme is that your program can build a cycle of references, where object A has a reference to object B, which has a reference back to object A. It’s entirely possible to hit this situation and have nothing in your code referring to either object. In this case, neither of the objects will ever hit a reference count of 0.

The generational garbage collector involves a complex algorithm that is beyond the scope of this article, but it will find some of these orphaned reference cycles and free them for you. It runs on an occasional basis controlled by settings described in the documentation. One of these parameters is to disable this garbage collector entirely.

When You Don’t Want Garbage Collection

When you’re comparing Python vs C++, as when you’re comparing any two tools, each advantage comes with a trade-off. Python doesn’t require explicit memory management, but occasionally it will spend a longer amount of time than expected on garbage collection. The inverse is true for C++: your program will have consistent response times, but you’ll need to expend more effort in managing memory.

In many programs the occasional garbage collection hit is unimportant. If you’re writing a script that only runs for 10 seconds, then you’re unlikely to notice the difference. Some situations, however, require consistent response times. Real-time systems are a great example, where responding to a piece of hardware in a fixed amount of time can be essential to the proper operation of your system.

Systems with hard real-time requirements are some of the systems for which Python is a poor language choice. Having a tightly controlled system where you’re certain of the timing is a good use of C++. These are the types of issues to consider when you’re deciding on the language for a project.

Time to update your Python vs C++ chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x
Direct Memory Control		x
Garbage Collection	x

Threading, Multiprocessing, and Async IO

The concurrency models in C++ and Python are similar, but they have different results and benefits. Both languages have support for threading, multiprocessing, and Async IO operations. Let’s look at each of these.

Threading

While both C++ and Python have threading built into the language, the results can be markedly different, depending on the problem you’re solving. Frequently, threading is used to address performance problems. In C++, threading can provide a general speed-up for both computationally bound and I/O bound problems, as threads can take full advantage of the cores on a multiprocessor system.

Python, on the other hand, has made a design trade-off to use the Global Interpreter Lock, or the GIL, to simplify its threading implementation. There are many benefits to the GIL, but the drawback is that only one thread will be running at a single time, even if there are multiple cores.

If your problem is I/O bound, like fetching several web pages at once, then this limitation will not bother you in the least. You’ll appreciate Python’s easier threading model and built-in methods for inter-thread communications. If your problem is CPU-bound, however, then the GIL will restrict your performance to that of a single processor. Fortunately, Python’s multiprocessing library has a similar interface to its threading library.

Multiprocessing

Multiprocessing support in Python is built into the standard library. It has a clean interface that allows you to spin up multiple processes and share information between them. You can create a pool of processes and spread work across them using several techniques.

While Python still uses similar OS primitives to create the new processes, much of the low-level complication is hidden from the developer.

C++ relies on fork() to provide multiprocessing support. While this gives you direct access to all of the controls and issues of spawning multiple processes, it’s also much more complex.

Async IO

While both Python and C++ support Async IO routines, they’re handled differently. In C++, the std::async methods are likely to use threading to achieve the Async IO nature of their operations. In Python, Async IO code will only run on a single thread.

There are trade-offs here as well. Using separate threads allows the C++ Async IO code to perform faster on computationally bound problems. The Python tasks used in its Async IO implementation are more lightweight, so it’s faster to spin up a large number of them to handle I/O bound issues.

Your Python vs C++ comparison chart remains unchanged for this section. Both languages support a full range of concurrency options, with varying trade-offs between speed and convenience.

Miscellaneous Issues

If you’re comparing Python vs C++ and looking at adding Python to your toolbelt, then there are a few other things to consider. While your current editor or IDE will certainly work for Python, you might want to add certain extensions or language packs. It’s also worth giving PyCharm a look, as it’s Python-specific.

Several C++ projects have Python bindings. Things like Qt, WxWidgets, and many messaging APIs having multiple-language bindings.

If you want to embed Python in C++, then you can use the Python/C API.

Finally, there are several methods for using your C++ skills to extend Python and add functionality, or to call your existing C++ libraries from within your Python code. Tools like CTypes, Cython, CFFI, Boost.Python and Swig can help you combine these languages and use each for what it’s best at.

Summary: Python vs C++

You’ve spent some time reading and thinking about the differences between Python vs C++. While Python has easier syntax and fewer sharp edges, it’s not a perfect fit for all problems. You’ve looked at the syntax, memory management, processing, and several other aspects of these two languages.

Let’s take a final look at your Python vs C++ comparison chart:

Feature	Python	C++
Faster Execution		x
Cross-Platform Execution	x
Single-Type Variables		x
Multiple-Type Variables	x
Comprehensions	x
Rich Set of Built-In Algorithms	x	x
Static Typing		x
Dynamic Typing	x
Strict Encapsulation		x
Direct Memory Control		x
Garbage Collection	x

If you’re comparing Python vs C++, then you can see from your chart that this is not a case where one is better than the other. Each of them is a tool that’s well crafted for various use cases. Just like you don’t use a hammer for driving screws, using the right language for the job will make your life easier!

Conclusion

Congrats! You’ve now seen some of the strengths and weaknesses of both Python and C++. You’ve learned some of the features of each language and how they are similar.

You’ve seen that C++ is great when you want:

Fast execution speed (potentially at the cost of development speed)
Complete control of memory

Conversely, Python is great when you want:

Fast development speed (potentially at the cost of execution speed)
Managed memory

You’re now ready to make a wise language choice when it comes to your next project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Absolute vs Relative Imports in Python

2019-09-10T14:00:00+00:00

If you’ve worked on a Python project that has more than one file, chances are you’ve had to use an import statement before. Even for Pythonistas with a couple of projects under their belt, imports can be confusing!

If you’re reading this because you’d like to gain a deeper understanding of imports in Python, particularly absolute and relative imports, then you’ve come to the right place! In this tutorial, you’ll learn the differences between the two, as well as their pros and cons.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

MATLAB vs Python: Why and How to Make the Switch

2019-09-09T14:00:00+00:00

MATLAB® is widely known as a high-quality environment for any work that involves arrays, matrices, or linear algebra. Python is newer to this arena but is becoming increasingly popular for similar tasks. As you’ll see in this article, Python has all of the computational power of MATLAB for science tasks and makes it fast and easy to develop robust applications. However, there are some important differences when comparing MATLAB vs Python that you’ll need to learn about to effectively switch over.

In this article, you’ll learn how to:

Evaluate the differences of using MATLAB vs Python
Set up an environment for Python that duplicates the majority of MATLAB functions
Convert scripts from MATLAB to Python
Avoid common issues you might have when switching from MATLAB to Python
Write code that looks and feels like Python

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

MATLAB vs Python: Comparing Features and Philosophy

Python is a high-level, general-purpose programming language designed for ease of use by human beings accomplishing all sorts of tasks. Python was created by Guido van Rossum and first released in the early 1990s. Python is a mature language developed by hundreds of collaborators around the world.

Python is used by developers working on small, personal projects all the way up to some of the largest internet companies in the world. Not only does Python run Reddit and Dropbox, but the original Google algorithm was written in Python. Also, the Python-based Django Framework runs Instagram and many other websites. On the science and engineering side, the data to create the 2019 photo of a black hole was processed in Python, and major companies like Netflix use Python in their data analytics work.

There is also an important philosophical difference in the MATLAB vs Python comparison. MATLAB is proprietary, closed-source software. For most people, a license to use MATLAB is quite expensive, which means that if you have code in MATLAB, then only people who can afford a license will be able to run it. Plus, users are charged for each additional toolbox they want to install to extend the basic functionality of MATLAB. Aside from the cost, the MATLAB language is developed exclusively by Mathworks. If Mathworks were ever to go out of business, then MATLAB would no longer be able to be developed and might eventually stop functioning.

On the other hand, Python is free and open-source software. Not only can you download Python at no cost, but you can also download, look at, and modify the source code as well. This is a big advantage for Python because it means that anyone can pick up the development of the language if the current developers were unable to continue for some reason.

If you’re a researcher or scientist, then using open-source software has some pretty big benefits. Paul Romer, the 2018 Nobel Laureate in Economics, is a recent convert to Python. By his estimation, switching to open-source software in general, and Python in particular, brought greater integrity and accountability to his research. This was because all of the code could be shared and run by any interested reader. Prof. Romer wrote an excellent article, Jupyter, Mathematica, and the Future of the Research Paper, about his experience with open-source software.

Moreover, since Python is available at no cost, a much broader audience can use the code you develop. As you’ll see a little later on in the article, Python has an awesome community that can help you get started with the language and advance your knowledge. There are tens of thousands of tutorials, articles, and books all about Python software development. Here are a few to get you started:

Plus, with so many developers in the community, there are hundreds of thousands of free packages to accomplish many of the tasks that you’ll want to do with Python. You’ll learn more about how to get these packages later on in this article.

Like MATLAB, Python is an interpreted language. This means that Python code can be ported between all of the major operating system platforms and CPU architectures out there, with only small changes required for different platforms. There are distributions of Python for desktop and laptop CPUs and microcontrollers like Adafruit. Python can also talk to other microcontrollers like Arduino with a simple programming interface that is almost identical no matter the host operating system.

For all of these reasons, and many more, Python is an excellent choice to replace MATLAB as your programming language of choice. Now that you’re convinced to try out Python, read on to find out how to get it on your computer and how to switch from MATLAB!

Note: GNU Octave is a free and open-source clone of MATLAB. In this sense, GNU Octave has the same philosophical advantages that Python has around code reproducibility and access to the software.

Octave’s syntax is mostly compatible with MATLAB syntax, so it provides a short learning curve for MATLAB developers who want to use open-source software. However, Octave can’t match Python’s community or the number of different kinds of applications that Python can serve, so we definitely recommend you switch whole hog over to Python.

Besides, this website is called Real Python, not Real Octave 😀

Setting Up Your Environment for Python

In this section, you’ll learn:

How to install Python on your computer for a seamless transition from MATLAB
How to install replacements for the MATLAB integrated development environment
How to use the replacements for MATLAB on your computer

Getting Python via Anaconda

Python can be downloaded from a number of different sources, called distributions. For instance, the Python that you can download from the official Python website is one distribution. Another very popular Python distribution, particularly for math, science, engineering, and data science applications, is the Anaconda distribution.

There are two main reasons that Anaconda is so popular:

Anaconda distributes pre-built packages for Windows, macOS, and Linux, which means that the installation process is really easy and the same for all three major platforms.
Anaconda includes all of the most popular packages for engineering and data science type workloads in one single installer.

For the purposes of creating an environment that is very similar to MATLAB, you should download and install Anaconda. As of this writing, there are two major versions of Python available: Python 2 and Python 3. You should definitely install the version of Anaconda for Python 3, since Python 2 will not be supported past January 1, 2020. Python 3.7 is the most recent version at the time of this writing, but Python 3.8 should be out a few months after this article is published. Either 3.7 or 3.8 will work the same for you, so choose the most recent version you can.

Once you have downloaded the Anaconda installer, you can follow the default set up procedures depending on your platform. You should install Anaconda in a directory that does not require administrator permission to modify, which is the default setting in the installer.

With Anaconda installed, there are a few specific programs you should know about. The easiest way to launch applications is to use the Anaconda Navigator. On Windows, you can find this in the Start Menu and on macOS you can find it in Launchpad. Here’s a screenshot of the Anaconda Navigator on Windows:

In the screenshot, you can see several installed applications, including JupyterLab, Jupyter Notebook, and Spyder, that you’ll learn more about later in this tutorial.

On Windows, there is one other application that you should know about. This is called Anaconda Prompt, and it is a command prompt set up specifically to work with conda on Windows. If you want to type conda commands in a terminal, rather than using the Navigator GUI, then you should use Anaconda Prompt on Windows.

On macOS, you can use any terminal application such as the default Terminal.app or iTerm2 to access conda from the command line. On Linux, you can use the terminal emulator of your choice and which specific emulator is installed will depend on your Linux distribution.

Terminology Note: You may be a little bit confused about conda versus Anaconda. The distinction is subtle but important. Anaconda is a distribution of Python that includes many of the necessary packages for scientific work of all kinds. conda is a cross-platform package management software that is included with the Anaconda distribution of Python. conda is the software that you use to build, install, and remove packages within the Anaconda distribution.

You can read all about how to use conda in Setting Up Python for Machine Learning on Windows. Although that tutorial focuses on Windows, the conda commands are the same on Windows, macOS, and Linux.

Python also includes another way to install packages, called pip. If you’re using Anaconda, you should always prefer to install packages using conda whenever possible. Sometimes, though, a package is only available with pip, and for those cases, you can read What Is Pip? A Guide for New Pythonistas.

Getting an Integrated Development Environment

One of the big advantages of MATLAB is that it includes a development environment with the software. This is the window that you’re most likely used to working in. There is a console in the center where you can type commands, a variable explorer on the right, and a directory listing on the left.

Unlike MATLAB, Python itself does not have a default development environment. It is up to each user to find one that fits their needs. Fortunately, Anaconda comes with two different integrated development environments (IDEs) that are similar to the MATLAB IDE to make your switch seamless. These are called Spyder and JupyterLab. In the next two sections, you’ll see a detailed introduction to Spyder and a brief overview of JupyterLab.

Spyder

Spyder is an IDE for Python that is developed specifically for scientific Python work. One of the really nice things about Spyder is that it has a mode specifically designed for people like you who are converting from MATLAB to Python. You’ll see that a little later on.

First, you should open Spyder. If you followed the instructions in the previous section, you can open Spyder using the Anaconda Navigator. Just find the Spyder icon and click the Launch button. You can also launch Spyder from the Start Menu if you’re using Windows or from Launchpad if you’re using macOS.

Changing the Default Window Layout in Spyder

The default window in Spyder looks like the image below. This is for version 3.3.4 of Spyder running on Windows 10. It should look quite similar on macOS or Linux:

Before you take a tour of the user interface, you can make the interface look a little more like MATLAB. In the View → Window layouts menu choose MATLAB layout. That will change the window automatically so it has the same areas that you’re used to from MATLAB, annotated on the figure below:

In the top left of the window is the File Explorer or directory listing. In this pane, you can find files that you want to edit or create new files and folders to work with.

In the top center is a file editor. In this editor, you can work on Python scripts that you want to save to re-run later on. By default, the editor opens a file called temp.py located in Spyder’s configuration directory. This file is meant as a temporary place to try things out before you save them in a file somewhere else on your computer.

In the bottom center is the console. Like in MATLAB, the console is where you can run commands to see what they do or when you want to debug some code. Variables created in the console are not saved if you close Spyder and open it up again. The console is technically running IPython by default.

Any commands that you type in the console will be logged into the history file in the bottom right pane of the window. Furthermore, any variables that you create in the console will be shown in the variable explorer in the top right pane.

Notice that you can adjust the size of any pane by putting your mouse over the divider between panes, clicking, and dragging the edge to the size that you want. You can close any of the panes by clicking the x in the top of the pane.

You can also break any pane out of the main window by clicking the button that looks like two windows in the top of the pane, right next to the x that closes the pane. When a pane is broken out of the main window, you can drag it around and rearrange it however you want. If you want to put the pane back in the main window, drag it with the mouse so a transparent blue or gray background appears and the neighboring panes resize, then let go and the pane will snap into place.

Once you have the panes arranged exactly how you want, you can ask Spyder to save the layout. Go to the View menu and find the Window layouts flyout again. Then click Save current layout and give it a name. This lets you reset to your preferred layout at any time if something gets changed by accident. You can also reset to one of the default configurations from this menu.

Running Statements in the Console in Spyder

In this section, you’re going to be writing some simple Python commands, but don’t worry if you don’t quite understand what they mean yet. You’ll learn more about Python syntax a little later on in this article. What you want to do right now is get a sense for how Spyder’s interface is similar to and different from the MATLAB interface.

You’ll be working a lot with the Spyder console in this article, so you should learn about how it works. In the console, you’ll see a line that starts with In [1]:, for input line 1. Spyder (really, the IPython console) numbers all of the input lines that you type. Since this is the first input you’re typing, the line number is 1. In the rest of this article, you’ll see references to “input line X,” where X is the number in the square brackets.

One of the first things I like to do with folks who are new to Python is show them the Zen of Python. This short poem gives you a sense of what Python is all about and how to approach working with Python.

To see the Zen of Python, type import this on input line 1 and then run the code by pressing Enter. You’ll see an output like below:

>>>

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This code has import this on input line 1. The output from running import this is to print the Zen of Python onto the console. We’ll return to several of the stanzas in this poem later on in the article.

In many of the code blocks in this article, you’ll see three greater-than signs (>>>) in the top right of the code block. If you click that, it will remove the input prompt and any output lines, so you can copy and paste the code right into your console.

Many Pythonistas maintain a healthy sense of humor. This is displayed in many places throughout the language, including the Zen of Python. For another one, in the Spyder console, type the following code, followed by Enter to run it:

>>>

In [2]: import antigravity

That statement will open your web browser to the webcomic called XKCD, specifically comic #353, where the author has discovered that Python has given him the ability to fly!

You’ve now successfully run your first two Python statements! Congratulations 😃🎉

If you look at the History Log, you should see the first two commands you typed in the console (import this and import antigravity). Let’s define some variables and do some basic arithmetic now. In the console, type the following statements, pressing Enter after each one:

>>>

In [3]: var_1 = 10
In [4]: var_2 = 20
In [5]: var_3 = var_1 + var_2
In [6]: var_3
Out[6]: 30

In this code, you defined 3 variables: var_1, var_2, and var_3. You assigned var_1 the value 10, var_2 the value 20, and var_3 the sum of var_1 and var_2. Then you showed the value of the var_3 variable by writing it as the only thing on the input line. The output from that statement is shown on the next Out line, and the number on the Out line matches the associated In line.

There are two main things for you to notice in these commands:

If a statement does not include an assignment (with an =), it is printed onto an Out line. In MATLAB, you would need to include a semicolon to suppress the output even from assignment statements, but that is not necessary in Python.
On input lines 3, 4, and 5, the Variable explorer in the top right pane updated.

After you run these three commands, your Variable explorer should look like the image below:

In this image, you can see a table with four columns:

Name shows the name that you gave to var_1, var_2, and var_3.
Type shows the Python type of the variable, in this case, all int for integer numbers.
Size shows the size of the data stored variable, which is more useful for lists and other data structures.
Value shows the current value of the variable.

Running Code in Files in Spyder

The last stop in our brief tour of the Spyder interface is the File editor pane. In this pane, you can create and edit Python scripts and run them using the console. By default, Spyder creates a temporary file called temp.py which is intended for you to temporarily store commands as you’re working before you move or save them in another file.

Let’s write some code into the temp.py file and see how to run it. The file starts with the following code, which you can just leave in place:

 1 # -*- coding: utf-8 -*-
 2 """
 3 Spyder Editor
 4 
 5 This is a temporary script file.
 6 """

In this code, you can see two Python syntax structures:

Line 1 has a comment. In Python, the comment character is the hash or pound sign (#). MATLAB uses the percent symbol (%) as the comment character. Anything following the hash on the line is a comment and is usually ignored by the Python interpreter.
Starting on line 2 is a string that provides some context for the contents of the file. This is often referred to as a documentation string or docstring for short. You’ll learn more about docstrings in a later section.

Now you can start adding code to this file. Starting on line 8 in temp.py, enter the following code that is similar to what you already typed in the console:

 8 var_4 = 10
 9 var_5 = 20
10 var_6 = var_4 + var_5

Then, there are three ways to run the code:

You can use the F5 keyboard shortcut to run the file just like in MATLAB.
You can click the green right-facing triangle in the menu bar just above the Editor and File explorer panes.
You can use the Run → Run menu option.

The first time you run a file, Spyder will open a dialog window asking you to confirm the options you want to use. For this test, the default options are fine and you can click Run at the bottom of the dialog box:

This will automatically execute the following code in the console:

>>>

In [7]: runfile('C:/Users/Eleanor/.spyder-py3/temp.py',
   ...:         wdir='C:/Users/Eleanor/.spyder-py3')

This code will run the file that you were working on. Notice that running the file added three variables into the Variable explorer: var_4, var_5, and var_6. These are the three variables that you defined in the file. You will also see runfile() added to the History log.

In Spyder, you can also create code cells that can be run individually. To create a code cell, add a line that starts with # %% into the file open in the editor:

11 # %% This is a code cell
12 var_7 = 42
13 var_8 = var_7 * 2
14 
15 # %% This is a second code cell
16 print("This code will be executed in this cell")

In this code, you have created your first code cell on line 11 with the # %% code. What follows is a line comment and is ignored by Python. On line 12, you are assigning var_7 to have the value 42 and then line 13 assigns var_8 to be var_7 times two. Line 15 starts another code cell that can be executed separately from the first one.

To execute the code cells, click the Run Current Cell or Run Current Cell and Go to the Next One buttons next to the generic Run button in the toolbar. You can also use the keyboard shortcuts Ctrl+Enter to run the current cell and leave it selected, or Shift+Enter to run the current cell and select the next cell.

Spyder also offers easy-to-use debugging features, just like in MATLAB. You can double-click any of the line numbers in the Editor to set a breakpoint in your code. You can run the code in debug mode using the blue right-facing triangle with two vertical lines from the toolbar, or the Ctrl+F5 keyboard shortcut. This will pause execution at any breakpoints you specify and open the ipdb debugger in the console which is an IPython-enhanced way to run the Python debugger pdb. You can read more in Python Debugging With pdb.

Summarizing Your Experience in Spyder

Now you have the basic tools to use Spyder as a replacement for the MATLAB integrated development environment. You know how to run code in the console or type code into a file and run the file. You also know where to look to see your directories and files, the variables that you’ve defined, and the history of the commands you typed.

Once you’re ready to start organizing your code into modules and packages, you can check out the following resources:

Spyder is a really big piece of software, and you’ve only just scratched the surface. You can learn a lot more about Spyder by reading the official documentation, the troubleshooting and FAQ guide, and the Spyder wiki.

JupyterLab

JupyterLab is an IDE developed by Project Jupyter. You may have heard of Jupyter Notebooks, particularly if you’re a data scientist. Well, JupyterLab is the next iteration of the Jupyter Notebook. Although at the time of this writing JupyterLab is still in beta, Project Jupyter expects that JupyterLab will eventually replace the current Notebook server interface. However, JupyterLab is fully compatible with existing Notebooks so the transition should be fairly seamless.

JupyterLab comes preinstalled with Anaconda, so you can launch it from the Anaconda Navigator. Find the JupyterLab box and click Launch. This will open your web browser to the address http://localhost:8888/lab.

The main JupyterLab window is shown in the picture below:

There are two main sections of the interface:

On the left is a File explorer that lets you open files from your computer.
On the right side of the window is how you can open create new Notebook files, work in an IPython console or system terminal, or create a new text file.

If you’re interested in learning more about JupyterLab, you can read a lot more about the next evolution of the Notebook in the blog post announcing the beta release or in the JupyterLab documentation. You can also learn about the Notebook interface in Jupyter Notebook: An Introduction and the Using Jupyter Notebooks course. One neat thing about the Jupyter Notebook-style document is that the code cells you created in Spyder are very similar to the code cells in a Jupyter Notebook.

Learning About Python’s Mathematical Libraries

Now you’ve got Python on your computer and you’ve got an IDE where you feel at home. So how do you learn about how to actually accomplish a task in Python? With MATLAB, you can use a search engine to find the topic you’re looking for just by including MATLAB in your query. With Python, you’ll usually get better search results if you can be a bit more specific in your query than just including Python.

In this section, you’ll take the next step to really feeling comfortable with Python by learning about how Python functionality is divided into several libraries. You’ll also learn what each library does so you can get top-notch results with your searches!

Python is sometimes called a batteries-included language. This means that most of the important functions you need are already included when you install Python. For instance, Python has a math library and a statistics library built-in that include the basic operations.

Sometimes, though, you want to do something that isn’t included in the language. One of the big advantages of Python is that someone else has probably done whatever you need to do and published the code to accomplish that task. There are several hundred-thousand publicly available and free packages that you can easily install to perform various tasks. These range from processing PDF files to building and hosting an interactive website to working with highly optimized mathematical and scientific functions.

Working with arrays or matrices, optimization, or plotting requires additional libraries to be installed. Fortunately, if you install Python with the Anaconda installer these libraries come preinstalled and you don’t need to worry. Even if you’re not using Anaconda, they are usually pretty easy to install for most operating systems.

The set of important libraries you’ll need to switch over from MATLAB are typically called the SciPy stack. At the base of the stack are libraries that provide fundamental array and matrix operations (NumPy), integration, optimization, signal processing, and linear algebra functions (SciPy), and plotting (Matplotlib). Other libraries that build on these to provide more advanced functionality include Pandas, scikit-learn, SymPy, and more.

NumPy (Numerical Python)

NumPy is probably the most fundamental package for scientific computing in Python. It provides a highly efficient interface to create and interact with multi-dimensional arrays. Nearly every other package in the SciPy stack uses or integrates with NumPy in some way.

NumPy arrays are the equivalent to the basic array data structure in MATLAB. With NumPy arrays, you can do things like inner and outer products, transposition, and element-wise operations. NumPy also contains a number of useful methods for reading text and binary data files, fitting polynomial functions, many mathematical functions (sine, cosine, square root, and so on), and generating random numbers.

The performance-sensitive parts of NumPy are all written in the C language, so they are very fast. NumPy can also take advantage of optimized linear algebra libraries such as Intel’s MKL or OpenBLAS to further increase performance.

Note:

Real Python has several articles that cover how you can use NumPy to speed up your Python code:

SciPy (Scientific Python)

The SciPy package (as distinct from the SciPy stack) is a library that provides a huge number of useful functions for scientific applications. If you need to do work that requires optimization, linear algebra or sparse linear algebra, discrete Fourier transforms, signal processing, physical constants, image processing, or numerical integration, then SciPy is the library for you! Since SciPy implements so many different features, it’s almost like having access to a bunch of the MATLAB toolboxes in one package.

SciPy relies heavily on NumPy arrays to do its work. Like NumPy, many of the algorithms in SciPy are implemented in C or Fortran, so they are also very fast. Also like NumPy, SciPy can take advantage of optimized linear algebra libraries to further improve performance.

Matplotlib (MATLAB-like Plotting Library)

Matplotlib is a library to produce high-quality and interactive two-dimensional plots. Matplotlib is designed to provide a plotting interface that is similar to the plot() function in MATLAB, so people switching from MATLAB should find it somewhat familiar. Although the core functions in Matplotlib are for 2-D data plots, there are extensions available that allow plotting in three dimensions with the mplot3d package, plotting geographic data with cartopy, and many more listed in the Matplotlib documentation.

Note:

Here are some more resources on Matplotlib:

Other Important Python Libraries

With NumPy, SciPy, and Matplotlib, you can switch a lot of your MATLAB code to Python. But there are a few more libraries that might be helpful to know about.

Pandas provides a DataFrame, an array with the ability to name rows and columns for easy access.
SymPy provides symbolic mathematics and a computer algebra system.
scikit-learn provides many functions related to machine learning tasks.
scikit-image provides functions related to image processing, compatible with the similar library in SciPy.
Tensorflow provides a common platform for many machine learning tasks.
Keras provides a library to generate neural networks.
multiprocessing provides a way to perform multi-process based parallelism. It’s built into Python.
Pint provides a unit library to conduct automatic conversion between physical unit systems.
PyTables provides a reader and writer for HDF5 format files.
PyMC3 provides Bayesian statistical modeling and probabilistic machine learning functionality.

Syntax Differences Between MATLAB® and Python

In this section, you’ll learn how to convert your MATLAB code into Python code. You’ll learn about the main syntax differences between MATLAB and Python, see an overview of basic array operations and how they differ between MATLAB and Python, and find out about some ways to attempt automatic conversion of your code.

The biggest technical difference between MATLAB and Python is that in MATLAB, everything is treated as an array, while in Python everything is a more general object. For instance, in MATLAB, strings are arrays of characters or arrays of strings, while in Python, strings have their own type of object called str. This has profound consequences for how you approach coding in each language, as you’ll see below.

With that out of the way, let’s get started! To help you, the sections below are organized into groups based on how likely you are to run into that syntax.

You Will Probably See This Syntax

The examples in this section represent code that you are very likely to see in the wild. These examples also demonstrate some of the more basic Python language features. You should make sure that you have a good grasp of these examples before moving on.

Comments Start With `#` in Python

In MATLAB, a comment is anything that follows a percent sign (%) on a line. In Python, comments are anything that follow the hash or pound sign (#). You already saw a Python comment in the earlier section about Spyder. In general, the Python interpreter ignores the content of comments, just like the MATLAB interpreter, so you can write whatever content you want in the comment. One exception to this rule in Python is the example you saw earlier in the section about Spyder:

# -*- coding: utf-8 -*-

When the Python interpreter reads this line, it will set the encoding that it uses to read the rest of the file. This comment must appear in one of the first two lines of the file to be valid.

Another difference between MATLAB and Python is in how inline documentation is written. In MATLAB, documentation is written at the start of a function in a comment, like the code sample below:

function [total] = addition(num_1,num_2)
% ADDITION  Adds two numbers together
%   TOTAL = ADDITION(NUM_1,NUM_2) adds NUM_1 and NUM_2 together
%
%   See also SUM and PLUS

However, Python does not use comments in this way. Instead, Python has an idea called documentation strings or docstrings for short. In Python, you would document the MATLAB function shown above like this:

def addition(num_1, num_2):
    """Adds two numbers together.

    Example
    -------
    >>> total = addition(10, 20)
    >>> total
    30

    """

Notice in this code that the docstring is between two sets of three quote characters ("""). This allows the docstring to run onto multiple lines with the whitespace and newlines preserved. The triple quote characters are a special case of string literals. Don’t worry too much about the syntax of defining a function yet. You’ll see more about that in a later section.

Whitespace at the Beginning of a Line Is Significant in Python

When you write code in MATLAB, blocks like if statements, for and while loops, and function definitions are finished with the end keyword. It is generally considered a good practice in MATLAB to indent the code within the blocks so that the code is visually grouped together, but it is not syntactically necessary.

For example, the following two blocks of code are functionally equivalent in MATLAB:

 1 num = 10;
 2 
 3 if num == 10
 4 disp("num is equal to 10")
 5 else
 6 disp("num is not equal to 10")
 7 end
 8 
 9 disp("I am now outside the if block")

In this code, you are first creating num to store the value 10 and then checking whether the value of num is equal to 10. If it is, you are displaying the phrase num is equal to 10 on the console from line 2. Otherwise, the else clause will kick in and display num is not equal to 10. Of course, if you run this code, you will see the num is equal to 10 output and then I am now outside the if block.

Now you should modify your code so it looks like the sample below:

 1 num = 10;
 2 
 3 if num == 10
 4     disp("num is equal to 10")
 5 else
 6     disp("num is not equal to 10")
 7 end
 8 
 9 disp("I am now outside the if block")

In this code, you have only changed lines 3 and 5 by adding some spaces or indentation in the front of the line. The code will perform identically to the previous example code, but with the indentation, it is much easier to tell what code goes in the if part of the statement and what code is in the else part of the statement.

In Python, indentation at the start of a line is used to delimit the beginning and end of class and function definitions, if statements, and for and while loops. There is no end keyword in Python. This means that indentation is very important in Python!

In addition, in Python the definition line of an if/else/elif statement, a for or while loop, a function, or a class is ended by a colon. In MATLAB, the colon is not used to end the line.

Consider this code example:

 1 num = 10
 2 
 3 if num == 10:
 4     print("num is equal to 10")
 5 else:
 6     print("num is not equal to 10")
 7 
 8 print("I am now outside the if block")

On the first line, you are defining num and setting its value to 10. On line 2, writing if num == 10: tests the value of num compared to 10. Notice the colon at the end of the line.

Next, line 3 must be indented in Python’s syntax. On that line, you are using print() to display some output to the console, in a similar way to disp() in MATLAB. You’ll read more about print() versus disp() in a later section.

On line 4, you are starting the else block. Notice that the e in the else keyword is vertically aligned with the i in the if keyword, and the line is ended by a colon. Because the else is dedented relative to print() on line 3, and because it is aligned with the if keyword, Python knows that the code within the if part of the block has finished and the else part is starting. Line 5 is indented by one level, so it forms the block of code to be executed when the else statement is satisfied.

Lastly, on line 6 you are printing a statement from outside the if/else block. This statement will be printed regardless of the value of num. Notice that the p in print() is vertically aligned with the i in if and the e in else. This is how Python knows that the code in the if/else block has ended. If you run the code above, Python will display num is equal to 10 followed by I am now outside the if block.

Now you should modify the code above to remove the indentation and see what happens. If you try to type the code without indentation into the Spyder/IPython console, you will get an IndentationError:

>>>

In [1]: num = 10
In [2]: if num == 10:
   ...: print("num is equal to 10")
  File "<ipython-input-2-f453ffd2bc4f>", line 2
    print("num is equal to 10")
        ^
IndentationError: expected an indented block

In this code, you first set the value of num to 10 and then tried to write the if statement without indentation. In fact, the IPython console is smart and automatically indents the line after the if statement for you, so you’ll have to delete the indentation to produce this error.

When you’re indenting your code, the official Python style guide called PEP 8 recommends using 4 space characters to represent one indentation level. Most text editors that are set up to work with Python files will automatically insert 4 spaces if you press the Tab key on your keyboard. You can choose to use the tab character for your code if you want, but you shouldn’t mix tabs and spaces or you’ll probably end up with a TabError if the indentation becomes mismatched.

Conditional Statements Use `elif` in Python

In MATLAB, you can construct conditional statements with if, elseif, and else. These kinds of statements allow you to control the flow of your program in response to different conditions.

You should try this idea out with the code below, and then compare the example of MATLAB vs Python for conditional statements:

 1 num = 10;
 2 if num == 10
 3     disp("num is equal to 10")
 4 elseif num == 20
 5     disp("num is equal to 20")
 6 else
 7     disp("num is neither 10 nor 20")
 8 end

In this code block, you are defining num to be equal to 10. Then you are checking if the value of num is 10, and if it is, using disp() to print output to the console. If num is 20, you are printing a different statement, and if num is neither 10 nor 20, you are printing the third statement.

In Python, the elseif keyword is replaced with elif:

 1 num = 10
 2 if num == 10:
 3     print("num is equal to 10")
 4 elif num == 20:
 5     print("num is equal to 20")
 6 else:
 7     print("num is neither 10 nor 20")

This code block is functionally equivalent to the previous MATLAB code block. There are 2 main differences. On line 4, elseif is replaced with elif, and there is no end statement to end the block. Instead, the if block ends when the next dedented line of code is found after the else. You can read more in the Python documentation for if statements.

Calling Functions and Indexing Sequences Use Different Brackets in Python

In MATLAB, when you want to call a function or when you want to index an array, you use round brackets (()), sometimes also called parentheses. Square brackets ([]) are used to create arrays.

You can test out the differences in MATLAB vs Python with the example code below:

>>>

>> arr = [10, 20, 30];
>> arr(1)

ans =

    10

>> sum(arr)

ans =

    60

In this code, you first create an array using the square brackets on the right side of the equal sign. Then, you retrieve the value of the first element by arr(1), using the round brackets as the indexing operator. On the third input line, you are calling sum() and using the round brackets to indicate the parameters that should be passed into sum(), in this case just arr. MATLAB computes the sum of the elements in arr and returns that result.

Python uses separate syntax for calling functions and indexing sequences. In Python, using round brackets means that a function should be executed and using square brackets will index a sequence:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[0]
Out[2]: 10

In [3]: sum(arr)
Out[3]: 60

In this code, you are defining a Python list on input line 1. Python lists have some important distinctions from arrays in MATLAB and arrays from the NumPy package. You can read more about Python lists in Lists and Tuples in Python, and you’ll learn more about NumPy arrays in a later section.

On the input line 2, you are displaying the value of the first element of the list with the indexing operation using square brackets. On input line 3, you are calling sum() using round brackets and passing in the list stored in arr. This results in the sum of the list elements being displayed on the last line. Notice that Python uses square brackets for indexing the list and round brackets for calling functions.

The First Index in a Sequence Is 0 in Python

In MATLAB, you can get the first value from an array by using 1 as the index. This style follows the natural numbering convention and starts how you would count the number of items in the sequence. You can try out the differences of MATLAB vs Python with this example:

>>>

>> arr = [10, 20, 30];
>> arr(1)

ans =

    10

>> arr(0)
Array indices must be positive integers or logical values.

In this code, you are creating an array with three numbers: 10, 20, and 30. Then you are displaying the value of the first element with the index 1, which is 10. Trying to access the zeroth element results in an error in MATLAB, as shown on the last two lines.

In Python, the index of the first element in a sequence is 0, not 1:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[0]
Out[2]: 10

In [3]: arr[1]
Out[3]: 20

In [4]: a_string = "a string"
In [5]: a_string[0]
Out[5]: 'a'

In [6]: a_string[1]
Out[6]: ' '

In this code, you are defining arr as a Python list with three elements on input line 1. On input line 2, you are displaying the value of the first element of the list, which has the index 0. Then you are displaying the second element of the list, which has the index 1.

On input lines 4, 5, and 6, you are defining a_string with the contents "a string" and then getting the first and second elements of the string. Notice that the second element (character) of the string is a space. This demonstrates a general Python feature, that many variable types operate as sequences and can be indexed, including lists, tuples, strings, and arrays.

The Last Element of a Sequence Has Index `-1` in Python

In MATLAB, you can get the last value from an array by using end as the index. This is really useful when you don’t know how long an array is, so you don’t know what number to access the last value.

Try out the differences in MATLAB vs Python with this example:

>>>

>> arr = [10, 20, 30];
>> arr(end)

ans =

    30

In this code, you are creating an array with three numbers, 10, 20, and 30. Then you are displaying the value of the last element with the index end, which is 30.

In Python, the last value in a sequence can be retrieved by using the index -1:

>>>

In [1]: arr = [10, 20, 30]
In [2]: arr[-1]
Out[2]: 30

In this code, you are defining a Python list with three elements on input line 1. On input line 2, you are displaying the value of the last element of the list, which has the index -1 and the value 30.

In fact, by using negative numbers as the index values you can work your way backwards through the sequence:

>>>

In [3]: arr[-2]
Out[3]: 20

In [4]: arr[-3]
Out[4]: 10

In this code, you are retrieving the second-to-last and third-to-last elements from the list, which have values of 20 and 10, respectively.

Exponentiation Is Done With `**` in Python

In MATLAB, when you want to raise a number to a power you use the caret operator (^). The caret operator is a binary operator that takes two numbers. Other binary operators include addition (+), subtraction (-), multiplication (*), and division (/), among others. The number on the left of the caret is the base and the number on the right is the exponent.

Try out the differences of MATLAB vs Python with this example:

>>>

>> 10^2

ans =

    100

In this code, you are raising 10 to the power of 2 using the caret resulting an answer of 100.

In Python, you use two asterisks (**) when you want to raise a number to a power:

>>>

In [1]: 10 ** 2
Out[1]: 100

In this code, you are raising 10 to the power of 2 using two asterisks resulting an answer of 100. Notice that there is no effect of including spaces on either side of the asterisks. In Python, the typical style is to have spaces on both sides of a binary operator.

The Length of a Sequence Is Found With `len()` in Python

In MATLAB, you can get the length of an array with length(). This function takes an array as the argument and returns back the size of the largest dimension in the array. You can see the basics of this function with this example:

>>>

>> length([10, 20, 30])

ans =

    3

>> length("a string")

ans =

    1

In this code, on the first input line you are finding the length of an array with 3 elements. As expected, length() returns an answer of 3. On the second input line, you are finding the length of the string array that contains one element. Notice that MATLAB implicitly creates a string array, even though you did not use the square brackets to indicate it is an array.

In Python, you can get the length of a sequence with len():

>>>

In [1]: len([10, 20, 30])
Out[1]: 3

In [2]: len("a string")
Out[2]: 8

In this code, on the input line 1 you are finding the length of a list with 3 elements. As expected, len() returns a length of 3. On input line 2, you are finding the length of a string as the input. In Python, strings are sequences and len() counts the number of characters in the string. In this case, a string has 8 characters.

Console Output Is Shown With `print()` in Python

In MATLAB, you can use disp(), fprintf(), and sprintf() to print the value of variables and other output to the console. disp() is the simplest of these commands but does not give you any control over the format of the output. fprintf() and sprintf() give you total control over the output format, but the format specification is a little bit complicated.

In Python, print() serves the same function as disp(). Unlike disp(), print() can send its output to a file similar to fprintf().

Python’s print() will display any number of arguments passed to it, separating them by a space in the output. This is different from disp() in MATLAB, which only takes one argument. The following example shows how Python’s print() can take any number of arguments, and the each argument is separated by a space in the output:

>>>

In [1]: val_1 = 10
In [2]: val_2 = 20
In [3]: str_1 = "any number of arguments"
In [4]: print(val_1, val_2, str_1)
10 20 any number of arguments

In this code, the input lines 1, 2, and 3 define val_1, val_2, and str_1, where val_1 and val_1 are integers, and str_1 is a string of text. On input line 4, you are printing the three variables using print(). The output below this line the value of the three variables are shown in the console output, separated by spaces.

You can control the separator used in the output between arguments to print() by using the sep keyword argument:

>>>

In [5]: print(val_1, val_2, str_1, sep="; ")
10; 20; any number of arguments

In this code, you are printing the same three variables but setting the separator to be a semicolon followed by a space. This separator is printed between the first and second and the second and third arguments, but not after the third argument. To control the character printed after the last value, you can use the end keyword argument to print():

>>>

In [6]: print(val_1, val_2, str_1, sep="; ", end=";")
10; 20; any number of arguments;

In this code, you have added the end keyword argument to print(), setting it to print a semicolon after the last value. This is shown in the output on line below the input.

Like disp() from MATLAB, print() cannot directly control the output format of variables. If you want more control over the format of the output, you should use f-strings or format(). In these strings, you can use very similar formatting style codes as fprintf() in MATLAB to format numbers. You can read a lot more about f-strings and other string formatting in Python 3’s f-Strings or the official documentation:

>>>

In [7]: print(f"Python's print() can handle {str_1}")
Python's print() can handle any number of arguments

In [8]: print(f"The value of val_1 = {val_1:8.3f}")
The value of val_1 =   10.000

In [9]: # The following line will only work in Python 3.8

In [10]: print(f"The value of {val_1=} and {val_2=}")
The value of val_1 = 10, and val_2 = 20

In this code, input line 7 includes an f-string, indicated by the f to start the string. This means that Python will substitute the value of any variables it encounters between {}, or curly braces, within the string. You can see that in the output, Python has replaced {str_1} with any number of arguments, the value of str_1. On input line 8, you are using some of the formatting syntax to print the value of val_1 as a floating point number with 8 columns in the output and 3 digits of precision.

On input line 10, a new feature that is coming in Python 3.8 (due out in late 2019) is demonstrated. If a variable name is immediately followed by an equals sign inside curly braces, the name of the variable and the value will be printed automatically.

You can take a deep dive into Python’s print() by checking out The Ultimate Guide to Python Print.

You Will Probably See These, but You Can Learn Them When You Need To

In this section, you’ll find examples of code that you’ll probably see in the wild, but you can wait a little while to understand them if you want. These examples use some intermediate features in Python but are still in the core of how Python works. Just like in the last section, you’ll see comparisons of the MATLAB vs Python syntax differences.

Function Definitions Start With `def` and `return` Values in Python

In MATLAB, you can define a function by placing the function keyword at the start of a line. This is followed by the name of any output variables, an equals (=) sign, then the name of the function and any input arguments in parentheses. Within the the function you have to assign to any variables you specified in the definition line as outputs. A simple example MATLAB function is shown below:

 1 function [total] = addition(num_1,num_2)
 2 total = num_1 + num_2;
 3 end

In this code, you see the function definition on line 1. There is only one output variable, called total, for this function. The name of the function is addition and it takes two arguments, which will be assigned the names num_1 and num_2 in the function body. Line 2 is the implementation of the function. The value of total is set equal to the sum of num_1 and num_2. The last line of the function is the end keyword that tells the MATLAB interpreter the definition of the function has finished.

To use this function in MATLAB, you should save it in a file called addition.m, matching the name of the function. Alternatively, it can be placed in file with other commands provided that the function definition is the last thing in the file and the file is not named addition.m. Then, you can run the function by typing the following code in the MATLAB console:

>>>

>> var_1 = 20;
>> var_2 = 10;
>> sum_of_vars = addition(var_1,var_2)

sum_of_vars =

    30

In this code, you have defined two variables called var_1 and var_2 that hold the values 20 and 10, respectively. Then you created a third variable called sum_of_vars that stores the output from addition(). Check out the Variable explorer, and you’ll see that sum_of_vars has the value 30, as expected. Notice that the name sum_of_vars did not have to be the same name as the output variable used in the function definition, which was total.

MATLAB does not require a function to provide an output value. In this case, you would remove the output variable and the equals sign from the function definition. Modify your addition.m file so that the code looks like this:

 1 function addition(num_1,num_2)
 2 total = num_1 + num_2;
 3 end

The only change in this code from the earlier code is that you deleted the [total] = from line 1, the other lines are exactly the same. Now if you try to assign the result of calling this function to a variable, MATLAB will generate an error in the console:

>>>

>> var_1 = 20;
>> var_2 = 10;
>> sum_of_vars = addition(var_1,var_2);
Error using addition
Too many output arguments.

In this code, you defined the same two variables var_1 and var_2 as before and called addition() in the same way as before. However, since addition() no longer specifies an output variable, MATLAB generates an error message that there are too many output arguments. Clicking on the word addition will open the definition of the function for you to edit or view the source code to fix the problem.

In Python, the def keyword starts a function definition. The def keyword must be followed by the name of the function and any arguments to the function inside parentheses, similar to MATLAB. The line with def must be ended with a colon (:).

Starting on the next line, the code that should be executed as part of the function must be indented one level. In Python, the function definition ends when a line of code starts at the same indentation level as the def keyword on the first line.

If your function returns some output back to the caller, Python does not require that you specify a name for an output variable. Instead, you use the return statement to send an output value from the function.

An equivalent function in Python to your first addition() example with an output variable is shown below:

 1 def addition(num_1, num_2):
 2     total = num_1 + num_2
 3     return total

In this code, you see the def keyword followed by the function name and the two arguments num_1 and num_2 on line 1. On line 2 you can see the creation of a new variable total to store the sum of num_1 and num_2, and on line 3 the value of total is returned to the point where this function was called. Notice that lines 2 and 3 are indented by 4 spaces because they make up the body of the function.

The variable that stores the sum of num_1 and num_2 can have any name, it doesn’t have to be called total. In fact, you don’t need to create a variable there at all. You can simplify your previous function definition by eliminating total and simply returning the value of num_1 + num_2:

 1 def addition(num_1, num_2):
 2     return num_1 + num_1

Line 1 in this code is the same as it was before, you have only changed line 2 and deleted line 3. Line 2 now computes the value of num_1 + num_2 and returns that value back to the caller of the function. Line 2 is indented by 4 spaces because it makes up the body of the function.

To use this function in Python, you do not need to save it in a file with a special name. You can place the function definition in any Python file, at any point in the file. There is no restriction that the function definition has to be last. In fact, you can even define functions right from the console, which is not possible in MATLAB.

Open Spyder and in the Console pane type:

>>>

In [1]: def addition(num_1, num_2):

On this line of code you are creating the function definition. In the Spyder/IPython console, once you start a function definition and press Enter, the start of the line becomes three dots and the cursor is automatically indented. Now you can type the remainder of the function definition. You’ll have to press Enter twice to complete the definition:

>>>

In [1]: def addition(num_1, num_2):
   ...:     return num_1 + num_2
   ...:

In this code, you have the definition of the function on the first line and the body of the function on the second line. The console automatically adds the ...: at the start of the lines to indicate these are continuation lines that apply to the function definition.

Once you’ve completed the definition, you can execute the function from the console as well. You should type this code:

>>>

In [2]: var_1 = 20
In [3]: var_2 = 10
In [4]: sum_of_vars = addition(var_1, var_2)
In [5]: sum_of_vars
Out[5]: 30

In this code, you first create two variables var_1 and var_2 that store the values you want to add together. Then, on input line 4, you assign sum_of_vars to the result that is returned from addition(). On input line 5, you are outputting the value of sum_of_vars to the console screen. This displays 30, the sum of 10 and 20.

In Python, if you do not explicitly put a return statement, your function will implicitly return the special value None. You should change your Python definition of addition() to see how this works. In the Spyder/IPython console, type the following:

>>>

In [6]: def addition(num_1, num_2):
   ...:     total = num_1 + num_2
   ...:

In this code, you have the same def line on input line 6. You have changed the first continuation line to assign the result of the addition to total instead of returning. Now you should see what happens when we execute this modified function:

>>>

In [7]: sum_of_vars = addition(var_1, var_2)
In [8]: sum_of_vars
In [9]:

In this code, on input line 7 you are assigning sum_of_vars to be the returned value from addition(). Then, on input line 8, you are showing the value of sum_of_vars on the console screen, just like before. This time though, there is no output! By default, Python prints nothing when it outputs a variable whose value is None. You can double check the value of the sum_of_vars variable by looking at the Variable explorer. In the Type column, it should list NoneType, telling you that sum_of_vars is the special None value.

Functions Accept Positional and Keyword Arguments in Python

In MATLAB, functions have input arguments specified on the first line, in the function definition. When you call a function in MATLAB, you can pass from zero up to the number of arguments that are specified. In the body of the function, you can check the number of input arguments the caller actually passed to execute different code. This is useful when you want different arguments to have different meaning, like in the example below:

 1 function [result] = addOrSubtract(num_1,num_2,subtract)
 2 % ADDORSUBTRACT  Add or subtract two value
 3 %   RESULT = addOrSubtract(NUM_1,NUM_2) adds NUM_1 and NUM_2 together
 4 %
 5 %   RESULT = addOrSubtract(NUM_1,NUM_2,true) subtracts NUM_2 from NUM_1
 6 
 7     switch nargin
 8         case 2
 9             result = num_1 + num_2;
10         case 3
11             result = num_1 - num_2;
12         otherwise
13             result = 0;
14     end
15 end

In this code, you are defining a function with three possible input arguments. On line 7, you are starting a switch/case block that determines how many input arguments were passed to the function by using the special variable nargin. This variable stores the actual number of arguments the caller passed into the function.

In your code above, you are defining three cases:

If the number of input arguments is 2, you are adding num_1 and num_2 together.
If the number of input arguments is 3, you are subtracting num_2 from num_1.
If fewer than 2 arguments are passed, the output will be 0.

If more than 3 arguments are passed, MATLAB will raise an error.

Now you should experiment with this function. Save the code above into a file called addOrSubtract.m and then on the MATLAB console, try the version with two input arguments:

>>>

>> addOrSubtract(10,20)

ans =
    30

In this code, you are calling addOrSubtract() with two arguments, so the arguments are added together, resulting in an answer of 30. Next, try calling addOrSubtract() with three arguments:

>>>

>> addOrSubtract(10,20,true)

ans =
   -10

In this code, you used three input arguments, and found that the second argument was subtracted from the first, resulting in an answer of -10. Third, try calling addOrSubtract() with one argument:

>>>

>> addOrSubtract(10)

ans =
     0

In this code, you used one input argument and found the answer was 0, because MATLAB only found one argument to the function and used the otherwise case. Finally, try calling addOrSubtract() with four arguments:

>>>

>> addOrSubtract(10,20,true,30)
Error using addOrSubtract
Too many input arguments.

In this code, you find that MATLAB raises an error because there were more input arguments passed than were defined in the function line.

There are four key takeaways from this example with MATLAB:

There is only one kind of argument in a function definition.
The meaning of an argument in the code is determined by its position in the function definition.
The maximum number of arguments that can be passed to a function is determined by the number of arguments specified in the function definition.
Any number of arguments up to the maximum can be passed by the caller.

In Python, there are two kinds of arguments you can specify when defining a function. These are required and optional arguments. The key difference between these is that required arguments must be passed when a function is called, while optional are given a default value in the function definition.

You can see the differences between these two styles in the next example:

 1 def add_or_subtract(num_1, num_2, subtract=False):
 2     """Add or subtract two numbers, depending on the value of subtract."""
 3     if subtract:
 4         return num_1 - num_2
 5     else:
 6         return num_1 + num_2

In this code, you are defining a function called add_or_subtract() that has three arguments: num_1, num_2, and subtract. In the function definition, you can see the two types of arguments. The first two arguments, num_1 and num_2, are required arguments.

The third argument, subtract, has a default value assigned to it by specifying a value after an equals sign in the function definition. This means that when the function is called, passing a value for subtract is optional. If no value is passed, the default as defined in the function definition line will be used. In this case, the default value is False.

In the body of the function, you are testing the value of subtract with the if statement to determine whether addition or subtraction should be performed. If subtract is True, num_2 will be subtracted from num_1. Otherwise, if subtract is False, then num_1 will be added to num_2. In either case, the result of the arithmetic operation will be returned to the caller.

In addition to the two types of arguments you can use when defining a function, there are two kinds of arguments you can specify when calling a function. These are called positional and keyword arguments. You can see the difference between these in the following example. First, try passing only two arguments to the function:

>>>

In [1]: add_or_subtract(10, 20)
Out[1]: 30

In this code, you passed only two arguments to add_or_subtract(), 10 and 20. In this case, you passed these values as positional arguments, and the meaning of the arguments is defined by their position in the function call.

Since only the two required arguments were passed, subtract will take on the default value, which is False. Therefore, 10 and 20 will be added together, which you can see on the output line. Next, try passing a value for subtract:

>>>

In [2]: add_or_subtract(10, 20, False)
Out[2]: 30
In [3]: add_or_subtract(10, 20, True)
Out[3]: -10

In this code, you passed three arguments to add_or_subtract(), with two different values for the subtract argument. First, you passed False on input line 2. The result was the addition of 10 and 20. Then, you passed True on input line 3, resulting in the difference between 10 and 20, or -10.

In these examples, you saw that it is possible in Python to define default values for arguments to a function. This means when you call the function, any arguments with default values are optional and do not have to be passed. If no value is passed for any default arguments, the default value will be used. However, you must pass a value for every argument without a default value. Otherwise, Python will raise an error:

>>>

In [4]: add_or_subtract(10)
Traceback (most recent call last):

  File "<ipython-input-4-f9d1f2ae4494>", line 1, in <module>
    add_or_subtract(10)

TypeError: add_or_subtract() missing 1 required positional argument: 'num_2'

In this code, you have only passed one of the two required arguments to add_or_subtract(), so Python raises a TypeError. The error message tells you that you did not pass a value for num_2, because it does not have a default value.

In these last three example, you have used positional arguments, so which parameter is assigned to the variables in the function depends on the order they are passed. There is another method to pass arguments to functions in Python, called keyword arguments. To use keyword arguments, you specify the name of the argument in the function call:

>>>

In [5]: add_or_subtract(num_1=10, num_2=20, subtract=True)
Out[5]: -10

In this code, you have used keyword arguments for all three arguments to add_or_subtract(). Keyword arguments are specified by stating the argument name, then an equals sign, then the value that argument should have. One of the big advantages of keyword arguments is that they make your code more explicit. (As the Zen of Python says, explicit is better than implicit.) However, they make the code somewhat longer, so it’s up to your judgement when to use keyword arguments or not.

Another benefit of keyword arguments is that they can be specified in any order:

>>>

In [6]: add_or_subtract(subtract=True, num_2=20, num_1=10)
Out[6]: -10

In this code, you have specified the three arguments for add_or_subtract() as keyword arguments, but the order is different from in the function definition. Nonetheless, Python connects the right variables together because they are specified as keywords instead of positional arguments.

You can also mix positional and keyword arguments together in the same function call. If positional and keyword arguments are mixed together, the positional arguments must be specified first, before any keyword arguments:

>>>

In [7]: add_or_subtract(10, 20, subtract=True)
Out[7]: -10

In this code, you have specified the values for num_1 and num_2 using positional arguments, and the value for subtract using a keyword argument. This is probably the most common case of using keyword arguments, because it provides a good balance between being explicit and being concise.

Finally, there is one last benefit of using keyword arguments and default values. Spyder, and other IDEs, provide introspection of function definitions. This will tell you the names of all of the defined function arguments, which ones have default arguments, and the value of the default arguments. This can save you time and make your code easier and faster to read.

There Are No `switch`/`case` Blocks in Python

In MATLAB, you can use switch/case blocks to execute code by checking the value of a variable for equality with some constants. This type of syntax is quite useful when you know you want to handle a few discrete cases. Try out a switch/case block with this example:

num = 10;
switch num
    case 10
        disp("num is 10")
    case 20
        disp("num is 20")
    otherwise
        disp("num is neither 10 nor 20")
end

In this code, you start by defining num and setting it equal to 10 and on the following lines you test the value of num. This code will result in the output num is 10 being displayed on the console, since num is equal to 10.

This syntax is an interesting comparison of MATLAB vs Python because Python does not have a similar syntax. Instead, you should use an if/elif/else block:

num = 10
if num == 10:
    print("num is 10")
elif num == 20:
    print("num is 20")
else:
    print("num is neither 10 nor 20")

In this code, you start by defining num and setting it equal to 10. On the next several lines you are writing an if/elif/else block to check the different values that you are interested in.

Namespaces Are One Honking Great Idea in Python

In MATLAB, all functions are found in a single scope. MATLAB has a defined search order for finding functions within the current scope. If you define your own function for something that MATLAB already includes, you may get unexpected behavior.

As you saw in the Zen of Python, namespaces are one honking great idea. Namespaces are a way to provide different scopes for names of functions, classes, and variables. This means you have to tell Python which library has the function you want to use. This is a good thing, especially in cases where multiple libraries provide the same function.

For instance, the built-in math library provides a square root function, as does the more advanced NumPy library. Without namespaces, it would be more difficult to tell Python which square root function you wanted to use.

To tell Python where a function is located, you first have to import the library, which creates the namespace for that library’s code. Then, when you want to use a function from the library, you tell Python which namespace to look in:

>>>

In [1]: import math
In [2]: math.sqrt(4)
Out[2]: 2.0

In this code, on input line 1 you imported the math library that is built-in to Python. Then, input line 2 computes the square root of 4 using the square root function from within the math library. The math.sqrt() line should be read as “from within math, find sqrt().”

The import keyword searches for the named library and binds the namespace to the same name as the library by default. You can read more about how Python searches for libraries in Python Modules and Packages – An Introduction.

You can also tell Python what name it should use for a library. For instance, it is very common to see numpy shortened to np with the following code:

>>>

In [3]: import numpy as np
In [4]: np.sqrt(4)
Out[4]: 2.0

In this code, input line 3 imports NumPy and tells Python to put the library into the np namespace. Then, whenever you want to use a function from NumPy, you use the np abbreviation to find that function. On input line 4, you are computing the square root of 4 again, but this time, using np.sqrt(). np.sqrt() should be read as “from within NumPy, find sqrt().”

There are two main caveats to using namespaces where you should be careful:

You should not name a variable with the same name as one of the functions built into Python. You can find a complete list of these functions in the Python documentation. The most common variable names that are also built-in functions and should not be used are dir, id, input, list, max, min, sum, str, type, and vars.
You should not name a Python file (one with the extension .py) with the same name as a library that you have installed. In other words, you should not create a Python file called math.py. This is because Python searches the current working directory first when it tries to import a library. If you have a file called math.py, that file will be found before the built-in math library and you will probably see an AttributeError.

The Most Recent Unassigned Result Is Available as `_` in Python

The MATLAB console uses ans to store the result of the most recent calculation if that result was not assigned to a variable. This is really useful when you forgot to assign the result of a calculation to a variable or when you just want to chain a few simple calculations together. To see the differences between MATLAB vs Python, try out this example:

>>>

>> sum([10, 20, 30])

ans =

    60

>> ans + 10

ans =

    70

In this code, you use sum() to calculate the sum of the array. Since there is no equal sign with a variable name on the left, MATLAB assigns the output from sum() to ans. You can then use that variable in further calculations, as you do here by adding 10 to the last result. Note that this will only work in the MATLAB console, not in a script file.

In the Python console (including the IPython/Spyder console), the output from the most recent calculation is stored in _ (the underscore character). Try the following code:

>>>

In [1]: sum([10, 20, 30])
Out[1]: 60

In [2]: _ + 10
Out[2]: 70

In this code, you use sum() to calculate the sum of the list. Since there is no equal sign with a variable name on the left, Python assigns the output from sum() to the underscore (_), in addition to printing it on the output line. You can then use that variable in further calculations, as you do here by adding 10 to the last result. Note that this will only work in the Python console, not in a script file.

In the IPython console, there is one additional feature enabled. You can append a number after the underscore to retrieve the result of any previous line. Try the following code:

>>>

In [3]: _1 + 20
Out[3]: 80

In [4]: _2 + 20
Out[4]: 90

In this code on input line 3 you are using _1 to mean the value of output line 1, the sum() line. To that result (60) you are adding 20, producing a result of 80. On input line 4, you are adding 20 to the value of output line 2, accessed with _2, so the result is 90.

Notice that the Spyder Variable explorer does not show this variable by default, whereas ans is shown in the MATLAB Variable explorer. In a few sections, you’ll see why the underscore isn’t shown by default and how you can see it.

Anonymous Functions Are Created With the `lambda` Keyword in Python

MATLAB uses the the at-symbol (@) to indicate that what follows is the definition of an anonymous function. Anonymous functions are functions that are not defined in a program file and do not use the function keyword. A program file is a MATLAB file with a filename ending in .m. Anonymous functions are limited to a single statement so they are intended for simple computations.

You can try out the differences of anonymous functions in MATLAB vs Python with this example:

>>>

>> sayHello = @(x) fprintf("Hello, %s\n",x);
>> sayHello("Eleanor")
Hello, Eleanor

In this code, the first input line defines the anonymous function with one input parameter, x. The body of the function follows, which uses fprintf() to format the input into a string. This function is assigned to sayHello. On the second input line, sayHello() is executed and passed "Eleanor" as the value. The result the string Hello, Eleanor printed on the console.

Anonymous functions are most often used when you need to pass one function into another function. In these cases, it is often not necessary to assign the function definition to a variable:

>>>

>> integral(@(x) x.^2,0,9)

ans =

    243

In this code, the first input line executes integral(), a function that computes the definite integral of a given function. The first argument to integral() must be a function, so this is a perfect place to use an anonymous function. Here, your anonymous function squares whatever the input value is. The other two arguments to integral() are the limits of the integration, such that the result of integrating x.^2 from 0 to 9 is 243.

Python uses the lambda keyword to define anonymous functions. Other than this syntax difference, anonymous functions work the same way in Python as in MATLAB:

>>>

In [1]: say_hello = lambda x: print(f"Hello, {x:s}")
In [2]: say_hello("Eleanor")
Hello, Eleanor

In this code, input line 1 defines the lambda function with one parameter, x. You use print() within the function definition to show an f-string with the value of the input parameter. The function is then stored in say_hello(). Input line 2 evaluates say_hello() with the input string "Eleanor" and produces the Hello, Eleanor output.

In Python, the official style guide called PEP 8 specifically disrecommends assigning lambda expressions to variable names, as you saw in the last example. If you want to give a function a name to refer to it several times, the preference is to use the def syntax and define a full function, even for one-line functions.

However, lambda functions are still useful when they are passed as arguments into another function:

>>>

In [3]: from scipy import integrate
In [4]: integrate.quad(lambda x: x ** 2, 0, 9)
Out[4]: (243.0, 2.6978419498391304e-12)

In this code, input line 3 imports the scipy.integrate library and stores it in the integrate namespace. On input line 4, you are using quad() from scipy.integrate to compute the integral using quadrature, very similar to integral() in MATLAB. The first argument to quad() is the function to be integrated, and you use a lambda function to specify that x ** 2 should be integrated. The second and third arguments to quad() specify that the integral should be conducted from 0 to 9.

You can see that the result on output line 4 has two values, 243.0 and 2.6978419498391304e-12. The first value is the result of the integration, and is equal to the result from MATLAB. The second value is an estimate of the absolute error in the result. An error this small is approximately the precision of the numbers used to store the result, so the answer is about as accurate as it can be.

You can read more about lambda in How to Use Python lambda Functions or watch the videos in the How to Use Python Lambda Functions course.

You Will Only Need These in Specialized Situations

In this section, the examples are more advanced concepts that you will need as you become more advanced in Python development. Some of the examples here deal with developing applications or higher-level code than the other sections. When you see these concepts in other code, you can dig in to them when you feel comfortable.

Class Definitions Start With `class` in Python

MATLAB has two ways to define a class. With the first way, you can put all of the class definition in a single file with the name of the class as the filename. Then within the file, you can use the classdef keyword to define the properties and methods that belong to the class.

With the second way, you can create a folder that starts with @ and has the same name as the class. In that folder, you can create a single file that has the same name as the class. The function definition in that file will be used as the class initializer, and it should call class() to instantiate the class. Methods of the class can be defined in other files in the same folder, where the name of each file must be the same as the name of the method.

Python only has one way to define a class, using the class keyword. Since Python uses indentation to find the end of the class definition, the entire definition must be contained in a single file:

class MyClass:
    # The rest of the class definition goes here

In this code, the first line defines the name of the class. It starts with the class keyword, followed by the name of the class and a colon. Underneath this line, all of the code that is part of the class definition (methods and attributes) must be indented. Once a line of code starts in the same column as the c in class, the class definition will be ended.

The second line in this code is a comment to note that the rest of the class definition would follow the class line.

As in all object-oriented code, Python classes can inherit from superclasses. The superclass of a given class can be given as a parameter in the class definition, as shown below:

class MyClass(MySuperClass):
    # The rest of the class definition goes here

In this code, the only change is that the name of the superclass is listed inside round brackets before the colon.

There Are No Private Properties or Methods in Python

MATLAB allows class properties and method to be set as one of four Access options:

public: Access to the property or method is unrestricted.
protected: Access to the property or method is only allowed in this class or subclasses.
private: Access to the property or method is only allowed in this class.
meta.class or {meta.class}: Access to the property or method is only allowed in the listed class or classes.

This allows you to specifically control the ways that a property or class method can be accessed.

In Python, there is no way to set a class or instance attribute or method as protected or private. All classes and class instances can have their attributes and methods changed at runtime. Python’s convention is that attributes and methods that start with an underscore (_) are intended to be private, or at least non-public. However, this convention is not enforced by any checks in the language and all attributes and methods can be modified by the user at runtime.

As you saw in the section about using the underscore to retrieve values in the console, the underscore is not visible by default in the Spyder Variable explorer. This is because Spyder and other tools respect the convention that underscore indicates something should be non-public. However, underscores can be shown in the Spyder Variable explorer if you click the gear icon in the top right of that pane and uncheck the Exclude private variables item. This will also show other non-public variables as well.

Python uses several special methods that start with a double-underscore (__), called dunder methods, to implement specific behavior for classes. The most commonly used dunder method is __init__(), which is the class initializer or constructor. You can read a lot more about dunder methods in Enriching Your Python Classes With Dunder (Magic, Special) Methods.

If you want more information about Python’s classes, you can read Object-Oriented Programming in Python vs Java. Even though that article is about Java, Java is similar to the MATLAB OOP paradigm in terms of the nature of attributes and methods.

A Class Refers to Itself as `self` in Python

MATLAB uses the name obj when a class wants to refer to the current instance of itself. The obj should be the first argument passed to an ordinary method. MATLAB also defines static methods that have no reference to the class instance.

Python uses the name self when a class wants to refer to the current instance of itself, but this is actually only a convention. You can call the first argument to an instance method any name you want, but self is the most common convention. Python also defines static methods that don’t take an argument of the class instance and class methods that take an argument of the class object instead of the instance. You can read more about instance, static, and class methods in Python’s Instance, Class, and Static Methods Demystified.

There Is One String Type in Python

In MATLAB, strings of characters are stored in string arrays when you use double quotes (") or in character arrays if you use single quotes ('). If you use both single and double quotes in an array assignment, the array will be promoted to a string array.

In character arrays, each character in the string occupies one column in the array. For multidimensional character arrays, each row of the array must have the same number of characters, which is to say, the same number of columns. This is shown in the example below:

>>>

 1 >> charArray = ['Real'; 'Python'];
 2 Dimensions of arrays being concatenated are not consistent.
 3 
 4 >> charArray = ['MATLAB'; 'Python'];
 5 >> size(charArray)
 6 
 7 ans =
 8 
 9      2     6

In this example, line 1 shows an attempt to define a 2-row character array using single quotes. However, the number of characters in Real is not the same as in Python, so MATLAB shows an error message that the dimensions are not consistent.

On line 4, you successfully create a character array, and on the third input line you are checking the size of the array. The output shows that there are 2 rows, as expected, and 6 columns, since the length of both MATLAB and Python are 6 characters.

This is not the case for string arrays. In string arrays, each string occupies one column in the array and each row in a multidimensional array must have the same number of strings, although each string can have different length. This is shown in the example below:

>>>

 1 >> stringArray = ["Real", "Python"; "Real"];
 2 Error using vertcat
 3 Dimensions of arrays being concatenated are not consistent.
 4 
 5 >> stringArray = ["Real"; "Python"];
 6 >> size(stringArray)
 7 
 8 ans =
 9 
10      2     1

In this code, line 1 shows an attempt to define a 2-row string array using double quotes. However, the number of strings on the first row (2) does not match the number of strings in the second row (1), so MATLAB raises an error.

On line 5, you successfully create a string array. Notice that even though the number of characters is different between Real and Python, MATLAB is able to create the string array. On line 6 you are checking the size of the string array, which shows that there are 2 rows and 1 column, as expected.

In Python, there is only one string literal type, called str. You can create a string literal using single quotes (') or double quotes ("), there is no difference between the two definitions. However, there are some good arguments to prefer double quotes when defining string literals in Python, which are well expressed by the Black code formatting library.

There is one additional way to define strings in Python, using triple-single-quotes (''') or triple-double-quotes ("""). This method of creating strings allows the strings to be defined across multiple lines with the newline characters retained. You can see an example of this in the section about comments and docstrings.

You can read a lot more about defining strings in Basic Data Types in Python and Strings and Character Data in Python.

You can generate similar data structures to the string arrays and character arrays in MATLAB using NumPy in Python. NumPy has several data types, or dtypes, that are related to strings. In Python 3, the default string dtype for arrays is a fixed-width Unicode string:

>>>

In [1]: import numpy as np
In [2]: arr = np.array(("Real", "Python"))
In [3]: arr
Out[3]: array(['Real', 'Python'], dtype='<U6')

In this code, you are importing the NumPy library on input line 1 and assigning it to the np abbreviation. On input line 2, you are creating a NumPy array with 2 string elements, Real and Python, and assigning the array to arr.

On input line 3, you are showing the value of arr. The output from the third line shows that arr is storing an array that has 2 elements, 'Real' and 'Python', as expected. Notice that although you defined the array with double-quote strings, Python is displaying them with single-quote strings. Remember that there is no difference between single and double quotes in Python.

Output line 3 also shows the dtype of the data in the array. For this array, the dtype is <U6. The three characters here represent the aspects of how the strings are arranged in memory. The < means that the byte order of the array is little endian. The U means the string is of the Unicode type. Finally, the 6 means the maximum length of an element is 6 characters. This was chosen as the length of the longest string in the input.

Notice that the string Real only has 4 characters. In NumPy string dtype arrays, elements can have fewer than the maximum number of characters without problems, but assigning to elements with strings that are longer than the maximum length will truncate the input:

>>>

In [4]: arr[0] = "My favorite language is Python"
In [5]: arr
Out[5]: array(['My fav', 'Python'], dtype='<U6')

In this code, you are attempting to reassign the first element of the array with the string My favorite language is Python. Clearly, this string is longer than 6 characters, so it is truncated to only 6 when it is assigned, My fav. (The space counts as 1 character.)

If you want to create an array that can hold strings of any length, you should pass the object dtype when you create the array:

>>>

In [6]: arr_2 = np.array(("Real", "Python"), dtype=object)
In [7]: arr_2
Out[7]: array(['Real', 'Python'], dtype=object)

In this code, you are creating a new array, arr_2, with two elements again, but this time you specified the dtype to be object, which you confirmed by showing the output on the console. Now you should see how the object dtype affects assigning long strings to an element:

>>>

In [8]: arr_2[0] = "My favorite language is Python"
In [9]: arr_2
Out[9]: array(['My favorite language is Python', 'Python'], dtype=object)

In this code, you are again assigning the first element of the array to have the value My favorite language is Python. You can see from the output line that that string is stored as the first element of the array without truncation, because the dtype is object. The disadvantage of using the object dtype is that it is usually much slower than the more specific U dtype, because it has to create a a whole Python object for each element rather than just a NumPy-optimized Unicode object.

One other difference you will notice from MATLAB is how the shape or size of the array is determined:

>>>

In [10]: arr.shape
Out[10]: (2,)
In [11]: arr_2.shape
Out[11]: (2,)

In this code, we are printing the shape of arr and arr_2. Notice that they both have the same shape, two elements in a one-dimensional array. This is similar to the string array from MATLAB, where each string counts as one element in the array. However, the fact that NumPy arrays with the U dtype have a fixed maximum size behaves more like the character array from MATLAB. You’ll see more about differences in how MATLAB and NumPy compute the shape of arrays in a later section.

Libraries Are Not Automatically Reloaded in Python

When executing a function or script, MATLAB will always use the most up-to-date copy of the file on the disk. Therefore, as you’re developing a script, you can run it in the console many times and new changes that you make will automatically be picked up.

Python operates somewhat differently. Remember that when you want to access code from a file, you have to import it into a namespace. When Python imports a file or module, it only reads the code the first time it is imported. This saves quite a bit of time if you’re importing the same file several times. However, if you’re testing your code in the interactive console prompt as you work on it, Python will not pick up any changes if you import it again.

When you’re developing a module, you have a few options to have Python reload your code when it is imported. If you’re using the Spyder IDE, this is not a problem at all, since Spyder has an automatic User Module Reloading feature enabled by default.

Otherwise, if you’re using the IPython console outside of Spyder, or the Jupyter Notebook, you can use a magic command defined in those interpreters called autoreload:

>>>

In [1]: %load_ext autoreload
In [2]: %autoreload 2

In this code, you are using the load_ext magic command to load the autoreload extension. In IPython and Jupyter Notebooks, commands prefixed with the percent sign % are magic commands. The autoreload extension defines the autoreload magic function, which you use on input line 2. You are passing the parameter 2 to the autoreload magic function, which means that all modules should be reloaded every time a line of code is executed.

An Overview of Basic Array Operations

As you have seen, Python does not include a high-speed library for arrays in its standard library. However, the excellent NumPy library is easily available if you install Anaconda. NumPy functions as the de facto array and matrix library for Python.

NumPy has two array-like types:

numpy.ndarray, also known as numpy.array
numpy.matrix

The main difference between these two types is that the ndarray can be any number of dimensions, while the matrix is limited to exactly two dimensions. For ndarray, all operations such as addition, subtraction, multiplication, exponentiation, and division operate element-wise. However, for the matrix type, operations like multiplication and exponentiation are matrix operations.

When you’re converting from MATLAB, the matrix type may seem more familiar. It offers similar behavior that you may be used to from MATLAB in terms of operation syntax. However, NumPy strongly recommends that you use the ndarray type because it is more flexible and because matrix will eventually be removed.

In the rest of this section, you will get to know the major differences between MATLAB and NumPy arrays. You can go in-depth on how to use NumPy arrays by reading Look Ma, No For-Loops: Array Programming With NumPy.

Basic Mathematical Operators Work Element-Wise in NumPy

MATLAB, with its heritage as a matrix scripting language, assumes that all arithmetic operators will be operating on arrays. Therefore, MATLAB treats the multiplication of matrices or vectors as matrix multiplication. Consider this example:

>>>

>> arr_1 = [1,2,3];
>> arr_2 = [4,5,6];
>> arr_1 * arr_2
Error using  *
Incorrect dimensions for matrix multiplication. Check that the number of
columns in the first matrix matches the number of rows in the second
matrix. To perform elementwise multiplication, use '.*'.

In this code, you are creating two 1x3 matrices, arr_1, and arr_2. Then, you are attempting to multiply them together. For these 1xN arrays, this is equivalent to taking the dot or scalar product. However, the scalar product only works when the left operand is 1xN and the right is Nx1, so MATLAB produces an error message and suggests the dot-star operator (.*) as the proper syntax for element-wise multiplication:

>>>

>> arr_1 .* arr_2

ans =

     4    10    18

In this code, you are performing the element-wise multiplication of arr_1 and arr_2. This multiplies the first element of arr_1 with the first element of arr_2 (4*1 = 4), second with second (2*5 = 10), and third with third (3*6 = 18).

To perform the scalar product, you can take the transpose of arr_2 to convert it to a 3x1 array:

>>>

>> arr_1 * transpose(arr_2)

ans =

    32

In this code, you are performing matrix multiplication with arr_1 and the transpose of arr_2. Note that you can use either transpose() or the quote operator (') to take the transpose of arr_2. Since arr_1 is 1x3 and transpose(arr_2) is 3x1, this results in the scalar, or dot, product.

With NumPy arrays, operations like multiplication with the asterisk (*) operate element-wise by default:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.array([1, 2, 3])
In [3]: arr_2 = np.array([4, 5, 6])

In [4]: arr_1 * arr_2
Out[4]: np.array([ 4, 10, 18])

In this code, you are first importing the NumPy package and assigning it to the name np. Then you are creating two one-dimensional arrays. Notice the syntax for creating arrays in NumPy. It starts with np.array(), which should be read as “from within np, find array().” Then, you have to pass a Python list or tuple to the array constructor that contains the elements of the array. In this case, you are passing a Python list, denoted by the square brackets.

Finally, on input line 4, you are multiplying arr_1 and arr_2. Notice that the result on output line 4 is another array with the elements 4, 10, and 18, the same result as the element-wise multiplication in MATLAB.

If you want to perform the dot or scalar product for two arrays in NumPy, you have two options. The preferred option is to use the matrix multiplication operator (@) added in Python 3.5. You may see some older code also use dot() from the NumPy library and pass the two arrays:

>>>

In [5]: arr_1 @ arr_2
Out[5]: 32

In [6]: np.dot(arr_1, arr_2)
Out[6]: 32

In this code, input line 5 uses the matrix multiplication operator to find the scalar product of arr_1 and arr_2. As expected, the result is 32. Input line 5 uses dot() and should be read as “from within np, find dot() and pass arr_1 and arr_2.” You can see that the result is identical.

Notice that NumPy did not require you to transpose arr_2 before performing the scalar product. You’ll learn more about this feature in the next section.

One-Dimensional Arrays Are Vectors in NumPy

As you saw in the last section, MATLAB insists that the dimensions of arrays align when performing matrix multiplication, while NumPy is a little bit more flexible. This is because of how one-dimensional arrays are treated in MATLAB versus in NumPy.

In MATLAB, every array always has at least two dimensions, even if only implicitly. You can see this by checking the size() of a single number:

>>>

>> size(1)

ans =

     1     1

Here, you are finding the size of the integer 1. You can see that the result is an array with 1 row and 1 column.

You can create row vectors or column vectors in MATLAB, and switch between them with the transpose operator (') or transpose():

>>>

>> arr_1 = [1,2,3];
>> arr_2 = [1;2;3];
>> arr_1' == arr_2

ans =

  3×1 logical array

   1
   1
   1

>> arr_2' == arr_1

ans =

  1×3 logical array

   1   1   1

In this code, you are creating two vectors: arr_1 and arr_2. Arrays with values in one dimension only are called vectors. arr_1 is a row vector because the elements are arranged in one row and three columns, whereas arr_2 is a column vector because the elements are arranged in three rows and one column. In MATLAB, elements are put into different columns by separating them with a comma in the assignment and elements are put into different rows by separating them with a semicolon.

Then, you are checking the equality of the transpose of arr_1 with arr_2, and you find that all of the elements are equal and the result is a column vector of logical values. Finally, you are checking the equality of the transpose of arr_2 with arr_1, and you find that all of the elements are equal and the result is a row vector of logical values.

You can see that in MATLAB, even vectors have two dimensions associated with them: rows and columns. When the transpose is performed, the rows are switched with the columns, and the shape of the array is changed. This means there are two types of vectors in MATLAB: row-vectors and column-vectors.

In NumPy, there are three types of one-dimensional arrays or vectors. The default is an N-element vector with only one dimension. This is different from the default in MATLAB, where every array has at least 2 dimensions. This one-dimensional vector in NumPy does not have a sense of rows and columns, since for a one-dimensional structure, it does not matter in general whether the elements are stored in rows or in columns, only how many elements there are.

You can see an example of creating this kind of array in the following example. In the next few examples, there are extra spaces added before and after parentheses to clarify the syntax. These spaces are usually not considered good Python style, but they’re in the example to help you see what’s going on:

>>>

In [1]: import numpy as np
In [2]: arr_vec = np.array( [1, 2, 3] )
In [3]: arr_vec.shape
Out[3]: (3,)

In this code, you are creating a default 3-element vector in NumPy. On input line 1, you import NumPy and make it available under np. On input line 2 you are creating the array and storing it in arr_vec. You are passing the list [1, 2, 3] to array(), where the list has 3 elements and none of the elements are themselves lists. This creates the 3-element array with only one dimension.

You can verify that this is the case by displaying the shape of the array, as shown on input line 3. That line should be read as “from within arr_vec (an array), find the shape.” The shape of the array is equivalent to size() in MATLAB. In this case, the shape is (3,), indicating there are three elements and only one dimension, since there is not a second number after the comma.

You can also create row-vectors and column-vectors in NumPy, analogous to the row-vectors and column-vectors in MATLAB. NumPy’s array() takes a flat list or a nested list as input. Using the flat list gets you a one-dimensional, N-element vector. By using the nested list, you can create arrays of any dimension that you want. A nested list means that there are one or more lists contained within an outer list. Here’s an example of a nested list:

[[1, 2, 3], [4, 5, 6]]

In this code, you see an outer list that has 2 elements. Each of these 2 elements of the outer list is another, nested, list that has three elements, the integers 1-3 and 4-6. In terms of arrays, you can think of the number of elements of each inner list as the number of columns, and the number of nested lists is the number of rows. This is easier to see if you change the formatting:

[
    [1, 2, 3],
    [4, 5, 6]
]

This code is still valid Python syntax, but it emphasizes how the inner lists are each a row of the array, and the number of elements in each inner list is the number of columns. In this case, we would have an array with 2 rows and 3 columns. We can use these nested lists to create row-vectors and column-vectors in NumPy arrays:

>>>

In [4]: arr_row = np.array( [[1, 2, 3]] )
In [5]: arr_row.shape
Out[5]: (1, 3)

In this code, you are creating a row array or vector by using a nested list. Input line 4 is passing [[1, 2, 3]] to array(). You can break out the formatting of this nested list to see how it looks:

[
    [1, 2, 3]
]

As you can see, there is one row in this nested list with three columns. On input line 5, you are displaying the shape of this array. As expected, the shape is (1, 3), or one row with three columns.

Finally, you can create a column array by including three nested lists in the input:

>>>

In [6]: arr_col = np.array( [[1], [2], [3]] )
In [7]: arr_col.shape
Out[7]: (3, 1)

In this code, input line 6 is passing [[1], [2], [3]] to the array constructor. You can break out the formatting of this nested list to see how it looks:

[
    [1],
    [2],
    [3]
]

As you can see, there are three rows in this nested list with one column each. On input line 7, you are displaying the shape of this array. As expected, the shape is (3, 1), or three rows with one column.

Since the general N-element vector has no sense of rows and columns, NumPy is able to shape the vector in whatever way makes sense for the operation being performed. You saw this in the last section, where the NumPy array did not need to be transposed to perform the scalar product, whereas the MATLAB array did need to be transposed.

Trying to take the transpose of the N-element vector does not change the shape of the array. You can take the transpose using either np.transpose() or the .T attribute of the array:

>>>

In [8]: arr_vec_transp = np.transpose(arr_vec)
In [9]: arr_vec_transp.shape
Out[9]: (3,)

In this code, you are taking the transpose of the N-element vector arr_vec and printing its shape. Notice that the shape is the same as the shape of the original arr_vec.

However, if you are using row-vectors and column-vectors, you will need to ensure that the dimensions are appropriate for the particular operation. For instance, trying to take the scalar product of the row vector with itself will result in an error:

>>>

In [10]: arr_row @ arr_row
Traceback (most recent call last):

  File "<ipython-input-10-2b447c0bc8d5>", line 1, in <module>
    arr_row @ arr_row

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,
with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 3)

In this code, trying to find the scalar product of the row vector with itself results in a ValueError informing you that the dimensions of the arrays are not aligned. Using dot() gives the same error but a slightly different message:

>>>

In [11]: np.dot(arr_row, arr_row)
Traceback (most recent call last):

  File "<ipython-input-11-d6e191b317ae>", line 1, in <module>
    np.dot(arr_row, arr_row)

ValueError: shapes (1,3) and (1,3) not aligned: 3 (dim 1) != 1 (dim 0)

In this code, you are using dot() from the np namespace to attempt to find the scalar product of two 1x3 row-vectors. Since this operation is not permitted, NumPy raises a ValueError, similar to the matrix multiplication operator.

Instead, you need to take the transpose of one of the arguments:

>>>

In [12]: arr_row.T
Out[12]:
array([[1],
       [2],
       [3]])
In [13]: sc_prod = arr_row @ arr_row.T
In [14]: sc_prod
Out[14]: array([[14]])

On input line 12, you are taking the transpose of the row vector to turn it into a column vector using the transpose attribute (.T). This is shown in corresponding output line, where the elements are arranged to form a column for printing purposes. Then, you are taking the scalar product of the vector with its transpose, producing an array with a single value, 14. Notice that this is a 1x1 array, so to access just the value, you need to access the first element in each dimension:

>>>

In [15]: sc_prod.shape
Out[15]: (1, 1)
In [16]: sc_prod[0, 0]
Out[16]: 14

In this code, you are verifying that the shape is 1x1, and then accessing the first element in each dimension located at the 0th index. Remember that Python uses 0 as the first index, not 1.

You can use the nested lists to create arrays of any shape that you want. To create a three-by-three array (two-dimensional), simply include three elements in each of your three nested lists:

>>>

In [17]: arr_2d = np.array( [[1, 2, 3], [4, 5, 6], [7, 8, 9]] )
In [18]: arr_2d.shape
Out[18]: (3, 3)

In this code, you have nested three lists with three elements each into the constructor. As shown by the shape, this produces a 3x3 array with the elements 1 through 9.

Creating Arrays Is Very Flexible in NumPy

MATLAB and NumPy both allow you to explicitly specify the specific elements in an array, as you have seen in the previous section. In addition to this direct creation of arrays, both MATLAB and NumPy support a number of other methods to create arrays without explicitly specifying each element. The NumPy project maintains a detailed list of the equivalent functions between MATLAB and NumPy.

Many functions operate identically between MATLAB and NumPy. This includes commonly used functions like linspace() and logspace() to generate evenly spaced data and ones() and zeros() to generate arrays of a given shape filled with ones and zeros, respectively. The full list of ways to create arrays in NumPy is listed in the official documentation.

The one big difference between MATLAB and NumPy in terms of array creation routines is that MATLAB supports simply using the colon to create an array, while NumPy does not. Instead, NumPy uses arange() to create an array between specified values.

In MATLAB, you can use a colon to create an array specification range. In general, you can use up to 2 colons in a specification. The syntax is as follows:

start : stop
start : step : stop

In this syntax, the, first method only uses one colon and specifies the start and stop values. The second method includes a second colon, where the value before the first colon is the start, the middle value is the step, and the last value is the stop.

Try out these examples to experiment with this syntax:

>>>

>> arr_1 = 1:6

arr_1 =

     1     2     3     4     5     6

>> size(arr_1)

ans =

     1     6

In this example, you are using the single colon with the start and stop to generate an array with the values from 1 to 6. You can see that when the step is omitted, it defaults to a value of 1. Notice that MATLAB includes both the start and the stop values in the array, and that the size of the array is 6 elements long. Next, change the value of the step size to create a new array:

>>>

>> arr_2 = 1:2:6

arr_2 =

     1     3     5

In this example, you are using the two colons syntax with the start, step, and stop. The start value is 1, the step is 2, and the stop value is 6, so MATLAB starts with 1, increments to 3, and then to 5. The next step would exceed the stop value, so MATLAB does not include the stop value in the array. Next, change the starting value to create another new array:

>>>

>> arr_3 = 2:2:6

arr_3 =

     2     4     6

In this example, you are again using the two colon method, but you are specifying the start value as 2 instead of 1. In this case, MATLAB starts at 2, increments to 4, increments to 6, and then has reached the stop value so does not go further. Notice that in this case, the the stop value of 6 is included in the array.

With NumPy, you can use arange() to create an array with specific start, stop, and step values. However, arange() has one big difference from MATLAB, which is that the stop value is not included in the resulting array. The reason for this is so that the size of the array is equal to stop - start for the default case of a step size of 1. Notice in MATLAB that the size of the array of the integers from 1 to 6 is 6, but 6 - 1 = 5.

There are three ways to use arange():

import numpy as np
np.arange(stop)
np.arange(start, stop)
np.arange(start, stop, step)

If you only pass one argument to arange(), it will be interpreted as the stop value. The start value defaults to 0 and the step defaults to 1. If you pass two arguments to arange(), they are interpreted as the start and stop values. Finally, you can pass all three of start, stop, and step to arange().

Notice that the order of the arguments is different from MATLAB, going start, stop, step in Python. If you’re having trouble remembering the order that these arguments go, remember that you can use keyword arguments in Python to be explicit about what each argument means.

You can try out arange() with the following examples:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.arange(1, 7)
In [3]: arr_1
Out[3]: array([1, 2, 3, 4, 5, 6])
In [4]: arr_1.shape
Out[4]: (6,)

In this example, you are creating an array that contains the values from 1 to 6. As in MATLAB, if the step is omitted, it defaults to 1. Notice that you had to pass the stop value 7 so that the array stopped at 6. However, the size of the resulting array is 7 - 1 = 6 elements long. Next, you should see how to change the step size:

>>>

In [5]: arr_2 = np.arange(1, 7, 2)
In [6]: arr_2
Out[6]: array([1, 3, 5])

In this code, you are creating an array that contains the values from 1 to 6, incrementing by two between each element. The step is two, so NumPy starts with 1, increments to 3, and then to 5. The next step would equal the stop value, but NumPy does not include the stop value in the array. Notice that the formula to compute the size of the array is a little bit different, since the step size is not 1.

With step sizes other than 1, the size of the array can be computed by (stop - start)/step if this results in an integer value. In this case, the size of the array is (7 - 1)/2 = 3 elements, as expected. If (stop - start)/step results in a floating point number, the size of the array is equal to the next largest integer as demonstrated in the next example:

>>>

In [7]: arr_3 = np.arange(2, 7, 2)
In [8]: arr_3
Out[8]: array([2, 4, 6])

In this example, you are creating an array that contains the values from 2 to 6, incrementing by two between each element. The step is two, so NumPy starts with 2, increments to 4, and then to 6. The next step would exceed the stop value, so NumPy stops at 6. Notice that the size of the array is (7 - 2)/2 = 2.5, so the next highest integer is 3 elements, as expected.

Finally, you should usually use integer arguments to arange() in NumPy and the colon operator in MATLAB. If you use floating point values (numbers with decimals), especially for the step, the elements may not come out exactly as you expect. If you want to use floating point numbers, linspace() is a better choice in general.

The Colon Operator Is Very Powerful in NumPy

In MATLAB, the colon operator is used to perform a number of useful tasks. As you saw, it can be used to create arrays, and it can also be used to index or slice arrays. When indexing arrays, MATLAB supports the end keyword to extend the specified range to the end of that dimension, as you saw earlier:

>>>

>> arr_1 = 1:2:6;
>> arr_1(2:end)

ans =

     3     5

In this code, you are indexing arr_1 starting at the second index and going to the end of the array. You can also specify a specific index as the stop value:

>>>

>> arr_2 = 1:6;
>> arr_2(2:4)

ans =

     2     3     4

In this code, you are creating an array arr_2 with the numbers 1 through 6, inclusive. Then, you are specifying the second element as the start value and the fourth element as the stop value in the slice. MATLAB supports the two-colon increment syntax when indexing as well:

>>>

>> arr_2(2:2:end)

ans =

     2     4     6

In this code, you are indexing the array, starting at the second element, skipping every other element, until the end of the array. You can also use end as the starting point of the slice with a negative step:

>>>

>> arr_2(end:-1:4)

ans =

     6     5     4

In this code, you are indexing arr_2 starting from the last value, decrementing by 1, and ending at the 4th element. Finally, you can slice all of the element in a dimension by using just a bare colon:

>>>

>> arr_2(:)

ans =

     1     2     3     4     5     6

In this code, you are selecting all of the first dimension of the array using just the colon.

NumPy and Python in general also use the colon for the slice syntax, but the order of the values is slightly different. In Python, the order is start : stop : step, whereas in MATLAB, it is start : step : stop, as you saw earlier. In addition, in NumPy you can omit start or stop and they will have default a value of 0 (or the first element) for start and the last element for stop. In MATLAB, you must specify start and stop if you want to specify either of them. Thus, Python does not have the end keyword, since you can omit stop to achieve the same behavior.

Try out the following examples of the slice syntax in NumPy:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.arange(1, 7, 2)
In [3]: arr_1[1:]
Out[3]: array([3, 5])

In this code, you are creating an array with the integers from 1 to 6, inclusive, skipping every other number. Then, you are slicing the array taking the second element (index 1) until the end of the array. Notice that the stop value was omitted, so it defaulted to the last element in the array.

You can also specify a specific element as the stop value. You saw in using arange() that the array did not include the stop value. The same is true of the slice syntax in Python, the slice will include everything up to, but not including, the stop index:

>>>

In [4]: arr_2 = np.arange(1, 7)
In [5]: arr_2[1:4]
Out[5]: array([2, 3, 4])

In this code, you are creating an array with the integers from 1 to 6, inclusive. Then, you are slicing the array starting at the second element (index 1, value 2) until the fourth element (index 3, value 4). However, you specified the stop index as 4 (the fifth element in the array, value 5). The reason Python includes up to the (stop - 1) index is the same reason arange() does not include the stop value, so that the length of the resulting array is equal to stop - start. Next, try changing the step of the slice:

>>>

In [6]: arr_2[1::2]
Out[6]: array([2, 4, 6])

In this code, you are slicing the array starting at the second element (index 1), going until the end of the array, and taking every second element. This results in an array with the values 2, 4, and 6. Notice that the stop value was omitted in the slice syntax, so it defaulted to the last element in the array.

You can also use a negative step in the slicing syntax for Python:

>>>

In [7]: arr_2[:2:-1]
Out[7]: array([6, 5, 4])

In this code, you are not specifying the start index of the slice, you are specifying the stop value should be index 2, and the step should be -1. Since the start index is not specified and the step is negative, the start value is assumed to be the last element in the array (or the first element in the reversed array). For the stop value, index 2 has the value of 3 and one index before that (in the reversed array) is index 3 with the value of 4.

Finally, just like in MATLAB, a bare colon means to select all of the elements from that dimension:

>>>

In [8]: arr_2[:]
Out[8]: array([1, 2, 3, 4, 5, 6])

Array Slices Are Views of Arrays in NumPy

In MATLAB, when you access a slice of an array and assign it to a variable, MATLAB will make a copy of that portion of the array into your new variable. This means that when you assign values to the slice, the original array is not affected. Try out this example to help explain the differences of MATLAB vs Python:

>>>

>> arr_1 = [1,2,3;4,5,6;7,8,9];
>> arr_2 = arr_1(2:end,2:end);

arr_2 =

     5     6
     8     9

>> arr_2(1, 1) = 10

arr_2 =

    10     6
     8     9

>> arr_1

arr_1 =

     1     2     3
     4     5     6
     7     8     9

In this code, you have created a 3x3 array arr_1 storing the values from 1 through 9. Then, you create a 2x2 slice of the original array storing from the second value to the end in both dimensions, arr_2. On the third input line, you assign the value 10 to the upper left element in arr_2. Finally, you print arr_1 again to verify that none of the values in arr_1 have changed.

Technical detail: MATLAB employs a copy-on-write memory management system, where an array may only be copied to a new memory location when it is modified. You can read more about MATLAB memory management in Memory Management for Functions and Variables on the Mathworks blog and in Internal Matlab memory optimizations.

In NumPy, slices of arrays are views to the original array. This behavior saves memory and time, since the values in the array don’t have to be copied to a new location. However, it means that changes that you make to a slice from an array will change the original array. You should try the following code to see how this works:

>>>

In [1]: import numpy as np
In [2]: arr_1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8 ,9]])
In [3]: arr_2 = arr_1[1:, 1:]
In [4]: arr_2
Out[4]:
array([[5, 6],
       [8, 9]])

In this code, you are creating a 3x3 array arr_1 storing the values from 1 through 9. Then, you create a 2x2 slice of the original array storing from the second value to the end in both dimensions, arr_2. Notice that the Python indexing is 0-based, so the second element has the index 1. Finally, you are printing arr_2 to verify that it is a 2x2 array.

Now you should see what happens when you change a value in arr_2. Like in the MATLAB example, you should change the upper left element of arr_2:

>>>

In [5]: arr_2[0, 0] = 10
In [6]: arr_2
Out[6]:
array([[10,  6],
       [ 8,  9]])
In [7]: arr_1
Out[7]:
array([[ 1,  2,  3],
       [ 4, 10,  6],
       [ 7,  8,  9]])

In this code, you first assign the upper left element in arr_2, at index (0, 0) to have a value of 10. Then you print arr_2 to verify that the appropriate value has changed. Finally, you print arr_1 and see that the value in the middle of the array has changed from 5 to 10!

This is what is meant by arr_2 being a view of arr_1. Since it is a view, arr_2 points to the same memory location as arr_1, so updating arr_2 also updates arr_1 because the value stored in the memory location accessed by both arr_2 and arr_1 has been updated. This also goes the other direction, where changing values in arr_1 will update the value in arr_2:

>>>

In [8]: arr_1[-1, -1] = 42
In [9]: arr_1
Out[9]:
array([[ 1,  2,  3],
       [ 4, 10,  6],
       [ 7,  8, 42]])
In [10]: arr_2
Out[10]:
array([[10,  6],
       [ 8, 42]])

In this code, you are assigning the bottom right element of arr_1 to have the value 42. Remember that in Python, an index of -1 means the last value on that dimension. Then you are printing arr_1 to verify that the lower right value has changed from 9 to 42. Finally, you are printing arr_2, and you see for arr_2 as well, the bottom right value has changed from 9 to 42.

If you want to generate a copy of an array, you can use np.copy(). Copying an array creates a new place in memory for the copy to be stored, so changes to the copied array do not affect the original:

>>>

In [11]: arr_3 = np.copy(arr_2)
In [12]: arr_3[1, 0] = 37
In [13]: arr_3
Out[13]:
array([[10,  6],
       [37, 42]])
In [14]: arr_2
Out[14]:
array([[10,  6],
       [ 8, 42]])

In this code, you are creating arr_3 as a copy of arr_2. Then, you are changing the element in the second row, first column to have the value of 37. Then, you are printing arr_3 to verify that the specified change has been made. Finally, you are printing arr_2 to verify that no changes have occurred in arr_2, as expected.

Tips and Tricks to Make Your Code Pythonic

Like any other programming language, Python code written by experienced Python developers often has a particular look and feel to it. This is because they are able to take advantage of specific idioms in Python to work with Python rather than against Python. Developers coming from other languages often miss out on what makes code Pythonic in their first projects.

In this section, you’ll learn some tips and tricks to make your code Pythonic and level up your Python skills. There are many more tips and tricks than you can learn here, so feel free to check out Write More Pythonic Code.

You Should Not Use Semicolons to End Lines in Python

In MATLAB, ending a line of code with a semicolon ; suppresses the output from that line. For instance, assigning a variable will print the value of the variable after the assignment if the semicolon is omitted.

In Python, you should not end lines of code with semicolons. It is unnecessary, since Python does not change its behavior whether the line is ended with a semicolon or not. So you can save yourself a key stroke and not bother including the semicolon in your scripts and libraries.

There is one case in Python where the semicolon is useful. When you want to execute several statements, but you cannot include a newline character in the input, you can separate the statements with semicolons. This is mostly useful to execute very short scripts from the command prompt or terminal. For instance, to find the particular Python executable that is running, you can type the following:

$ python -c "import sys; print(sys.executable)"
/home/eleanor/anaconda3/bin/python

In this code, you are executing the Python interpreter in the python executable and passing the -c switch. This switch takes the next argument and executes it within the interpreter. Since the shell environment would execute if you pressed Enter to insert a new line, you can type the whole script on one line.

In this case, you have two logical statements that need to be separated by the semicolon. First you are importing the built-in sys library and then you are printing the value of sys.executable. In this example, the Python interpreter that the shell is running comes from the /home/eleanor/anaconda3/bin/python file.

You Should Not Import `*` From a Module in Python

In a previous section, you read about how namespaces are one honking great idea in Python. In MATLAB, all functions are part of the global namespace by default, so every function and class name has to be unique. Python solves this problem by using namespaces and requiring you to specify which module a function should come from.

You will find tutorials around the Web that suggest you write the following:

from a_module import *

In this code, you are using the * to indicate that Python should import everything that is contained in a_module and put it in the current scope without a prefix. This is slightly more convenient, because you no longer have to prefix functions and classes from a_module with anything, you can just use them directly. However, it is not a good practice because you don’t know what names are defined in a_module and whether or not they will override any existing names in your current scope.

Technical Note: When you from a_module import *, Python imports all of the names listed in a special variable called __all__ in a_module. However, if that variable is not defined, Python will import all of the variables, functions, and classes defined in a_module.

You Should Take Advantage of the Different Data Types in Python

MATLAB, with its heritage as a linear algebra and array focused language, treats most data types as arrays of some sort. This often makes it a little bit tricky to work with more advanced data types like structs, containers.Map, cell arrays, and more.

Python has several built-in data types that are very flexible and can be used to accomplish a number of useful tasks. The major ones that you’ll learn about in this section are lists and dictionaries.

Lists

Python lists are mutable sequences of values. Lists can contain heterogeneous data, which means that each element of the list can be of a different type. Because lists are mutable, you can change the value of any element in the list, or add or remove values from the list, without creating a new list object.

Since lists are sequences, you can create loops that iterate over them. In Python, you do not need to access each element of a list with an index in a for loop, as you would do in MATLAB:

>>>

>> arr_1 = 1:2:6;
>> for i = 1:length(arr_1)
    disp(arr_1(i))
end
1
3
5

In this code, you are creating an array arr_1 with the integers from 1 to 6, taking every other number. Then you are creating a for loop where the loop variable goes from 1 to the length of arr_1. Finally, you are displaying the value of the element of arr_1 at the loop variable on each step by using the loop variable i to index arr_1.

In Python, you should not use an index for the list when you loop over it. Instead, you should loop directly over the items in a list:

>>>

In [1]: lst_1 = [1, "b", 3.0]
In [2]: for item in lst_1:
   ...:     print(item)
   ...:
1
b
3.0

In this code, on input line 1 you are first creating a Python list with three elements:

The integer 1
The string "b"
The float 3.0

This list is assigned to lst_1. Then you are using a for loop to access each item in the list in turn. On each iteration, the next value in the list is put into the variable item that you specified on the for line. Then, you are printing the value of item on each iteration.

Notice in the previous example that you could loop over the value of each element in the list without using an index. Nonetheless, sometimes you want to access the index of each item in the list as you’re looping over it. For those cases, Python provides enumerate() that returns the index and the value of the item:

>>>

In [3]: for index, item in enumerate(lst_1):
   ...:     print(f"The index is {index} and the item is {item}")
   ...:
The index is 0 and the item is 1
The index is 1 and the item is b
The index is 2 and the item is 3.0

In this code, you are looping over lst_1 again, but this time, you are using enumerate() to get both the index and the item. Then you are printing the value of the index and item on each loop iteration. As you can see from the result, the index values start at 0 as expected, but you do not need to use the index to access the item from the list.

In summary, you should not write Python code like this:

for i in range(len(lst_1)):
    print(lst_1[i])

In this code, you are creating a range of integers from 0 to the length of lst_1 and then you are accessing each element in the list by its index. This can lead to off-by-one and fencepost errors. Instead, you should write code that loops over the list directly:

for item in lst_1:
    print(item)

You can read a lot more about lists in Lists and Tuples in Python and about for loops and iteration in Python “for” Loops (Definite Iteration). There is also a more advanced concept called list comprehensions that you can learn about in Using List Comprehensions Effectively.

Dictionaries

In MATLAB, you can create a map data type with containers.Map(). This kind of data structure is useful when you have two pieces of data that are always related to each other and you want to connect them together. For instance, you can map cities to their population with a containers.Map():

>>>

>> cities = containers.Map({'Cleveland';'Pittsburgh';'Hartford'},
                           [383793,301048,122587]);
>> cities('Cleveland')

ans =

     383793

In this code, you are creating a containers.Map() on the first line. The first argument is a cell array of character arrays with the city names. These are called the keys of the map. The second argument is an array of populations. These are called the values of the map. Then, you are accessing the value of the population in Cleveland by indexing the map with a character array.

You can assign new values into the map by assigning to an undefined key value:

>>>

>> cities('Providence') = 180393;

If you try to access a key that does not exist, you will receive an error message:

>>>

>> cities('New York City')
Error using containers.Map/subsref
The specified key is not present in this container.

Python has an equivalent data structure called a dictionary. To create a Python dictionary, you can use curly braces and specify the keys and values with each other:

>>>

In [1]: cities = {
   ...: "Cleveland": 383_793,
   ...: "Pittsburgh": 301_048,
   ...: "Hartford": 122_587}
In [2]: cities["Cleveland"]
Out[2]: 383793

In this code, on input line 1 you are creating the dictionary of cities using curly braces. Notice that the key and value are specified together, separated by a colon. The values are specified with _ in the numbers, a feature available since Python 3.6. This does not change the value of the number, it only makes it easier to read very large numbers. Then, you are accessing the value at the "Cleveland" key using square brackets, similar to the indexing syntax for lists and arrays.

You can add new keys to the dictionary by assigning to them:

>>>

In [3]: cities["Providence"] = 180_393

In this code, you assigned a new key "Providence" to the dictionary with a value of 180,393. If you try to access a key that is not in the dictionary, you will get a KeyError:

>>>

In [4]: cities["New York City"]
Traceback (most recent call last):

  File "<ipython-input-4-6ebe5b35f3ea>", line 1, in <module>
    cities["New York City"]

KeyError: 'New York City'

In this code, you are trying to access the dictionary using the "New York City" key. However, this key does not exist in the dictionary so Python raises a KeyError letting you know that "New York City" is not an option.

You can read a lot more about Python dictionaries in Dictionaries in Python and KeyError exceptions in Python KeyError Exceptions and How to Handle Them. You can also iterate through dictionaries and use dictionary comprehensions, similar to list comprehensions. You can read about these topics in How to Iterate Through a Dictionary in Python.

Exceptions Help You Control Program Flow in Python

MATLAB and Python both use errors and exceptions to let you know when something has gone wrong in the code. In this section, you will learn about common exceptions in Python and how you can handle them appropriately.

If you want an introduction to Python exceptions overall, you can read Python Exceptions: An Introduction. When a Python exception is raised, it produces a traceback. You can read about how to interpret the traceback in Understanding Python Tracebacks. Understanding tracebacks is very helpful to interpret and correct Python exceptions in general. There are a few specific cases that usually have the same resolution. You’ll see those described in the rest of this section.

NameError

Python NameError exceptions are usually the result of a variable being undefined. When you see a NameError, check your code for typos and misspelled variable names. You can use the debugging features and the variable explorer in Spyder to find out which variables are defined.

SyntaxError

Python SyntaxError exceptions mean that you have input some improper syntax. This is usually caused by mismatched brackets, when you have only the opening bracket or closing bracket but not the matching one. These exceptions usually point to the line after the place where the problem is located.

Another common SyntaxError is using only one equals sign in an if statement. In this case, you either meant not-equals (!=) or equals (==), so you can correct the line.

KeyError

Python KeyError exceptions occur when you try to access a key in a dictionary that does not exist. You can use .get() to retrieve a key from a dictionary if it exists, or return a default value if the key does not exist. You can read more about KeyError exceptions in Python KeyError Exceptions and How to Handle Them.

IndexError

Python IndexError exceptions occur when you are trying to access the index of an array or list that does not exist. This usually means the array or list you are trying to access has fewer elements than the index you are trying to access. You can use the debugging features and the variable explorer in Spyder to see the size of lists and arrays and make sure you’re only accessing indices that exist.

ImportError/ModuleNotFoundError

Python ImportError and ModuleNotFoundError exceptions occur when you try to import a module that Python cannot find. This might be because it is installed in a different conda environment or virtualenv, or it might be because you forgot to install the package.

The solution to this error is usually to conda install or pip install the package and make sure the correct environment is activated. If you’re not using the base environment in conda, you also need to make sure to install Spyder or Jupyter into your environment.

TypeError/ValueError

Python TypeError exceptions happen when an argument is of the wrong type. This happens most commonly when you pass an argument of the wrong type into a function. For instance, a function that works with numbers would raise a TypeError if a string were passed in.

A related exception is the ValueError. This exception happens when an argument is of the correct type, but has an incorrect value. For instance, a function that works only with positive numbers would raise a ValueError if a negative number were passed in.

AttributeError

Python AttributeError exceptions happen when you try to access an attribute of an object when the object does not have that attribute. You will often see this error associated with the message NoneType object has no attribute. This message most likely means that a function has returned None instead of the object you were expecting and you are trying to access an attribute that would be present on the real object, but is not defined for None.

How to Handle Exceptions in Python

MATLAB allows you to try a statement of code and catch any errors that are thrown by the code. Once you have caught an error, you can do further processing of the error and assign variables based on the type of error. The MATLAB documentation has several good examples of how this would look in MATLAB.

In Python, one big difference from MATLAB is that you can choose to catch only certain types of exceptions and handle them. This allows all other exceptions to continue to be displayed to the user. If you want to learn more about how to do this in Python, you can read The try and except Block: Handling Exceptions.

To see how this works, you can try the following example:

>>>

In [1]: import math
In [2]: def my_sqrt(number):
   ...:    print(f"You passed the argument: {number!r}")
   ...:    try:
   ...:        return math.sqrt(number)
   ...:    except ValueError:
   ...:        print("You passed a number that cannot be operated on")
   ...:    except TypeError:
   ...:        print("You passed an argument that was not a number")
   ...:

In [3]: my_sqrt(4.0)
You passed the argument: 4.0
Out[3]: 2.0

In [4]: my_sqrt(-1.0)
You passed the argument: -1.0
You passed a number that cannot be operated on

In [4]: my_sqrt("4.0")
You passed the argument: '4.0'
You passed an argument that was not a number

In this code, on input line 1 you are importing the built-in math library. Then, starting on input line 2, you are defining a function called my_sqrt() that will take one argument, called number. Inside the function definition, you first print the argument that the user passed.

Next, you enter the try/except block. First, you try to take the square root of the input argument and return the result. If taking the square root of the argument results in an error, Python will catch that error and check which type of error was raised.

You have defined code that handles two specific exceptions: ValueError and TypeError. If math.sqrt() raises a ValueError, your code will print a message that the number cannot be operated on. If math.sqrt() raises a TypeError, your code will print a message that the argument was not a number. If any other type of exception is raised by math.sqrt(), that error will be passed through without any processing, since there is no handler for any other error types.

More specifically, Python checks for any error that is raised by the code in the try block. In your case, you only defined one line of code in the try block, but this is not required, and you can have as many lines as you want there. However, it is usually a good practice to minimize the number of lines of code in the try block so you can be very specific about which code is raising any errors.

On input line 3, you are testing out my_sqrt(). First, you pass the value 4.0 to the function. The function prints the argument, and math.sqrt() has no problems taking the square root of 4, resulting in 2.0 on the output line.

On input line 4, you are passing -1.0 as the argument to my_sqrt(). As you may recall, taking the square root of negative numbers results in a complex number, which the math.sqrt() function is not equipped to handle. Taking the square root of a negative number using math.sqrt() raises a ValueError. You exception handler catches this ValueError and prints the message that the number cannot be operated on.

On input line 5, you are passing "4.0" as the argument to my_sqrt(). In this case, math.sqrt() does not know how to take the square root of a string, even though that string appears to represent a number. You can see that you have passed a string by the quotes in the statement giving the value of the argument: You passed the argument: '4.0'. Since math.sqrt() cannot take the square root of a string, it raises a TypeError, and your function prints the message that the argument was not a number.

There Is an Official Guide to Writing Good Code in Python

The Python community has developed a set of recommendations for how to style your Python code. These are codified in a document called PEP 8, which stands for Python Enhancement Proposal #8. PEP 8 can be found in full on the Python website. You can also learn more about good Python style in How to Write Beautiful Python Code With PEP 8 and Idiomatic Python 101.

Perhaps the most important principle in PEP 8 is the saying that “a foolish consistency is the hobgoblin of little minds.” This means that you should follow the recommendations in PEP 8 for almost all of your code, but there may be some limited cases where it is a good idea not to follow the PEP 8 recommendations. For instance, if you are working with an existing codebase that has its own style, you should follow that style where it diverges from PEP 8. You can see an excellent discussion of this principle from Raymond Hettinger, one of the core Python developers, in a talk from PyCon 2015.

Aside from reading PEP 8, you can use a few Python packages to automatically make sure that your code fits the style guidelines. Flake8 is a code linter that reads your code and makes suggestions for how you can improve it. This is similar to the feature in the MATLAB code editor that makes improvement suggestions. In addition, packages such as Black, yapf, and autopep8 will automatically format your code to be compliant with PEP 8 or your own style rules. Using these packages can help your code feel more Pythonic and help you learn good Python style!

Python Has a Fantastic and Supportive Community

Python is known for having a very supportive, open, and welcoming community. Whether you are a brand new developer or an experienced one, whether you’re brand new to Python or have been to a dozen conferences, the community is there to support you and what you want to do.

The community starts with the Python Package Index (called PyPI or the CheeseShop, a reference to the Monty Python sketch), which houses hundreds of thousands of different Python packages that you can download for free. These packages can be installed using pip, a package manager that comes bundled with Python. This means that adding the functionality you need to Python can be as simple as pip install package or if you’re using Anaconda, conda install package.

Since Python is used in so many different areas of software development, data science, science, and engineering, there are always people around who want to talk about Python. Most large cities around the world have Python meetup groups. You can go to these groups to learn about Python by hearing people speak about their work or work on some open-source code.

A few times a year, these groups coalesce into different PyCons which happen on every continent around the globe. PyCon North America is the largest of these, with several thousand attendees every year. You can read all about what it’s like to attend in How to Get the Most Out of PyCon.

Python also has a very strong online community. If you have a question about programming in Python, you can ask on StackOverflow and some of the world-leading Python experts will be able to help you out. Make sure to follow the instructions for how to ask a question on StackOverflow. Remember that the more effort you put into your question, the more likely you are to either find the answer yourself (hooray!) or get a good answer from someone else.

If you want to follow along with Python’s development, you can sign up for one of the mailing lists covering different aspects of Python’s community. The general mailing list for asking questions about writing programs in Python is called comp.lang.python. If you are interested in the development of Python itself, you can follow the python-dev mailing list.

If you’re interested in learning much, much more about Python development, you can check out the Real Python Learning Paths!

Areas Where You Should Still Use MATLAB®

In spite of the awesome community and terrific packages, there are still one or two areas where MATLAB works better than Python. The main place where Python can’t compete with MATLAB is the Simulink Toolbox. This toolbox offers advanced capabilities for signal processing and modeling in a convenient graphical interface.

Python does not have an equivalent graphical interface to these kinds of functions. However, to the extent that Simulink is a convenient interface to ordinary differential equation solvers, Python has equivalent solvers as in MATLAB and the underlying functionality of Simulink can certainly be replicated in Python.

Otherwise, you can do anything in Python that you can do in MATLAB! If you can think of work that you can do with MATLAB, but you’re not sure how to do it in Python, let us know in the comments and we’ll be able to help with suggestions.

Conclusion

Congratulations, you now have the knowledge you need to switch your MATLAB code to Python! In this article, you learned a little bit about what Python is, how to set up your computer to use Python, and how to convert your code from MATLAB to Python.

Python is a really huge language and community, with lots to learn and lots of people to learn from. Remember, you weren’t a MATLAB expert the first time you opened the MATLAB development environment, and the same is true about the first time you write some Python code. Come back to this article as often as you need to improve your skills and learn more about becoming a Python wizard!

Python args and kwargs: Demystified

2019-09-04T14:00:00+00:00

Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: *args and **kwargs. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in main(), then this article is for you. You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.

By the end of the article, you’ll know:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

This article assumes that you already know how to define Python functions and work with lists and dictionaries.

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Passing Multiple Arguments to a Function

*args and **kwargs allow you to pass multiple arguments or keyword arguments to a function. Consider the following example. This is a simple function that takes two arguments and returns their sum:

def my_sum(a, b):
    return a + b

This function works fine, but it’s limited to only two arguments. What if you need to sum a varying number of arguments, where the specific number of arguments passed is only determined at runtime? Wouldn’t it be great to create a function that could sum all the integers passed to it, no matter how many there are?

Using the Python args Variable in Function Definitions

There are a few ways you can pass a varying number of arguments to a function. The first way is often the most intuitive for people that have experience with collections. You simply pass a list or a set of all the arguments to your function. So for my_sum(), you could pass a list of all the integers you need to add:

# sum_integers_list.py
def my_sum(my_integers):
    result = 0
    for x in my_integers:
        result += x
    return result

list_of_integers = [1, 2, 3]
print(my_sum(list_of_integers))

This implementation works, but whenever you call this function you’ll also need to create a list of arguments to pass to it. This can be inconvenient, especially if you don’t know up front all the values that should go into the list.

This is where *args can be really useful, because it allows you to pass a varying number of positional arguments. Take the following example:

# sum_integers_args.py
def my_sum(*args):
    result = 0
    # Iterating over the Python args tuple
    for x in args:
        result += x
    return result

print(my_sum(1, 2, 3))

In this example, you’re no longer passing a list to my_sum(). Instead, you’re passing three different positional arguments. my_sum() takes all the parameters that are provided in the input and packs them all into a single iterable object named args.

Note that args is just a name. You’re not required to use the name args. You can choose any name that you prefer, such as integers:

# sum_integers_args_2.py
def my_sum(*integers):
    result = 0
    for x in integers:
        result += x
    return result

print(my_sum(1, 2, 3))

The function still works, even if you pass the iterable object as integers instead of args. All that matters here is that you use the unpacking operator (*).

Bear in mind that the iterable object you’ll get using the unpacking operator * is not a list but a tuple. A tuple is similar to a list in that they both support slicing and iteration. However, tuples are very different in at least one aspect: lists are mutable, while tuples are not. To test this, run the following code. This script tries to change a value of a list:

# change_list.py
my_list = [1, 2, 3]
my_list[0] = 9
print(my_list)

The value located at the very first index of the list should be updated to 9. If you execute this script, you will see that the list indeed gets modified:

$ python change_list.py
[9, 2, 3]

The first value is no longer 0, but the updated value 9. Now, try to do the same with a tuple:

# change_tuple.py
my_tuple = (1, 2, 3)
my_tuple[0] = 9
print(my_tuple)

Here, you see the same values, except they’re held together as a tuple. If you try to execute this script, you will see that the Python interpreter returns an error:

$ python change_tuple.py
Traceback (most recent call last):
  File "change_tuple.py", line 3, in <module>
    my_tuple[0] = 9
TypeError: 'tuple' object does not support item assignment

This is because a tuple is an immutable object, and its values cannot be changed after assignment. Keep this in mind when you’re working with tuples and *args.

Using the Python kwargs Variable in Function Definitions

Okay, now you’ve understood what *args is for, but what about **kwargs? **kwargs works just like *args, but instead of accepting positional arguments it accepts keyword (or named) arguments. Take the following example:

# concatenate.py
def concatenate(**kwargs):
    result = ""
    # Iterating over the Python kwargs dictionary
    for arg in kwargs.values():
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

When you execute the script above, concatenate() will iterate through the Python kwargs dictionary and concatenate all the values it finds:

$ python concatenate.py
RealPythonIsGreat!

Like args, kwargs is just a name that can be changed to whatever you want. Again, what is important here is the use of the unpacking operator (**).

So, the previous example could be written like this:

# concatenate_2.py
def concatenate(**words):
    result = ""
    for arg in words.values():
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

Note that in the example above the iterable object is a standard dict. If you iterate over the dictionary and want to return its values, like in the example shown, then you must use .values().

In fact, if you forget to use this method, you will find yourself iterating through the keys of your Python kwargs dictionary instead, like in the following example:

# concatenate_keys.py
def concatenate(**kwargs):
    result = ""
    # Iterating over the keys of the Python kwargs dictionary
    for arg in kwargs:
        result += arg
    return result

print(concatenate(a="Real", b="Python", c="Is", d="Great", e="!"))

Now, if you try to execute this example, you’ll notice the following output:

$ python concatenate_keys.py
abcde

As you can see, if you don’t specify .values(), your function will iterate over the keys of your Python kwargs dictionary, returning the wrong result.

Ordering Arguments in a Function

Now that you have learned what *args and **kwargs are for, you are ready to start writing functions that take a varying number of input arguments. But what if you want to create a function that takes a changeable number of both positional and named arguments?

In this case, you have to bear in mind that order counts. Just as non-default arguments have to precede default arguments, so *args must come before **kwargs.

To recap, the correct order for your parameters is:

Standard arguments
*args arguments
**kwargs arguments

For example, this function definition is correct:

# correct_function_definition.py
def my_function(a, b, *args, **kwargs):
    pass

The *args variable is appropriately listed before **kwargs. But what if you try to modify the order of the arguments? For example, consider the following function:

# wrong_function_definition.py
def my_function(a, b, **kwargs, *args):
    pass

Now, **kwargs comes before *args in the function definition. If you try to run this example, you’ll receive an error from the interpreter:

$ python wrong_function_definition.py
  File "wrong_function_definition.py", line 2
    def my_function(a, b, **kwargs, *args):
                                    ^
SyntaxError: invalid syntax

In this case, since *args comes after **kwargs, the Python interpreter throws a SyntaxError.

Unpacking With the Asterisk Operators: `*` & `**`

You are now able to use *args and **kwargs to define Python functions that take a varying number of input arguments. Let’s go a little deeper to understand something more about the unpacking operators.

The single and double asterisk unpacking operators were introduced in Python 2. As of the 3.5 release, they have become even more powerful, thanks to PEP 448. In short, the unpacking operators are operators that unpack the values from iterable objects in Python. The single asterisk operator * can be used on any iterable that Python provides, while the double asterisk operator ** can only be used on dictionaries.

Let’s start with an example:

# print_list.py
my_list = [1, 2, 3]
print(my_list)

This code defines a list and then prints it to the standard output:

$ python print_list.py
[1, 2, 3]

Note how the list is printed, along with the corresponding brackets and commas.

Now, try to prepend the unpacking operator * to the name of your list:

# print_unpacked_list.py
my_list = [1, 2, 3]
print(*my_list)

Here, the * operator tells print() to unpack the list first.

In this case, the output is no longer the list itself, but rather the content of the list:

$ python print_unpacked_list.py
1 2 3

Can you see the difference between this execution and the one from print_list.py? Instead of a list, print() has taken three separate arguments as the input.

Another thing you’ll notice is that in print_unpacked_list.py, you used the unpacking operator * to call a function, instead of in a function definition. In this case, print() takes all the items of a list as though they were single arguments.

You can also use this method to call your own functions, but if your function requires a specific number of arguments, then the iterable you unpack must have the same number of arguments.

To test this behavior, consider this script:

# unpacking_call.py
def my_sum(a, b, c):
    print(a + b + c)

my_list = [1, 2, 3]
my_sum(*my_list)

Here, my_sum() explicitly states that a, b, and c are required arguments.

If you run this script, you’ll get the sum of the three numbers in my_list:

$ python unpacking_call.py
6

The 3 elements in my_list match up perfectly with the required arguments in my_sum().

Now look at the following script, where my_list has 4 arguments instead of 3:

# wrong_unpacking_call.py
def my_sum(a, b, c):
    print(a + b + c)

my_list = [1, 2, 3, 4]
my_sum(*my_list)

In this example, my_sum() still expects just three arguments, but the * operator gets 4 items from the list. If you try to execute this script, you’ll see that the Python interpreter is unable to run it:

$ python wrong_unpacking_call.py
Traceback (most recent call last):
  File "wrong_unpacking_call.py", line 6, in <module>
    my_sum(*my_list)
TypeError: my_sum() takes 3 positional arguments but 4 were given

When you use the * operator to unpack a list and pass arguments to a function, it’s exactly as though you’re passing every single argument alone. This means that you can use multiple unpacking operators to get values from several lists and pass them all to a single function.

To test this behavior, consider the following example:

# sum_integers_args_3.py
def my_sum(*args):
    result = 0
    for x in args:
        result += x
    return result

list1 = [1, 2, 3]
list2 = [4, 5]
list3 = [6, 7, 8, 9]

print(my_sum(*list1, *list2, *list3))

If you run this example, all three lists are unpacked. Each individual item is passed to my_sum(), resulting in the following output:

$ python sum_integers_args_3.py
45

There are other convenient uses of the unpacking operator. For example, say you need to split a list into three different parts. The output should show the first value, the last value, and all the values in between. With the unpacking operator, you can do this in just one line of code:

# extract_list_body.py
my_list = [1, 2, 3, 4, 5, 6]

a, *b, c = my_list

print(a)
print(b)
print(c)

In this example, my_list contains 6 items. The first variable is assigned to a, the last to c, and all other values are packed into a new list b. If you run the script, print() will show you that your three variables have the values you would expect:

$ python extract_list_body.py
1
[2, 3, 4, 5]
6

Another interesting thing you can do with the unpacking operator * is to split the items of any iterable object. This could be very useful if you need to merge two lists, for instance:

# merging_lists.py
my_first_list = [1, 2, 3]
my_second_list = [4, 5, 6]
my_merged_list = [*my_first_list, *my_second_list]

print(my_merged_list)

The unpacking operator * is prepended to both my_first_list and my_second_list.

If you run this script, you’ll see that the result is a merged list:

$ python merging_lists.py
[1, 2, 3, 4, 5, 6]

You can even merge two different dictionaries by using the unpacking operator **:

# merging_dicts.py
my_first_dict = {"A": 1, "B": 2}
my_second_dict = {"C": 3, "D": 4}
my_merged_dict = {**my_first_dict, **my_second_dict}

print(my_merged_dict)

Here, the iterables to merge are my_first_dict and my_second_dict.

Executing this code outputs a merged dictionary:

$ python merging_dicts.py
{'A': 1, 'B': 2, 'C': 3, 'D': 4}

Remember that the * operator works on any iterable object. It can also be used to unpack a string:

# string_to_list.py
a = [*"RealPython"]
print(a)

In Python, strings are iterable objects, so * will unpack it and place all individual values in a list a:

$ python string_to_list.py
['R', 'e', 'a', 'l', 'P', 'y', 't', 'h', 'o', 'n']

The previous example seems great, but when you work with these operators it’s important to keep in mind the seventh rule of The Zen of Python by Tim Peters: Readability counts.

To see why, consider the following example:

# mysterious_statement.py
*a, = "RealPython"
print(a)

There’s the unpacking operator *, followed by a variable, a comma, and an assignment. That’s a lot packed into one line! In fact, this code is no different from the previous example. It just takes the string RealPython and assigns all the items to the new list a, thanks to the unpacking operator *.

The comma after the a does the trick. When you use the unpacking operator with variable assignment, Python requires that your resulting variable is either a list or a tuple. With the trailing comma, you have actually defined a tuple with just one named variable a.

While this is a neat trick, many Pythonistas would not consider this code to be very readable. As such, it’s best to use these kinds of constructions sparingly.

Conclusion

You are now able to use *args and **kwargs to accept a changeable number of arguments in your functions. You have also learned something more about the unpacking operators.

You’ve learned:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

If you still have questions, don’t hesitate to reach out in the comments section below! To learn more about the use of the asterisks in Python, have a look at Trey Hunner’s article on the subject.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Lists and Tuples in Python

2019-09-03T14:00:00+00:00

In this course, you’ll learn about working with lists and tuples. Lists and tuples are arguably Python’s most versatile, useful data types. You’ll find them in virtually every non-trivial Python program.

Here’s what you’ll learn in this tutorial: You’ll cover the important characteristics of lists and tuples. You’ll learn how to define them and how to manipulate them. When you’re finished, you’ll have a good feel for when and how to use these object types in a Python program.

Take the Quiz: Test your knowledge with our interactive “Python Lists and Tuples” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Natural Language Processing With spaCy in Python

2019-09-02T14:00:00+00:00

spaCy is a free and open-source library for Natural Language Processing (NLP) in Python with a lot of in-built capabilities. It’s becoming increasingly popular for processing and analyzing data in NLP. Unstructured textual data is produced at a large scale, and it’s important to process and derive insights from unstructured data. To do that, you need to represent the data in a format that can be understood by computers. NLP can help you do that.

In this tutorial, you’ll learn:

What the foundational terms and concepts in NLP are
How to implement those concepts in spaCy
How to customize and extend built-in functionalities in spaCy
How to perform basic statistical analysis on a text
How to create a pipeline to process unstructured text
How to parse a sentence and extract meaningful insights from it

What Are NLP and spaCy?

NLP is a subfield of Artificial Intelligence and is concerned with interactions between computers and human languages. NLP is the process of analyzing, understanding, and deriving meaning from human languages for computers.

NLP helps you extract insights from unstructured text and has several use cases, such as:

spaCy is a free, open-source library for NLP in Python. It’s written in Cython and is designed to build information extraction or natural language understanding systems. It’s built for production use and provides a concise and user-friendly API.

Installation

In this section, you’ll install spaCy and then download data and models for the English language.

How to Install spaCy

spaCy can be installed using pip, a Python package manager. You can use a virtual environment to avoid depending on system-wide packages. To learn more about virtual environments and pip, check out What Is Pip? A Guide for New Pythonistas and Python Virtual Environments: A Primer.

Create a new virtual environment:

$ python3 -m venv env

Activate this virtual environment and install spaCy:

$ source ./env/bin/activate
$ pip install spacy

How to Download Models and Data

spaCy has different types of models. The default model for the English language is en_core_web_sm.

Activate the virtual environment created in the previous step and download models and data for the English language:

$ python -m spacy download en_core_web_sm

Verify if the download was successful or not by loading it:

>>>

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')

If the nlp object is created, then it means that spaCy was installed and that models and data were successfully downloaded.

Using spaCy

In this section, you’ll use spaCy for a given input string and a text file. Load the language model instance in spaCy:

>>>

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')

Here, the nlp object is a language model instance. You can assume that, throughout this tutorial, nlp refers to the language model loaded by en_core_web_sm. Now you can use spaCy to read a string or a text file.

How to Read a String

You can use spaCy to create a processed Doc object, which is a container for accessing linguistic annotations, for a given input string:

>>>

>>> introduction_text = ('This tutorial is about Natural'
...     ' Language Processing in Spacy.')
>>> introduction_doc = nlp(introduction_text)
>>> # Extract tokens for the given doc
>>> print ([token.text for token in introduction_doc])
['This', 'tutorial', 'is', 'about', 'Natural', 'Language',
'Processing', 'in', 'Spacy', '.']

In the above example, notice how the text is converted to an object that is understood by spaCy. You can use this method to convert any text into a processed Doc object and deduce attributes, which will be covered in the coming sections.

How to Read a Text File

In this section, you’ll create a processed Doc object for a text file:

>>>

>>> file_name = 'introduction.txt'
>>> introduction_file_text = open(file_name).read()
>>> introduction_file_doc = nlp(introduction_file_text)
>>> # Extract tokens for the given doc
>>> print ([token.text for token in introduction_file_doc])
['This', 'tutorial', 'is', 'about', 'Natural', 'Language',
'Processing', 'in', 'Spacy', '.', '\n']

This is how you can convert a text file into a processed Doc object.

Note:

You can assume that:

Variable names ending with the suffix _text are Unicode string objects.
Variable name ending with the suffix _doc are spaCy’s language model objects.

Sentence Detection

Sentence Detection is the process of locating the start and end of sentences in a given text. This allows you to you divide a text into linguistically meaningful units. You’ll use these units when you’re processing your text to perform tasks such as part of speech tagging and entity extraction.

In spaCy, the sents property is used to extract sentences. Here’s how you would extract the total number of sentences and the sentences for a given input text:

>>>

>>> about_text = ('Gus Proto is a Python developer currently'
...               ' working for a London-based Fintech'
...               ' company. He is interested in learning'
...               ' Natural Language Processing.')
>>> about_doc = nlp(about_text)
>>> sentences = list(about_doc.sents)
>>> len(sentences)
2
>>> for sentence in sentences:
...     print (sentence)
...
'Gus Proto is a Python developer currently working for a
London-based Fintech company.'
'He is interested in learning Natural Language Processing.'

In the above example, spaCy is correctly able to identify sentences in the English language, using a full stop(.) as the sentence delimiter. You can also customize the sentence detection to detect sentences on custom delimiters.

Here’s an example, where an ellipsis(...) is used as the delimiter:

>>>

>>> def set_custom_boundaries(doc):
...     # Adds support to use `...` as the delimiter for sentence detection
...     for token in doc[:-1]:
...         if token.text == '...':
...             doc[token.i+1].is_sent_start = True
...     return doc
...
>>> ellipsis_text = ('Gus, can you, ... never mind, I forgot'
...                  ' what I was saying. So, do you think'
...                  ' we should ...')
>>> # Load a new model instance
>>> custom_nlp = spacy.load('en_core_web_sm')
>>> custom_nlp.add_pipe(set_custom_boundaries, before='parser')
>>> custom_ellipsis_doc = custom_nlp(ellipsis_text)
>>> custom_ellipsis_sentences = list(custom_ellipsis_doc.sents)
>>> for sentence in custom_ellipsis_sentences:
...     print(sentence)
...
Gus, can you, ...
never mind, I forgot what I was saying.
So, do you think we should ...
>>> # Sentence Detection with no customization
>>> ellipsis_doc = nlp(ellipsis_text)
>>> ellipsis_sentences = list(ellipsis_doc.sents)
>>> for sentence in ellipsis_sentences:
...     print(sentence)
...
Gus, can you, ... never mind, I forgot what I was saying.
So, do you think we should ...

Note that custom_ellipsis_sentences contain three sentences, whereas ellipsis_sentences contains two sentences. These sentences are still obtained via the sents attribute, as you saw before.

Tokenization in spaCy

Tokenization is the next step after sentence detection. It allows you to identify the basic units in your text. These basic units are called tokens. Tokenization is useful because it breaks a text into meaningful units. These units are used for further analysis, like part of speech tagging.

In spaCy, you can print tokens by iterating on the Doc object:

>>>

>>> for token in about_doc:
...     print (token, token.idx)
...
Gus 0
Proto 4
is 10
a 13
Python 15
developer 22
currently 32
working 42
for 50
a 54
London 56
- 62
based 63
Fintech 69
company 77
. 84
He 86
is 89
interested 92
in 103
learning 106
Natural 115
Language 123
Processing 132
. 142

Note how spaCy preserves the starting index of the tokens. It’s useful for in-place word replacement. spaCy provides various attributes for the Token class:

>>>

>>> for token in about_doc:
...     print (token, token.idx, token.text_with_ws,
...            token.is_alpha, token.is_punct, token.is_space,
...            token.shape_, token.is_stop)
...
Gus 0 Gus  True False False Xxx False
Proto 4 Proto  True False False Xxxxx False
is 10 is  True False False xx True
a 13 a  True False False x True
Python 15 Python  True False False Xxxxx False
developer 22 developer  True False False xxxx False
currently 32 currently  True False False xxxx False
working 42 working  True False False xxxx False
for 50 for  True False False xxx True
a 54 a  True False False x True
London 56 London True False False Xxxxx False
- 62 - False True False - False
based 63 based  True False False xxxx False
Fintech 69 Fintech  True False False Xxxxx False
company 77 company True False False xxxx False
. 84 .  False True False . False
He 86 He  True False False Xx True
is 89 is  True False False xx True
interested 92 interested  True False False xxxx False
in 103 in  True False False xx True
learning 106 learning  True False False xxxx False
Natural 115 Natural  True False False Xxxxx False
Language 123 Language  True False False Xxxxx False
Processing 132 Processing True False False Xxxxx False
. 142 . False True False . False

In this example, some of the commonly required attributes are accessed:

text_with_ws prints token text with trailing space (if present).
is_alpha detects if the token consists of alphabetic characters or not.
is_punct detects if the token is a punctuation symbol or not.
is_space detects if the token is a space or not.
shape_ prints out the shape of the word.
is_stop detects if the token is a stop word or not.

Note: You’ll learn more about stop words in the next section.

You can also customize the tokenization process to detect tokens on custom characters. This is often used for hyphenated words, which are words joined with hyphen. For example, “London-based” is a hyphenated word.

spaCy allows you to customize tokenization by updating the tokenizer property on the nlp object:

>>>

>>> import re
>>> import spacy
>>> from spacy.tokenizer import Tokenizer
>>> custom_nlp = spacy.load('en_core_web_sm')
>>> prefix_re = spacy.util.compile_prefix_regex(custom_nlp.Defaults.prefixes)
>>> suffix_re = spacy.util.compile_suffix_regex(custom_nlp.Defaults.suffixes)
>>> infix_re = re.compile(r'''[-~]''')
>>> def customize_tokenizer(nlp):
...     # Adds support to use `-` as the delimiter for tokenization
...     return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
...                      suffix_search=suffix_re.search,
...                      infix_finditer=infix_re.finditer,
...                      token_match=None
...                      )
...

>>> custom_nlp.tokenizer = customize_tokenizer(custom_nlp)
>>> custom_tokenizer_about_doc = custom_nlp(about_text)
>>> print([token.text for token in custom_tokenizer_about_doc])
['Gus', 'Proto', 'is', 'a', 'Python', 'developer', 'currently',
'working', 'for', 'a', 'London', '-', 'based', 'Fintech',
'company', '.', 'He', 'is', 'interested', 'in', 'learning',
'Natural', 'Language', 'Processing', '.']

In order for you to customize, you can pass various parameters to the Tokenizer class:

nlp.vocab is a storage container for special cases and is used to handle cases like contractions and emoticons.
prefix_search is the function that is used to handle preceding punctuation, such as opening parentheses.
infix_finditer is the function that is used to handle non-whitespace separators, such as hyphens.
suffix_search is the function that is used to handle succeeding punctuation, such as closing parentheses.
token_match is an optional boolean function that is used to match strings that should never be split. It overrides the previous rules and is useful for entities like URLs or numbers.

Note: spaCy already detects hyphenated words as individual tokens. The above code is just an example to show how tokenization can be customized. It can be used for any other character.

Stop Words

Stop words are the most common words in a language. In the English language, some examples of stop words are the, are, but, and they. Most sentences need to contain stop words in order to be full sentences that make sense.

Generally, stop words are removed because they aren’t significant and distort the word frequency analysis. spaCy has a list of stop words for the English language:

>>>

>>> import spacy
>>> spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS
>>> len(spacy_stopwords)
326
>>> for stop_word in list(spacy_stopwords)[:10]:
...     print(stop_word)
...
using
becomes
had
itself
once
often
is
herein
who
too

You can remove stop words from the input text:

>>>

>>> for token in about_doc:
...     if not token.is_stop:
...         print (token)
...
Gus
Proto
Python
developer
currently
working
London
-
based
Fintech
company
.
interested
learning
Natural
Language
Processing
.

Stop words like is, a, for, the, and in are not printed in the output above. You can also create a list of tokens not containing stop words:

>>>

>>> about_no_stopword_doc = [token for token in about_doc if not token.is_stop]
>>> print (about_no_stopword_doc)
[Gus, Proto, Python, developer, currently, working, London,
-, based, Fintech, company, ., interested, learning, Natural,
Language, Processing, .]

about_no_stopword_doc can be joined with spaces to form a sentence with no stop words.

Lemmatization

Lemmatization is the process of reducing inflected forms of a word while still ensuring that the reduced form belongs to the language. This reduced form or root word is called a lemma.

For example, organizes, organized and organizing are all forms of organize. Here, organize is the lemma. The inflection of a word allows you to express different grammatical categories like tense (organized vs organize), number (trains vs train), and so on. Lemmatization is necessary because it helps you reduce the inflected forms of a word so that they can be analyzed as a single item. It can also help you normalize the text.

spaCy has the attribute lemma_ on the Token class. This attribute has the lemmatized form of a token:

>>>

>>> conference_help_text = ('Gus is helping organize a developer'
...     'conference on Applications of Natural Language'
...     ' Processing. He keeps organizing local Python meetups'
...     ' and several internal talks at his workplace.')
>>> conference_help_doc = nlp(conference_help_text)
>>> for token in conference_help_doc:
...     print (token, token.lemma_)
...
Gus Gus
is be
helping help
organize organize
a a
developer developer
conference conference
on on
Applications Applications
of of
Natural Natural
Language Language
Processing Processing
. .
He -PRON-
keeps keep
organizing organize
local local
Python Python
meetups meetup
and and
several several
internal internal
talks talk
at at
his -PRON-
workplace workplace
. .

In this example, organizing reduces to its lemma form organize. If you do not lemmatize the text, then organize and organizing will be counted as different tokens, even though they both have a similar meaning. Lemmatization helps you avoid duplicate words that have similar meanings.

Word Frequency

You can now convert a given text into tokens and perform statistical analysis over it. This analysis can give you various insights about word patterns, such as common words or unique words in the text:

>>>

>>> from collections import Counter
>>> complete_text = ('Gus Proto is a Python developer currently'
...     'working for a London-based Fintech company. He is'
...     ' interested in learning Natural Language Processing.'
...     ' There is a developer conference happening on 21 July'
...     ' 2019 in London. It is titled "Applications of Natural'
...     ' Language Processing". There is a helpline number '
...     ' available at +1-1234567891. Gus is helping organize it.'
...     ' He keeps organizing local Python meetups and several'
...     ' internal talks at his workplace. Gus is also presenting'
...     ' a talk. The talk will introduce the reader about "Use'
...     ' cases of Natural Language Processing in Fintech".'
...     ' Apart from his work, he is very passionate about music.'
...     ' Gus is learning to play the Piano. He has enrolled '
...     ' himself in the weekend batch of Great Piano Academy.'
...     ' Great Piano Academy is situated in Mayfair or the City'
...     ' of London and has world-class piano instructors.')
...
>>> complete_doc = nlp(complete_text)
>>> # Remove stop words and punctuation symbols
>>> words = [token.text for token in complete_doc
...          if not token.is_stop and not token.is_punct]
>>> word_freq = Counter(words)
>>> # 5 commonly occurring words with their frequencies
>>> common_words = word_freq.most_common(5)
>>> print (common_words)
[('Gus', 4), ('London', 3), ('Natural', 3), ('Language', 3), ('Processing', 3)]
>>> # Unique words
>>> unique_words = [word for (word, freq) in word_freq.items() if freq == 1]
>>> print (unique_words)
['Proto', 'currently', 'working', 'based', 'company',
'interested', 'conference', 'happening', '21', 'July',
'2019', 'titled', 'Applications', 'helpline', 'number',
'available', '+1', '1234567891', 'helping', 'organize',
'keeps', 'organizing', 'local', 'meetups', 'internal',
'talks', 'workplace', 'presenting', 'introduce', 'reader',
'Use', 'cases', 'Apart', 'work', 'passionate', 'music', 'play',
'enrolled', 'weekend', 'batch', 'situated', 'Mayfair', 'City',
'world', 'class', 'piano', 'instructors']

By looking at the common words, you can see that the text as a whole is probably about Gus, London, or Natural Language Processing. This way, you can take any unstructured text and perform statistical analysis to know what it’s about.

Here’s another example of the same text with stop words:

>>>

>>> words_all = [token.text for token in complete_doc if not token.is_punct]
>>> word_freq_all = Counter(words_all)
>>> # 5 commonly occurring words with their frequencies
>>> common_words_all = word_freq_all.most_common(5)
>>> print (common_words_all)
[('is', 10), ('a', 5), ('in', 5), ('Gus', 4), ('of', 4)]

Four out of five of the most common words are stop words, which don’t tell you much about the text. If you consider stop words while doing word frequency analysis, then you won’t be able to derive meaningful insights from the input text. This is why removing stop words is so important.

Part of Speech Tagging

Part of speech or POS is a grammatical role that explains how a particular word is used in a sentence. There are eight parts of speech:

Noun
Pronoun
Adjective
Verb
Adverb
Preposition
Conjunction
Interjection

Part of speech tagging is the process of assigning a POS tag to each token depending on its usage in the sentence. POS tags are useful for assigning a syntactic category like noun or verb to each word.

In spaCy, POS tags are available as an attribute on the Token object:

>>>

>>> for token in about_doc:
...     print (token, token.tag_, token.pos_, spacy.explain(token.tag_))
...
Gus NNP PROPN noun, proper singular
Proto NNP PROPN noun, proper singular
is VBZ VERB verb, 3rd person singular present
a DT DET determiner
Python NNP PROPN noun, proper singular
developer NN NOUN noun, singular or mass
currently RB ADV adverb
working VBG VERB verb, gerund or present participle
for IN ADP conjunction, subordinating or preposition
a DT DET determiner
London NNP PROPN noun, proper singular
- HYPH PUNCT punctuation mark, hyphen
based VBN VERB verb, past participle
Fintech NNP PROPN noun, proper singular
company NN NOUN noun, singular or mass
. . PUNCT punctuation mark, sentence closer
He PRP PRON pronoun, personal
is VBZ VERB verb, 3rd person singular present
interested JJ ADJ adjective
in IN ADP conjunction, subordinating or preposition
learning VBG VERB verb, gerund or present participle
Natural NNP PROPN noun, proper singular
Language NNP PROPN noun, proper singular
Processing NNP PROPN noun, proper singular
. . PUNCT punctuation mark, sentence closer

Here, two attributes of the Token class are accessed:

tag_ lists the fine-grained part of speech.
pos_ lists the coarse-grained part of speech.

spacy.explain gives descriptive details about a particular POS tag. spaCy provides a complete tag list along with an explanation for each tag.

Using POS tags, you can extract a particular category of words:

>>>

>>> nouns = []
>>> adjectives = []
>>> for token in about_doc:
...     if token.pos_ == 'NOUN':
...         nouns.append(token)
...     if token.pos_ == 'ADJ':
...         adjectives.append(token)
...
>>> nouns
[developer, company]
>>> adjectives
[interested]

You can use this to derive insights, remove the most common nouns, or see which adjectives are used for a particular noun.

Visualization: Using displaCy

spaCy comes with a built-in visualizer called displaCy. You can use it to visualize a dependency parse or named entities in a browser or a Jupyter notebook.

You can use displaCy to find POS tags for tokens:

>>>

>>> from spacy import displacy
>>> about_interest_text = ('He is interested in learning'
...     ' Natural Language Processing.')
>>> about_interest_doc = nlp(about_interest_text)
>>> displacy.serve(about_interest_doc, style='dep')

The above code will spin a simple web server. You can see the visualization by opening http://127.0.0.1:5000 in your browser:

displaCy: Part of Speech Tagging Demo

In the image above, each token is assigned a POS tag written just below the token.

Note: Here’s how you can use displaCy in a Jupyter notebook:

>>>

>>> displacy.render(about_interest_doc, style='dep', jupyter=True)

Preprocessing Functions

You can create a preprocessing function that takes text as input and applies the following operations:

Lowercases the text
Lemmatizes each token
Removes punctuation symbols
Removes stop words

A preprocessing function converts text to an analyzable format. It’s necessary for most NLP tasks. Here’s an example:

>>>

>>> def is_token_allowed(token):
...     '''
...         Only allow valid tokens which are not stop words
...         and punctuation symbols.
...     '''
...     if (not token or not token.string.strip() or
...         token.is_stop or token.is_punct):
...         return False
...     return True
...
>>> def preprocess_token(token):
...     # Reduce token to its lowercase lemma form
...     return token.lemma_.strip().lower()
...
>>> complete_filtered_tokens = [preprocess_token(token)
...     for token in complete_doc if is_token_allowed(token)]
>>> complete_filtered_tokens
['gus', 'proto', 'python', 'developer', 'currently', 'work',
'london', 'base', 'fintech', 'company', 'interested', 'learn',
'natural', 'language', 'processing', 'developer', 'conference',
'happen', '21', 'july', '2019', 'london', 'title',
'applications', 'natural', 'language', 'processing', 'helpline',
'number', 'available', '+1', '1234567891', 'gus', 'help',
'organize', 'keep', 'organize', 'local', 'python', 'meetup',
'internal', 'talk', 'workplace', 'gus', 'present', 'talk', 'talk',
'introduce', 'reader', 'use', 'case', 'natural', 'language',
'processing', 'fintech', 'apart', 'work', 'passionate', 'music',
'gus', 'learn', 'play', 'piano', 'enrol', 'weekend', 'batch',
'great', 'piano', 'academy', 'great', 'piano', 'academy',
'situate', 'mayfair', 'city', 'london', 'world', 'class',
'piano', 'instructor']

Note that the complete_filtered_tokens does not contain any stop word or punctuation symbols and consists of lemmatized lowercase tokens.

Rule-Based Matching Using spaCy

Rule-based matching is one of the steps in extracting information from unstructured text. It’s used to identify and extract tokens and phrases according to patterns (such as lowercase) and grammatical features (such as part of speech).

Rule-based matching can use regular expressions to extract entities (such as phone numbers) from an unstructured text. It’s different from extracting text using regular expressions only in the sense that regular expressions don’t consider the lexical and grammatical attributes of the text.

With rule-based matching, you can extract a first name and a last name, which are always proper nouns:

>>>

>>> from spacy.matcher import Matcher
>>> matcher = Matcher(nlp.vocab)
>>> def extract_full_name(nlp_doc):
...     pattern = [{'POS': 'PROPN'}, {'POS': 'PROPN'}]
...     matcher.add('FULL_NAME', None, pattern)
...     matches = matcher(nlp_doc)
...     for match_id, start, end in matches:
...         span = nlp_doc[start:end]
...         return span.text
...
>>> extract_full_name(about_doc)
'Gus Proto'

In this example, pattern is a list of objects that defines the combination of tokens to be matched. Both POS tags in it are PROPN (proper noun). So, the pattern consists of two objects in which the POS tags for both tokens should be PROPN. This pattern is then added to Matcher using FULL_NAME and the the match_id. Finally, matches are obtained with their starting and end indexes.

You can also use rule-based matching to extract phone numbers:

>>>

>>> from spacy.matcher import Matcher
>>> matcher = Matcher(nlp.vocab)
>>> conference_org_text = ('There is a developer conference'
...     'happening on 21 July 2019 in London. It is titled'
...     ' "Applications of Natural Language Processing".'
...     ' There is a helpline number available'
...     ' at (123) 456-789')
...
>>> def extract_phone_number(nlp_doc):
...     pattern = [{'ORTH': '('}, {'SHAPE': 'ddd'},
...                {'ORTH': ')'}, {'SHAPE': 'ddd'},
...                {'ORTH': '-', 'OP': '?'},
...                {'SHAPE': 'ddd'}]
...     matcher.add('PHONE_NUMBER', None, pattern)
...     matches = matcher(nlp_doc)
...     for match_id, start, end in matches:
...         span = nlp_doc[start:end]
...         return span.text
...
>>> conference_org_doc = nlp(conference_org_text)
>>> extract_phone_number(conference_org_doc)
'(123) 456-789'

In this example, only the pattern is updated in order to match phone numbers from the previous example. Here, some attributes of the token are also used:

ORTH gives the exact text of the token.
SHAPE transforms the token string to show orthographic features.
OP defines operators. Using ? as a value means that the pattern is optional, meaning it can match 0 or 1 times.

Note: For simplicity, phone numbers are assumed to be of a particular format: (123) 456-789. You can change this depending on your use case.

Rule-based matching helps you identify and extract tokens and phrases according to lexical patterns (such as lowercase) and grammatical features(such as part of speech).

Dependency Parsing Using spaCy

Dependency parsing is the process of extracting the dependency parse of a sentence to represent its grammatical structure. It defines the dependency relationship between headwords and their dependents. The head of a sentence has no dependency and is called the root of the sentence. The verb is usually the head of the sentence. All other words are linked to the headword.

The dependencies can be mapped in a directed graph representation:

Words are the nodes.
The grammatical relationships are the edges.

Dependency parsing helps you know what role a word plays in the text and how different words relate to each other. It’s also used in shallow parsing and named entity recognition.

Here’s how you can use dependency parsing to see the relationships between words:

>>>

>>> piano_text = 'Gus is learning piano'
>>> piano_doc = nlp(piano_text)
>>> for token in piano_doc:
...     print (token.text, token.tag_, token.head.text, token.dep_)
...
Gus NNP learning nsubj
is VBZ learning aux
learning VBG learning ROOT
piano NN learning dobj

In this example, the sentence contains three relationships:

nsubj is the subject of the word. Its headword is a verb.
aux is an auxiliary word. Its headword is a verb.
dobj is the direct object of the verb. Its headword is a verb.

There is a detailed list of relationships with descriptions. You can use displaCy to visualize the dependency tree:

>>>

>>> displacy.serve(piano_doc, style='dep')

This code will produce a visualization that can be accessed by opening http://127.0.0.1:5000 in your browser:

displaCy: Dependency Parse Demo

This image shows you that the subject of the sentence is the proper noun Gus and that it has a learn relationship with piano.

Navigating the Tree and Subtree

The dependency parse tree has all the properties of a tree. This tree contains information about sentence structure and grammar and can be traversed in different ways to extract relationships.

spaCy provides attributes like children, lefts, rights, and subtree to navigate the parse tree:

>>>

>>> one_line_about_text = ('Gus Proto is a Python developer'
...     ' currently working for a London-based Fintech company')
>>> one_line_about_doc = nlp(one_line_about_text)
>>> # Extract children of `developer`
>>> print([token.text for token in one_line_about_doc[5].children])
['a', 'Python', 'working']
>>> # Extract previous neighboring node of `developer`
>>> print (one_line_about_doc[5].nbor(-1))
Python
>>> # Extract next neighboring node of `developer`
>>> print (one_line_about_doc[5].nbor())
currently
>>> # Extract all tokens on the left of `developer`
>>> print([token.text for token in one_line_about_doc[5].lefts])
['a', 'Python']
>>> # Extract tokens on the right of `developer`
>>> print([token.text for token in one_line_about_doc[5].rights])
['working']
>>> # Print subtree of `developer`
>>> print (list(one_line_about_doc[5].subtree))
[a, Python, developer, currently, working, for, a, London, -,
based, Fintech, company]

You can construct a function that takes a subtree as an argument and returns a string by merging words in it:

>>>

>>> def flatten_tree(tree):
...     return ''.join([token.text_with_ws for token in list(tree)]).strip()
...
>>> # Print flattened subtree of `developer`
>>> print (flatten_tree(one_line_about_doc[5].subtree))
a Python developer currently working for a London-based Fintech company

You can use this function to print all the tokens in a subtree.

Shallow Parsing

Shallow parsing, or chunking, is the process of extracting phrases from unstructured text. Chunking groups adjacent tokens into phrases on the basis of their POS tags. There are some standard well-known chunks such as noun phrases, verb phrases, and prepositional phrases.

Noun Phrase Detection

A noun phrase is a phrase that has a noun as its head. It could also include other kinds of words, such as adjectives, ordinals, determiners. Noun phrases are useful for explaining the context of the sentence. They help you infer what is being talked about in the sentence.

spaCy has the property noun_chunks on Doc object. You can use it to extract noun phrases:

>>>

>>> conference_text = ('There is a developer conference'
...     ' happening on 21 July 2019 in London.')
>>> conference_doc = nlp(conference_text)
>>> # Extract Noun Phrases
>>> for chunk in conference_doc.noun_chunks:
...     print (chunk)
...
a developer conference
21 July
London

By looking at noun phrases, you can get information about your text. For example, a developer conference indicates that the text mentions a conference, while the date 21 July lets you know that conference is scheduled for 21 July. You can figure out whether the conference is in the past or the future. London tells you that the conference is in London.

Verb Phrase Detection

A verb phrase is a syntactic unit composed of at least one verb. This verb can be followed by other chunks, such as noun phrases. Verb phrases are useful for understanding the actions that nouns are involved in.

spaCy has no built-in functionality to extract verb phrases, so you’ll need a library called textacy:

Note:

You can use pip to install textacy:

$ pip install textacy

Now that you have textacy installed, you can use it to extract verb phrases based on grammar rules:

>>>

>>> import textacy
>>> about_talk_text = ('The talk will introduce reader about Use'
...                    ' cases of Natural Language Processing in'
...                    ' Fintech')
>>> pattern = r'(<VERB>?<ADV>*<VERB>+)'
>>> about_talk_doc = textacy.make_spacy_doc(about_talk_text,
...                                         lang='en_core_web_sm')
>>> verb_phrases = textacy.extract.pos_regex_matches(about_talk_doc, pattern)
>>> # Print all Verb Phrase
>>> for chunk in verb_phrases:
...     print(chunk.text)
...
will introduce
>>> # Extract Noun Phrase to explain what nouns are involved
>>> for chunk in about_talk_doc.noun_chunks:
...     print (chunk)
...
The talk
reader
Use cases
Natural Language Processing
Fintech

In this example, the verb phrase introduce indicates that something will be introduced. By looking at noun phrases, you can see that there is a talk that will introduce the reader to use cases of Natural Language Processing or Fintech.

The above code extracts all the verb phrases using a regular expression pattern of POS tags. You can tweak the pattern for verb phrases depending upon your use case.

Note: In the previous example, you could have also done dependency parsing to see what the relationships between the words were.

Named Entity Recognition

Named Entity Recognition (NER) is the process of locating named entities in unstructured text and then classifying them into pre-defined categories, such as person names, organizations, locations, monetary values, percentages, time expressions, and so on.

You can use NER to know more about the meaning of your text. For example, you could use it to populate tags for a set of documents in order to improve the keyword search. You could also use it to categorize customer support tickets into relevant categories.

spaCy has the property ents on Doc objects. You can use it to extract named entities:

>>>

>>> piano_class_text = ('Great Piano Academy is situated'
...     ' in Mayfair or the City of London and has'
...     ' world-class piano instructors.')
>>> piano_class_doc = nlp(piano_class_text)
>>> for ent in piano_class_doc.ents:
...     print(ent.text, ent.start_char, ent.end_char,
...           ent.label_, spacy.explain(ent.label_))
...
Great Piano Academy 0 19 ORG Companies, agencies, institutions, etc.
Mayfair 35 42 GPE Countries, cities, states
the City of London 46 64 GPE Countries, cities, states

In the above example, ent is a Span object with various attributes:

text gives the Unicode text representation of the entity.
start_char denotes the character offset for the start of the entity.
end_char denotes the character offset for the end of the entity.
label_ gives the label of the entity.

spacy.explain gives descriptive details about an entity label. The spaCy model has a pre-trained list of entity classes. You can use displaCy to visualize these entities:

>>>

>>> displacy.serve(piano_class_doc, style='ent')

If you open http://127.0.0.1:5000 in your browser, then you can see the visualization:

displaCy: Named Entity Recognition Demo

You can use NER to redact people’s names from a text. For example, you might want to do this in order to hide personal information collected in a survey. You can use spaCy to do that:

>>>

>>> survey_text = ('Out of 5 people surveyed, James Robert,'
...                ' Julie Fuller and Benjamin Brooks like'
...                ' apples. Kelly Cox and Matthew Evans'
...                ' like oranges.')
...
>>> def replace_person_names(token):
...     if token.ent_iob != 0 and token.ent_type_ == 'PERSON':
...         return '[REDACTED] '
...     return token.string
...
>>> def redact_names(nlp_doc):
...     for ent in nlp_doc.ents:
...         ent.merge()
...     tokens = map(replace_person_names, nlp_doc)
...     return ''.join(tokens)
...
>>> survey_doc = nlp(survey_text)
>>> redact_names(survey_doc)
'Out of 5 people surveyed, [REDACTED] , [REDACTED] and'
' [REDACTED] like apples. [REDACTED] and [REDACTED]'
' like oranges.'

In this example, replace_person_names() uses ent_iob. It gives the IOB code of the named entity tag using inside-outside-beginning (IOB) tagging. Here, it can assume a value other than zero, because zero means that no entity tag is set.

Conclusion

spaCy is a powerful and advanced library that is gaining huge popularity for NLP applications due to its speed, ease of use, accuracy, and extensibility. Congratulations! You now know:

What the foundational terms and concepts in NLP are
How to implement those concepts in spaCy
How to customize and extend built-in functionalities in spaCy
How to perform basic statistical analysis on a text
How to create a pipeline to process unstructured text
How to parse a sentence and extract meaningful insights from it

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyCharm for Productive Python Development (Guide)

2019-08-28T14:00:00+00:00

As a programmer, you should be focused on the business logic and creating useful applications for your users. In doing that, PyCharm by JetBrains saves you a lot of time by taking care of the routine and by making a number of other tasks such as debugging and visualization easy.

In this article, you’ll learn about:

Installing PyCharm
Writing code in PyCharm
Running your code in PyCharm
Debugging and testing your code in PyCharm
Editing an existing project in PyCharm
Searching and navigating in PyCharm
Using Version Control in PyCharm
Using Plugins and External Tools in PyCharm
Using PyCharm Professional features, such as Django support and Scientific mode

This article assumes that you’re familiar with Python development and already have some form of Python installed on your system. Python 3.6 will be used for this tutorial. Screenshots and demos provided are for macOS. Because PyCharm runs on all major platforms, you may see slightly different UI elements and may need to modify certain commands.

Note:

PyCharm comes in three editions:

PyCharm Edu is free and for educational purposes.
PyCharm Community is free as well and intended for pure Python development.
PyCharm Professional is paid, has everything the Community edition has and also is very well suited for Web and Scientific development with support for such frameworks as Django and Flask, Database and SQL, and scientific tools such as Jupyter.

For more details on their differences, check out the PyCharm Editions Comparison Matrix by JetBrains. The company also has special offers for students, teachers, open source projects, and other cases.

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

Installing PyCharm

This article will use PyCharm Community Edition 2019.1 as it’s free and available on every major platform. Only the section about the professional features will use PyCharm Professional Edition 2019.1.

The recommended way of installing PyCharm is with the JetBrains Toolbox App. With its help, you’ll be able to install different JetBrains products or several versions of the same product, update, rollback, and easily remove any tool when necessary. You’ll also be able to quickly open any project in the right IDE and version.

To install the Toolbox App, refer to the documentation by JetBrains. It will automatically give you the right instructions depending on your OS. In case it didn’t recognize your OS correctly, you can always find it from the drop down list on the top right section:

After installing, launch the app and accept the user agreement. Under the Tools tab, you’ll see a list of available products. Find PyCharm Community there and click Install:

Voilà! You have PyCharm available on your machine. If you don’t want to use the Toolbox app, then you can also do a stand-alone installation of PyCharm.

Launch PyCharm, and you’ll see the import settings popup:

PyCharm will automatically detect that this is a fresh install and choose Do not import settings for you. Click OK, and PyCharm will ask you to select a keymap scheme. Leave the default and click Next: UI Themes on the bottom right:

PyCharm will then ask you to choose a dark theme called Darcula or a light theme. Choose whichever you prefer and click Next: Launcher Script:

I’ll be using the dark theme Darcula throughout this tutorial. You can find and install other themes as plugins, or you can also import them.

On the next page, leave the defaults and click Next: Featured plugins. There, PyCharm will show you a list of plugins you may want to install because most users like to use them. Click Start using PyCharm, and now you are ready to write some code!

Writing Code in PyCharm

In PyCharm, you do everything in the context of a project. Thus, the first thing you need to do is create one.

After installing and opening PyCharm, you are on the welcome screen. Click Create New Project, and you’ll see the New Project popup:

Specify the project location and expand the Project Interpreter drop down. Here, you have options to create a new project interpreter or reuse an existing one. Choose New environment using. Right next to it, you have a drop down list to select one of Virtualenv, Pipenv, or Conda, which are the tools that help to keep dependencies required by different projects separate by creating isolated Python environments for them.

You are free to select whichever you like, but Virtualenv is used for this tutorial. If you choose to, you can specify the environment location and choose the base interpreter from the list, which is a list of Python interpreters (such as Python2.7 and Python3.6) installed on your system. Usually, the defaults are fine. Then you have to select boxes to inherit global site-packages to your new environment and make it available to all other projects. Leave them unselected.

Click Create on the bottom right and you will see the new project created:

You will also see a small Tip of the Day popup where PyCharm gives you one trick to learn at each startup. Go ahead and close this popup.

It is now time to start a new Python program. Type Cmd+N if you are on Mac or Alt+Ins if you are on Windows or Linux. Then, choose Python File. You can also select File → New from the menu. Name the new file guess_game.py and click OK. You will see a PyCharm window similar to the following:

For our test code, let’s quickly code up a simple guessing game in which the program chooses a number that the user has to guess. For every guess, the program will tell if the user’s guess was smaller or bigger than the secret number. The game ends when the user guesses the number. Here’s the code for the game:

 1 from random import randint
 2 
 3 def play():
 4     random_int = randint(0, 100)
 5 
 6     while True:
 7         user_guess = int(input("What number did we guess (0-100)?"))
 8 
 9         if user_guess == randint:
10             print(f"You found the number ({random_int}). Congrats!")
11             break
12 
13         if user_guess < random_int:
14             print("Your number is less than the number we guessed.")
15             continue
16 
17         if user_guess > random_int:
18             print("Your number is more than the number we guessed.")
19             continue
20 
21 
22 if __name__ == '__main__':
23     play()

Type this code directly rather than copying and pasting. You’ll see something like this:

As you can see, PyCharm provides Intelligent Coding Assistance with code completion, code inspections, on-the-fly error highlighting, and quick-fix suggestions. In particular, note how when you typed main and then hit tab, PyCharm auto-completed the whole main clause for you.

Also note how, if you forget to type if before the condition, append .if, and then hit Tab, PyCharm fixes the if clause for you. The same is true with True.while. That’s PyCharm’s Postfix completions working for you to help reduce backward caret jumps.

Running Code in PyCharm

Now that you’ve coded up the game, it’s time for you to run it.

You have three ways of running this program:

Use the shortcut Ctrl+Shift+R on Mac or Ctrl+Shift+F10 on Windows or Linux.
Right-click the background and choose Run ‘guess_game’ from the menu.
Since this program has the __main__ clause, you can click on the little green arrow to the left of the __main__ clause and choose Run ‘guess_game’ from there.

Use any one of the options above to run the program, and you’ll see the Run Tool pane appear at the bottom of the window, with your code output showing:

Play the game for a little bit to see if you can find the number guessed. Pro tip: start with 50.

Debugging in PyCharm

Did you find the number? If so, you may have seen something weird after you found the number. Instead of printing the congratulations message and exiting, the program seems to start over. That’s a bug right there. To discover why the program starts over, you’ll now debug the program.

First, place a breakpoint by clicking on the blank space to the left of line number 8:

This will be the point where the program will be suspended, and you can start exploring what went wrong from there on. Next, choose one of the following three ways to start debugging:

Press Ctrl+Shift+D on Mac or Shift+Alt+F9 on Windows or Linux.
Right-click the background and choose Debug ‘guess_game’.
Click on the little green arrow to the left of the __main__ clause and choose Debug ‘guess_game from there.

Afterwards, you’ll see a Debug window open at the bottom:

Follow the steps below to debug the program:

Notice that the current line is highlighted in blue.
See that random_int and its value are listed in the Debug window. Make a note of this number. (In the picture, the number is 85.)
Hit F8 to execute the current line and step over to the next one. You can also use F7 to step into the function in the current line, if necessary. As you continue executing the statements, the changes in the variables will be automatically reflected in the Debugger window.
Notice that there is the Console tab right next to the Debugger tab that opened. This Console tab and the Debugger tab are mutually exclusive. In the Console tab, you will be interacting with your program, and in the Debugger tab you will do the debugging actions.
Switch to the Console tab to enter your guess.
Type the number shown, and then hit Enter.
Switch back to the Debugger tab.
Hit F8 again to evaluate the if statement. Notice that you are now on line 14. But wait a minute! Why didn’t it go to the line 11? The reason is that the if statement on line 10 evaluated to False. But why did it evaluate to False when you entered the number that was chosen?
Look carefully at line 10 and notice that we are comparing user_guess with the wrong thing. Instead of comparing it with random_int, we are comparing it with randint, the function that was imported from the random package.
Change it to random_int, restart the debugging, and follow the same steps again. You will see that, this time, it will go to line 11, and line 10 will evaluate to True:

Congratulations! You fixed the bug.

Testing in PyCharm

No application is reliable without unit tests. PyCharm helps you write and run them very quickly and comfortably. By default, unittest is used as the test runner, but PyCharm also supports other testing frameworks such as pytest, nose, doctest, tox, and trial. You can, for example, enable pytest for your project like this:

Open the Settings/Preferences → Tools → Python Integrated Tools settings dialog.
Select pytest in the Default test runner field.
Click OK to save the settings.

For this example, we’ll be using the default test runner unittest.

In the same project, create a file called calculator.py and put the following Calculator class in it:

 1 class Calculator:
 2     def add(self, a, b):
 3         return a + b
 4 
 5     def multiply(self, a, b):
 6         return a * b

PyCharm makes it very easy to create tests for your existing code. With the calculator.py file open, execute any one of the following that you like:

Press Shift+Cmd+T on Mac or Ctrl+Shift+T on Windows or Linux.
Right-click in the background of the class and then choose Go To and Test.
On the main menu, choose Navigate → Test.

Choose Create New Test…, and you will see the following window:

Leave the defaults of Target directory, Test file name, and Test class name. Select both of the methods and click OK. Voila! PyCharm automatically created a file called test_calculator.py and created the following stub tests for you in it:

 1 from unittest import TestCase
 2 
 3 class TestCalculator(TestCase):
 4     def test_add(self):
 5         self.fail()
 6 
 7     def test_multiply(self):
 8         self.fail()

Run the tests using one of the methods below:

Press Ctrl+R on Mac or Shift+F10 on Windows or Linux.
Right-click the background and choose Run ‘Unittests for test_calculator.py’.
Click on the little green arrow to the left of the test class name and choose Run ‘Unittests for test_calculator.py’.

You’ll see the tests window open on the bottom with all the tests failing:

Notice that you have the hierarchy of the test results on the left and the output of the terminal on the right.

Now, implement test_add by changing the code to the following:

 1 from unittest import TestCase
 2 
 3 from calculator import Calculator
 4 
 5 class TestCalculator(TestCase):
 6     def test_add(self):
 7         self.calculator = Calculator()
 8         self.assertEqual(self.calculator.add(3, 4), 7)
 9 
10     def test_multiply(self):
11         self.fail()

Run the tests again, and you’ll see that one test passed and the other failed. Explore the options to show passed tests, to show ignored tests, to sort tests alphabetically, and to sort tests by duration:

Note that the sleep(0.1) method that you see in the GIF above is intentionally used to make one of the tests slower so that sorting by duration works.

Editing an Existing Project in PyCharm

These single file projects are great for examples, but you’ll often work on much larger projects over a longer period of time. In this section, you’ll take a look at how PyCharm works with a larger project.

To explore the project-focused features of PyCharm, you’ll use the Alcazar web framework that was built for learning purposes. To continue following along, clone the repo locally:

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

Once you have a project locally, open it in PyCharm using one of the following methods:

Click File → Open on the main menu.
Click Open on the Welcome Screen if you are there.

After either of these steps, find the folder containing the project on your computer and open it.

If this project contains a virtual environment, then PyCharm will automatically use this virtual environment and make it the project interpreter.

If you need to configure a different virtualenv, then open Preferences on Mac by pressing Cmd+, or Settings on Windows or Linux by pressing Ctrl+Alt+S and find the Project: ProjectName section. Open the drop-down and choose Project Interpreter:

Choose the virtualenv from the drop-down list. If it’s not there, then click on the settings button to the right of the drop-down list and then choose Add…. The rest of the steps should be the same as when we were creating a new project.

Searching and Navigating in PyCharm

In a big project where it’s difficult for a single person to remember where everything is located, it’s very important to be able to quickly navigate and find what you looking for. PyCharm has you covered here as well. Use the project you opened in the section above to practice these shortcuts:

Searching for a fragment in the current file: Press Cmd+F on Mac or Ctrl+F on Windows or Linux.
Searching for a fragment in the entire project: Press Cmd+Shift+F on Mac or Ctrl+Shift+F on Windows or Linux.
Searching for a class: Press Cmd+O on Mac or Ctrl+N on Windows or Linux.
Searching for a file: Press Cmd+Shift+O on Mac or Ctrl+Shift+N on Windows or Linux.
Searching all if you don’t know whether it’s a file, class, or a code fragment that you are looking for: Press Shift twice.

As for the navigation, the following shortcuts may save you a lot of time:

Going to the declaration of a variable: Press Cmd on Mac or Ctrl on Windows or Linux, and click on the variable.
Finding usages of a class, a method, or any symbol: Press Alt+F7.
Seeing your recent changes: Press Shift+Alt+C or go to View → Recent Changes on the main menu.
Seeing your recent files: Press Cmd+E on Mac or Ctrl+E on Windows or Linux, or go to View → Recent Files on the main menu.
Going backward and forward through your history of navigation after you jumped around: Press Cmd+[ / Cmd+] on Mac or Ctrl+Alt+Left / Ctrl+Alt+Right on Windows or Linux.

For more details, see the official documentation.

Using Version Control in PyCharm

Version control systems such as Git and Mercurial are some of the most important tools in the modern software development world. So, it is essential for an IDE to support them. PyCharm does that very well by integrating with a lot of popular VC systems such as Git (and Github), Mercurial, Perforce and, Subversion.

Note: Git is used for the following examples.

Configuring VCS

To enable VCS integration. Go to VCS → VCS Operations Popup… from the menu on the top or press Ctrl+V on Mac or Alt+` on Windows or Linux. Choose Enable Version Control Integration…. You’ll see the following window open:

Choose Git from the drop down list, click OK, and you have VCS enabled for your project. Note that if you opened an existing project that has version control enabled, then PyCharm will see that and automatically enable it.

Now, if you go to the VCS Operations Popup…, you’ll see a different popup with the options to do git add, git stash, git branch, git commit, git push and more:

If you can’t find what you need, you can most probably find it by going to VCS from the top menu and choosing Git, where you can even create and view pull requests.

Committing and Conflict Resolution

These are two features of VCS integration in PyCharm that I personally use and enjoy a lot! Let’s say you have finished your work and want to commit it. Go to VCS → VCS Operations Popup… → Commit… or press Cmd+K on Mac or Ctrl+K on Windows or Linux. You’ll see the following window open:

In this window, you can do the following:

Choose which files to commit
Write your commit message
Do all kinds of checks and cleanup before commit
See the difference of changes
Commit and push at once by pressing the arrow to the right of the Commit button on the right bottom and choosing Commit and Push…

It can feel magical and fast, especially if you’re used to doing everything manually on the command line.

When you work in a team, merge conflicts do happen. When somebody commits changes to a file that you’re working on, but their changes overlap with yours because both of you changed the same lines, then VCS will not be able to figure out if it should choose your changes or those of your teammate. So you’ll get these unfortunate arrows and symbols:

This looks strange, and it’s difficult to figure out which changes should be deleted and which ones should stay. PyCharm to the rescue! It has a much nicer and cleaner way of resolving conflicts. Go to VCS in the top menu, choose Git and then Resolve conflicts…. Choose the file whose conflicts you want to resolve and click on Merge. You will see the following window open:

On the left column, you will see your changes. On the right one, the changes made by your teammate. Finally, in the middle column, you will see the result. The conflicting lines are highlighted, and you can see a little X and >>/<< right beside those lines. Press the arrows to accept the changes and the X to decline. After you resolve all those conflicts, click the Apply button:

In the GIF above, for the first conflicting line, the author declined his own changes and accepted those of his teammates. Conversely, the author accepted his own changes and declined his teammates’ for the second conflicting line.

There’s a lot more that you can do with the VCS integration in PyCharm. For more details, see this documentation.

Using Plugins and External Tools in PyCharm

You can find almost everything you need for development in PyCharm. If you can’t, there is most probably a plugin that adds that functionality you need to PyCharm. For example, they can:

Add support for various languages and frameworks
Boost your productivity with shortcut hints, file watchers, and so on
Help you learn a new programming language with coding exercises

For instance, IdeaVim adds Vim emulation to PyCharm. If you like Vim, this can be a pretty good combination.

Material Theme UI changes the appearance of PyCharm to a Material Design look and feel:

Vue.js adds support for Vue.js projects. Markdown provides the capability to edit Markdown files within the IDE and see the rendered HTML in a live preview. You can find and install all of the available plugins by going to the Preferences → Plugins on Mac or Settings → Plugins on Windows or Linux, under the Marketplace tab:

If you can’t find what you need, you can even develop your own plugin.

If you can’t find the right plugin and don’t want to develop your own because there’s already a package in PyPI, then you can add it to PyCharm as an external tool. Take Flake8, the code analyzer, as an example.

First, install flake8 in your virtualenv with pip install flake8 in the Terminal app of your choice. You can also use the one integrated into PyCharm:

Then, go to Preferences → Tools on Mac or Settings → Tools on Windows/Linux, and then choose External Tools. Then click on the little + button at the bottom (1). In the new popup window, insert the details as shown below and click OK for both windows:

Here, Program (2) refers to the Flake8 executable that can be found in the folder /bin of your virtual environment. Arguments (3) refers to which file you want to analyze with the help of Flake8. Working directory is the directory of your project.

You could hardcode the absolute paths for everything here, but that would mean that you couldn’t use this external tool in other projects. You would be able to use it only inside one project for one file.

So you need to use something called Macros. Macros are basically variables in the format of $name$ that change according to your context. For example, $FileName$ is first.py when you’re editing first.py, and it is second.py when you’re editing second.py. You can see their list and insert any of them by clicking on the Insert Macro… buttons. Because you used macros here, the values will change according to the project you’re currently working on, and Flake8 will continue to do its job properly.

In order to use it, create a file example.py and put the following code in it:

 1 CONSTANT_VAR = 1
 2 
 3 
 4 
 5 def add(a, b):
 6     c = "hello"
 7     return a + b

It deliberately breaks some of the Flake8 rules. Right-click the background of this file. Choose External Tools and then Flake8. Voilà! The output of the Flake8 analysis will appear at the bottom:

In order to make it even better, you can add a shortcut for it. Go to Preferences on Mac or to Settings on Windows or Linux. Then, go to Keymap → External Tools → External Tools. Double-click Flake8 and choose Add Keyboard Shortcut. You’ll see this window:

In the image above, the shortcut is Ctrl+Alt+A for this tool. Add your preferred shortcut in the textbox and click OK for both windows. Now you can now use that shortcut to analyze the file you’re currently working on with Flake8.

PyCharm Professional Features

PyCharm Professional is a paid version of PyCharm with more out-of-the-box features and integrations. In this section, you’ll mainly be presented with overviews of its main features and links to the official documentation, where each feature is discussed in detail. Remember that none of the following features is available in the Community edition.

Django Support

PyCharm has extensive support for Django, one of the most popular and beloved Python web frameworks. To make sure that it’s enabled, do the following:

Open Preferences on Mac or Settings on Windows or Linux.
Choose Languages and Frameworks.
Choose Django.
Check the checkbox Enable Django support.
Apply changes.

Now that you’ve enabled Django support, your Django development journey will be a lot easier in PyCharm:

When creating a project, you’ll have a dedicated Django project type. This means that, when you choose this type, you’ll have all the necessary files and settings. This is the equivalent of using django-admin startproject mysite.
You can run manage.py commands directly inside PyCharm.
Django templates are supported, including:
- Syntax and error highlighting
- Code completion
- Navigation
- Completion for block names
- Completion for custom tags and filters
- Quick documentation for tags and filters
- Capability to debug them
Code completion in all other Django parts such as views, URLs and models, and code insight support for Django ORM.
Model dependency diagrams for Django models.

For more details on Django support, see the official documentation.

Database Support

Modern database development is a complex task with many supporting systems and workflows. That’s why JetBrains, the company behind PyCharm, developed a standalone IDE called DataGrip for that. It’s a separate product from PyCharm with a separate license.

Luckily, PyCharm supports all the features that are available in DataGrip through a plugin called Database tools and SQL, which is enabled by default. With the help of it, you can query, create and manage databases whether they’re working locally, on a server, or in the cloud. The plugin supports MySQL, PostgreSQL, Microsoft SQL Server, SQLite, MariaDB, Oracle, Apache Cassandra, and others. For more information on what you can do with this plugin, check out the comprehensive documentation on the database support.

Thread Concurrency Visualization

Django Channels, asyncio, and the recent frameworks like Starlette are examples of a growing trend in asynchronous Python programming. While it’s true that asynchronous programs do bring a lot of benefits to the table, it’s also notoriously hard to write and debug them. In such cases, Thread Concurrency Visualization can be just what the doctor ordered because it helps you take full control over your multi-threaded applications and optimize them.

Check out the comprehensive documentation of this feature for more details.

Profiler

Speaking of optimization, profiling is another technique that you can use to optimize your code. With its help, you can see which parts of your code are taking most of the execution time. A profiler runs in the following order of priority:

If you don’t have vmprof or yappi installed, then it’ll fall back to the standard cProfile. It’s well-documented, so I won’t rehash it here.

Scientific Mode

Python is not only a language for general and web programming. It also emerged as the best tool for data science and machine learning over these last years thanks to libraries and tools like NumPy, SciPy, scikit-learn, Matplotlib, Jupyter, and more. With such powerful libraries available, you need a powerful IDE to support all the functions such as graphing and analyzing those libraries have. PyCharm provides everything you need as thoroughly documented here.

Remote Development

One common cause of bugs in many applications is that development and production environments differ. Although, in most cases, it’s not possible to provide an exact copy of the production environment for development, pursuing it is a worthy goal.

With PyCharm, you can debug your application using an interpreter that is located on the other computer, such as a Linux VM. As a result, you can have the same interpreter as your production environment to fix and avoid many bugs resulting from the difference between development and production environments. Make sure to check out the official documentation to learn more.

Conclusion

PyCharm is one of best, if not the best, full-featured, dedicated, and versatile IDEs for Python development. It offers a ton of benefits, saving you a lot of time by helping you with routine tasks. Now you know how to be productive with it!

In this article, you learned about a lot, including:

Installing PyCharm
Writing code in PyCharm
Running your code in PyCharm
Debugging and testing your code in PyCharm
Editing an existing project in PyCharm
Searching and navigating in PyCharm
Using Version Control in PyCharm
Using Plugins and External Tools in PyCharm
Using PyCharm Professional features, such as Django support and Scientific mode

If there’s anything you’d like to ask or share, please reach out in the comments below. There’s also a lot more information at the PyCharm website for you to explore.

Clone Repo: Click here to clone the repo you'll use to explore the project-focused features of PyCharm in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use Python Lambda Functions

2019-08-27T14:00:00+00:00

Python and other languages like Java, C#, and even C++ have had lambda functions added to their syntax, whereas languages like LISP or the ML family of languages, Haskell, OCaml, and F#, use lambdas as a core concept. Python lambdas are little, anonymous functions, subject to a more restrictive but more concise syntax than regular Python functions.

By the end of this course, you’ll know:

How Python lambdas came to be
How lambdas compare with regular function objects
How to write lambda functions
Which functions in the Python standard library leverage lambdas
When to use or avoid Python lambda functions

This course is mainly for intermediate to experienced Python programmers, but it is accessible to any curious minds with interest in programming. All the examples included in this tutorial have been tested with Python 3.7.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Guide to Excel Spreadsheets in Python With openpyxl

2019-08-26T14:00:00+00:00

Excel spreadsheets are one of those things you might have to deal with at some point. Either it’s because your boss loves them or because marketing needs them, you might have to learn how to work with spreadsheets, and that’s when knowing openpyxl comes in handy!

Spreadsheets are a very intuitive and user-friendly way to manipulate large datasets without any prior technical background. That’s why they’re still so commonly used today.

In this article, you’ll learn how to use openpyxl to:

Manipulate Excel spreadsheets with confidence
Extract information from spreadsheets
Create simple or more complex spreadsheets, including adding styles, charts, and so on

This article is written for intermediate developers who have a pretty good knowledge of Python data structures, such as dicts and lists, but also feel comfortable around OOP and more intermediate level topics.

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

Before You Begin

If you ever get asked to extract some data from a database or log file into an Excel spreadsheet, or if you often have to convert an Excel spreadsheet into some more usable programmatic form, then this tutorial is perfect for you. Let’s jump into the openpyxl caravan!

Practical Use Cases

First things first, when would you need to use a package like openpyxl in a real-world scenario? You’ll see a few examples below, but really, there are hundreds of possible scenarios where this knowledge could come in handy.

Importing New Products Into a Database

You are responsible for tech in an online store company, and your boss doesn’t want to pay for a cool and expensive CMS system.

Every time they want to add new products to the online store, they come to you with an Excel spreadsheet with a few hundred rows and, for each of them, you have the product name, description, price, and so forth.

Now, to import the data, you’ll have to iterate over each spreadsheet row and add each product to the online store.

Exporting Database Data Into a Spreadsheet

Say you have a Database table where you record all your users’ information, including name, phone number, email address, and so forth.

Now, the Marketing team wants to contact all users to give them some discounted offer or promotion. However, they don’t have access to the Database, or they don’t know how to use SQL to extract that information easily.

What can you do to help? Well, you can make a quick script using openpyxl that iterates over every single User record and puts all the essential information into an Excel spreadsheet.

That’s gonna earn you an extra slice of cake at your company’s next birthday party!

Appending Information to an Existing Spreadsheet

You may also have to open a spreadsheet, read the information in it and, according to some business logic, append more data to it.

For example, using the online store scenario again, say you get an Excel spreadsheet with a list of users and you need to append to each row the total amount they’ve spent in your store.

This data is in the Database and, in order to do this, you have to read the spreadsheet, iterate through each row, fetch the total amount spent from the Database and then write back to the spreadsheet.

Not a problem for openpyxl!

Learning Some Basic Excel Terminology

Here’s a quick list of basic terms you’ll see when you’re working with Excel spreadsheets:

Term	Explanation
Spreadsheet or Workbook	A Spreadsheet is the main file you are creating or working with.
Worksheet or Sheet	A Sheet is used to split different kinds of content within the same spreadsheet. A Spreadsheet can have one or more Sheets.
Column	A Column is a vertical line, and it’s represented by an uppercase letter: A.
Row	A Row is a horizontal line, and it’s represented by a number: 1.
Cell	A Cell is a combination of Column and Row, represented by both an uppercase letter and a number: A1.

Getting Started With openpyxl

Now that you’re aware of the benefits of a tool like openpyxl, let’s get down to it and start by installing the package. For this tutorial, you should use Python 3.7 and openpyxl 2.6.2. To install the package, you can do the following:

$ pip install openpyxl

After you install the package, you should be able to create a super simple spreadsheet with the following code:

from openpyxl import Workbook

workbook = Workbook()
sheet = workbook.active

sheet["A1"] = "hello"
sheet["B1"] = "world!"

workbook.save(filename="hello_world.xlsx")

The code above should create a file called hello_world.xlsx in the folder you are using to run the code. If you open that file with Excel you should see something like this:

Woohoo, your first spreadsheet created!

Reading Excel Spreadsheets With openpyxl

Let’s start with the most essential thing one can do with a spreadsheet: read it.

You’ll go from a straightforward approach to reading a spreadsheet to more complex examples where you read the data and convert it into more useful Python structures.

Dataset for This Tutorial

Before you dive deep into some code examples, you should download this sample dataset and store it somewhere as sample.xlsx:

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

This is one of the datasets you’ll be using throughout this tutorial, and it’s a spreadsheet with a sample of real data from Amazon’s online product reviews. This dataset is only a tiny fraction of what Amazon provides, but for testing purposes, it’s more than enough.

A Simple Approach to Reading an Excel Spreadsheet

Finally, let’s start reading some spreadsheets! To begin with, open our sample spreadsheet:

>>>

>>> from openpyxl import load_workbook
>>> workbook = load_workbook(filename="sample.xlsx")
>>> workbook.sheetnames
['Sheet 1']

>>> sheet = workbook.active
>>> sheet
<Worksheet "Sheet 1">

>>> sheet.title
'Sheet 1'

In the code above, you first open the spreadsheet sample.xlsx using load_workbook(), and then you can use workbook.sheetnames to see all the sheets you have available to work with. After that, workbook.active selects the first available sheet and, in this case, you can see that it selects Sheet 1 automatically. Using these methods is the default way of opening a spreadsheet, and you’ll see it many times during this tutorial.

Now, after opening a spreadsheet, you can easily retrieve data from it like this:

>>>

>>> sheet["A1"]
<Cell 'Sheet 1'.A1>

>>> sheet["A1"].value
'marketplace'

>>> sheet["F10"].value
"G-Shock Men's Grey Sport Watch"

To return the actual value of a cell, you need to do .value. Otherwise, you’ll get the main Cell object. You can also use the method .cell() to retrieve a cell using index notation. Remember to add .value to get the actual value and not a Cell object:

>>>

>>> sheet.cell(row=10, column=6)
<Cell 'Sheet 1'.F10>

>>> sheet.cell(row=10, column=6).value
"G-Shock Men's Grey Sport Watch"

You can see that the results returned are the same, no matter which way you decide to go with. However, in this tutorial, you’ll be mostly using the first approach: ["A1"].

Note: Even though in Python you’re used to a zero-indexed notation, with spreadsheets you’ll always use a one-indexed notation where the first row or column always has index 1.

The above shows you the quickest way to open a spreadsheet. However, you can pass additional parameters to change the way a spreadsheet is loaded.

Additional Reading Options

There are a few arguments you can pass to load_workbook() that change the way a spreadsheet is loaded. The most important ones are the following two Booleans:

read_only loads a spreadsheet in read-only mode allowing you to open very large Excel files.
data_only ignores loading formulas and instead loads only the resulting values.

Importing Data From a Spreadsheet

Now that you’ve learned the basics about loading a spreadsheet, it’s about time you get to the fun part: the iteration and actual usage of the values within the spreadsheet.

This section is where you’ll learn all the different ways you can iterate through the data, but also how to convert that data into something usable and, more importantly, how to do it in a Pythonic way.

Iterating Through the Data

There are a few different ways you can iterate through the data depending on your needs.

You can slice the data with a combination of columns and rows:

>>>

>>> sheet["A1:C2"]
((<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>),
 (<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>))

You can get ranges of rows or columns:

>>>

>>> # Get all cells from column A
>>> sheet["A"]
(<Cell 'Sheet 1'.A1>,
 <Cell 'Sheet 1'.A2>,
 ...
 <Cell 'Sheet 1'.A99>,
 <Cell 'Sheet 1'.A100>)

>>> # Get all cells for a range of columns
>>> sheet["A:B"]
((<Cell 'Sheet 1'.A1>,
  <Cell 'Sheet 1'.A2>,
  ...
  <Cell 'Sheet 1'.A99>,
  <Cell 'Sheet 1'.A100>),
 (<Cell 'Sheet 1'.B1>,
  <Cell 'Sheet 1'.B2>,
  ...
  <Cell 'Sheet 1'.B99>,
  <Cell 'Sheet 1'.B100>))

>>> # Get all cells from row 5
>>> sheet[5]
(<Cell 'Sheet 1'.A5>,
 <Cell 'Sheet 1'.B5>,
 ...
 <Cell 'Sheet 1'.N5>,
 <Cell 'Sheet 1'.O5>)

>>> # Get all cells for a range of rows
>>> sheet[5:6]
((<Cell 'Sheet 1'.A5>,
  <Cell 'Sheet 1'.B5>,
  ...
  <Cell 'Sheet 1'.N5>,
  <Cell 'Sheet 1'.O5>),
 (<Cell 'Sheet 1'.A6>,
  <Cell 'Sheet 1'.B6>,
  ...
  <Cell 'Sheet 1'.N6>,
  <Cell 'Sheet 1'.O6>))

You’ll notice that all of the above examples return a tuple. If you want to refresh your memory on how to handle tuples in Python, check out the article on Lists and Tuples in Python.

There are also multiple ways of using normal Python generators to go through the data. The main methods you can use to achieve this are:

.iter_rows()
.iter_cols()

Both methods can receive the following arguments:

min_row
max_row
min_col
max_col

These arguments are used to set boundaries for the iteration:

>>>

>>> for row in sheet.iter_rows(min_row=1,
...                            max_row=2,
...                            min_col=1,
...                            max_col=3):
...     print(row)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>)
(<Cell 'Sheet 1'.A2>, <Cell 'Sheet 1'.B2>, <Cell 'Sheet 1'.C2>)


>>> for column in sheet.iter_cols(min_row=1,
...                               max_row=2,
...                               min_col=1,
...                               max_col=3):
...     print(column)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.A2>)
(<Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.B2>)
(<Cell 'Sheet 1'.C1>, <Cell 'Sheet 1'.C2>)

You’ll notice that in the first example, when iterating through the rows using .iter_rows(), you get one tuple element per row selected. While when using .iter_cols() and iterating through columns, you’ll get one tuple per column instead.

One additional argument you can pass to both methods is the Boolean values_only. When it’s set to True, the values of the cell are returned, instead of the Cell object:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=2,
...                              min_col=1,
...                              max_col=3,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id')
('US', 3653882, 'R3O9SGZBVQBV76')

If you want to iterate through the whole dataset, then you can also use the attributes .rows or .columns directly, which are shortcuts to using .iter_rows() and .iter_cols() without any arguments:

>>>

>>> for row in sheet.rows:
...     print(row)
(<Cell 'Sheet 1'.A1>, <Cell 'Sheet 1'.B1>, <Cell 'Sheet 1'.C1>
...
<Cell 'Sheet 1'.M100>, <Cell 'Sheet 1'.N100>, <Cell 'Sheet 1'.O100>)

These shortcuts are very useful when you’re iterating through the whole dataset.

Manipulate Data Using Python’s Default Data Structures

Now that you know the basics of iterating through the data in a workbook, let’s look at smart ways of converting that data into Python structures.

As you saw earlier, the result from all iterations comes in the form of tuples. However, since a tuple is nothing more than an immutable list, you can easily access its data and transform it into other structures.

For example, say you want to extract product information from the sample.xlsx spreadsheet and into a dictionary where each key is a product ID.

A straightforward way to do this is to iterate over all the rows, pick the columns you know are related to product information, and then store that in a dictionary. Let’s code this out!

First of all, have a look at the headers and see what information you care most about:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=1,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id', 'product_id', ...)

This code returns a list of all the column names you have in the spreadsheet. To start, grab the columns with names:

product_id
product_parent
product_title
product_category

Lucky for you, the columns you need are all next to each other so you can use the min_column and max_column to easily get the data you want:

>>>

>>> for value in sheet.iter_rows(min_row=2,
...                              min_col=4,
...                              max_col=7,
...                              values_only=True):
...     print(value)
('B00FALQ1ZC', 937001370, 'Invicta Women\'s 15150 "Angel" 18k Yellow...)
('B00D3RGO20', 484010722, "Kenneth Cole New York Women's KC4944...)
...

Nice! Now that you know how to get all the important product information you need, let’s put that data into a dictionary:

import json
from openpyxl import load_workbook

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

products = {}

# Using the values_only because you want to return the cells' values
for row in sheet.iter_rows(min_row=2,
                           min_col=4,
                           max_col=7,
                           values_only=True):
    product_id = row[0]
    product = {
        "parent": row[1],
        "title": row[2],
        "category": row[3]
    }
    products[product_id] = product

# Using json here to be able to format the output for displaying later
print(json.dumps(products))

The code above returns a JSON similar to this:

{
  "B00FALQ1ZC": {
    "parent": 937001370,
    "title": "Invicta Women's 15150 ...",
    "category": "Watches"
  },
  "B00D3RGO20": {
    "parent": 484010722,
    "title": "Kenneth Cole New York ...",
    "category": "Watches"
  }
}

Here you can see that the output is trimmed to 2 products only, but if you run the script as it is, then you should get 98 products.

Convert Data Into Python Classes

To finalize the reading section of this tutorial, let’s dive into Python classes and see how you could improve on the example above and better structure the data.

For this, you’ll be using the new Python Data Classes that are available from Python 3.7. If you’re using an older version of Python, then you can use the default Classes instead.

So, first things first, let’s look at the data you have and decide what you want to store and how you want to store it.

As you saw right at the start, this data comes from Amazon, and it’s a list of product reviews. You can check the list of all the columns and their meaning on Amazon.

There are two significant elements you can extract from the data available:

Products
Reviews

A Product has:

ID
Title
Parent
Category

The Review has a few more fields:

ID
Customer ID
Stars
Headline
Body
Date

You can ignore a few of the review fields to make things a bit simpler.

So, a straightforward implementation of these two classes could be written in a separate file classes.py:

import datetime
from dataclasses import dataclass

@dataclass
class Product:
    id: str
    parent: str
    title: str
    category: str

@dataclass
class Review:
    id: str
    customer_id: str
    stars: int
    headline: str
    body: str
    date: datetime.datetime

After defining your data classes, you need to convert the data from the spreadsheet into these new structures.

Before doing the conversion, it’s worth looking at our header again and creating a mapping between columns and the fields you need:

>>>

>>> for value in sheet.iter_rows(min_row=1,
...                              max_row=1,
...                              values_only=True):
...     print(value)
('marketplace', 'customer_id', 'review_id', 'product_id', ...)

>>> # Or an alternative
>>> for cell in sheet[1]:
...     print(cell.value)
marketplace
customer_id
review_id
product_id
product_parent
...

Let’s create a file mapping.py where you have a list of all the field names and their column location (zero-indexed) on the spreadsheet:

# Product fields
PRODUCT_ID = 3
PRODUCT_PARENT = 4
PRODUCT_TITLE = 5
PRODUCT_CATEGORY = 6

# Review fields
REVIEW_ID = 2
REVIEW_CUSTOMER = 1
REVIEW_STARS = 7
REVIEW_HEADLINE = 12
REVIEW_BODY = 13
REVIEW_DATE = 14

You don’t necessarily have to do the mapping above. It’s more for readability when parsing the row data, so you don’t end up with a lot of magic numbers lying around.

Finally, let’s look at the code needed to parse the spreadsheet data into a list of product and review objects:

from datetime import datetime
from openpyxl import load_workbook
from classes import Product, Review
from mapping import PRODUCT_ID, PRODUCT_PARENT, PRODUCT_TITLE, \
    PRODUCT_CATEGORY, REVIEW_DATE, REVIEW_ID, REVIEW_CUSTOMER, \
    REVIEW_STARS, REVIEW_HEADLINE, REVIEW_BODY

# Using the read_only method since you're not gonna be editing the spreadsheet
workbook = load_workbook(filename="sample.xlsx", read_only=True)
sheet = workbook.active

products = []
reviews = []

# Using the values_only because you just want to return the cell value
for row in sheet.iter_rows(min_row=2, values_only=True):
    product = Product(id=row[PRODUCT_ID],
                      parent=row[PRODUCT_PARENT],
                      title=row[PRODUCT_TITLE],
                      category=row[PRODUCT_CATEGORY])
    products.append(product)

    # You need to parse the date from the spreadsheet into a datetime format
    spread_date = row[REVIEW_DATE]
    parsed_date = datetime.strptime(spread_date, "%Y-%m-%d")

    review = Review(id=row[REVIEW_ID],
                    customer_id=row[REVIEW_CUSTOMER],
                    stars=row[REVIEW_STARS],
                    headline=row[REVIEW_HEADLINE],
                    body=row[REVIEW_BODY],
                    date=parsed_date)
    reviews.append(review)

print(products[0])
print(reviews[0])

After you run the code above, you should get some output like this:

Product(id='B00FALQ1ZC', parent=937001370, ...)
Review(id='R3O9SGZBVQBV76', customer_id=3653882, ...)

That’s it! Now you should have the data in a very simple and digestible class format, and you can start thinking of storing this in a Database or any other type of data storage you like.

Using this kind of OOP strategy to parse spreadsheets makes handling the data much simpler later on.

Appending New Data

Before you start creating very complex spreadsheets, have a quick look at an example of how to append data to an existing spreadsheet.

Go back to the first example spreadsheet you created (hello_world.xlsx) and try opening it and appending some data to it, like this:

from openpyxl import load_workbook

# Start by opening the spreadsheet and selecting the main sheet
workbook = load_workbook(filename="hello_world.xlsx")
sheet = workbook.active

# Write what you want into a specific cell
sheet["C1"] = "writing ;)"

# Save the spreadsheet
workbook.save(filename="hello_world_append.xlsx"

Et voilà, if you open the new hello_world_append.xlsx spreadsheet, you’ll see the following change:

Notice the additional writing ;) on cell C1.

Writing Excel Spreadsheets With openpyxl

There are a lot of different things you can write to a spreadsheet, from simple text or number values to complex formulas, charts, or even images.

Let’s start creating some spreadsheets!

Creating a Simple Spreadsheet

Previously, you saw a very quick example of how to write “Hello world!” into a spreadsheet, so you can start with that:

 1 from openpyxl import Workbook
 2 
 3 filename = "hello_world.xlsx"
 4 
 5 workbook = Workbook()
 6 sheet = workbook.active
 7 
 8 sheet["A1"] = "hello"
 9 sheet["B1"] = "world!"
10 
11 workbook.save(filename=filename)

The highlighted lines in the code above are the most important ones for writing. In the code, you can see that:

Line 5 shows you how to create a new empty workbook.
Lines 8 and 9 show you how to add data to specific cells.
Line 11 shows you how to save the spreadsheet when you’re done.

Even though these lines above can be straightforward, it’s still good to know them well for when things get a bit more complicated.

Note: You’ll be using the hello_world.xlsx spreadsheet for some of the upcoming examples, so keep it handy.

One thing you can do to help with coming code examples is add the following method to your Python file or console:

>>>

>>> def print_rows():
...     for row in sheet.iter_rows(values_only=True):
...         print(row)

It makes it easier to print all of your spreadsheet values by just calling print_rows().

Basic Spreadsheet Operations

Before you get into the more advanced topics, it’s good for you to know how to manage the most simple elements of a spreadsheet.

Adding and Updating Cell Values

You already learned how to add values to a spreadsheet like this:

>>>

>>> sheet["A1"] = "value"

There’s another way you can do this, by first selecting a cell and then changing its value:

>>>

>>> cell = sheet["A1"]
>>> cell
<Cell 'Sheet'.A1>

>>> cell.value
'hello'

>>> cell.value = "hey"
>>> cell.value
'hey'

The new value is only stored into the spreadsheet once you call workbook.save().

The openpyxl creates a cell when adding a value, if that cell didn’t exist before:

>>>

>>> # Before, our spreadsheet has only 1 row
>>> print_rows()
('hello', 'world!')

>>> # Try adding a value to row 10
>>> sheet["B10"] = "test"
>>> print_rows()
('hello', 'world!')
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, None)
(None, 'test')

As you can see, when trying to add a value to cell B10, you end up with a tuple with 10 rows, just so you can have that test value.

Managing Rows and Columns

One of the most common things you have to do when manipulating spreadsheets is adding or removing rows and columns. The openpyxl package allows you to do that in a very straightforward way by using the methods:

.insert_rows()
.delete_rows()
.insert_cols()
.delete_cols()

Every single one of those methods can receive two arguments:

idx
amount

Using our basic hello_world.xlsx example again, let’s see how these methods work:

>>>

>>> print_rows()
('hello', 'world!')

>>> # Insert a column before the existing column 1 ("A")
>>> sheet.insert_cols(idx=1)
>>> print_rows()
(None, 'hello', 'world!')

>>> # Insert 5 columns between column 2 ("B") and 3 ("C")
>>> sheet.insert_cols(idx=3, amount=5)
>>> print_rows()
(None, 'hello', None, None, None, None, None, 'world!')

>>> # Delete the created columns
>>> sheet.delete_cols(idx=3, amount=5)
>>> sheet.delete_cols(idx=1)
>>> print_rows()
('hello', 'world!')

>>> # Insert a new row in the beginning
>>> sheet.insert_rows(idx=1)
>>> print_rows()
(None, None)
('hello', 'world!')

>>> # Insert 3 new rows in the beginning
>>> sheet.insert_rows(idx=1, amount=3)
>>> print_rows()
(None, None)
(None, None)
(None, None)
(None, None)
('hello', 'world!')

>>> # Delete the first 4 rows
>>> sheet.delete_rows(idx=1, amount=4)
>>> print_rows()
('hello', 'world!')

The only thing you need to remember is that when inserting new data (rows or columns), the insertion happens before the idx parameter.

So, if you do insert_rows(1), it inserts a new row before the existing first row.

It’s the same for columns: when you call insert_cols(2), it inserts a new column right before the already existing second column (B).

However, when deleting rows or columns, .delete_... deletes data starting from the index passed as an argument.

For example, when doing delete_rows(2) it deletes row 2, and when doing delete_cols(3) it deletes the third column (C).

Managing Sheets

Sheet management is also one of those things you might need to know, even though it might be something that you don’t use that often.

If you look back at the code examples from this tutorial, you’ll notice the following recurring piece of code:

sheet = workbook.active

This is the way to select the default sheet from a spreadsheet. However, if you’re opening a spreadsheet with multiple sheets, then you can always select a specific one like this:

>>>

>>> # Let's say you have two sheets: "Products" and "Company Sales"
>>> workbook.sheetnames
['Products', 'Company Sales']

>>> # You can select a sheet using its title
>>> products_sheet = workbook["Products"]
>>> sales_sheet = workbook["Company Sales"]

You can also change a sheet title very easily:

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> products_sheet = workbook["Products"]
>>> products_sheet.title = "New Products"

>>> workbook.sheetnames
['New Products', 'Company Sales']

If you want to create or delete sheets, then you can also do that with .create_sheet() and .remove():

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> operations_sheet = workbook.create_sheet("Operations")
>>> workbook.sheetnames
['Products', 'Company Sales', 'Operations']

>>> # You can also define the position to create the sheet at
>>> hr_sheet = workbook.create_sheet("HR", 0)
>>> workbook.sheetnames
['HR', 'Products', 'Company Sales', 'Operations']

>>> # To remove them, just pass the sheet as an argument to the .remove()
>>> workbook.remove(operations_sheet)
>>> workbook.sheetnames
['HR', 'Products', 'Company Sales']

>>> workbook.remove(hr_sheet)
>>> workbook.sheetnames
['Products', 'Company Sales']

One other thing you can do is make duplicates of a sheet using copy_worksheet():

>>>

>>> workbook.sheetnames
['Products', 'Company Sales']

>>> products_sheet = workbook["Products"]
>>> workbook.copy_worksheet(products_sheet)
<Worksheet "Products Copy">

>>> workbook.sheetnames
['Products', 'Company Sales', 'Products Copy']

If you open your spreadsheet after saving the above code, you’ll notice that the sheet Products Copy is a duplicate of the sheet Products.

Freezing Rows and Columns

Something that you might want to do when working with big spreadsheets is to freeze a few rows or columns, so they remain visible when you scroll right or down.

Freezing data allows you to keep an eye on important rows or columns, regardless of where you scroll in the spreadsheet.

Again, openpyxl also has a way to accomplish this by using the worksheet freeze_panes attribute. For this example, go back to our sample.xlsx spreadsheet and try doing the following:

>>>

>>> workbook = load_workbook(filename="sample.xlsx")
>>> sheet = workbook.active
>>> sheet.freeze_panes = "C2"
>>> workbook.save("sample_frozen.xlsx")

If you open the sample_frozen.xlsx spreadsheet in your favorite spreadsheet editor, you’ll notice that row 1 and columns A and B are frozen and are always visible no matter where you navigate within the spreadsheet.

This feature is handy, for example, to keep headers within sight, so you always know what each column represents.

Here’s how it looks in the editor:

Notice how you’re at the end of the spreadsheet, and yet, you can see both row 1 and columns A and B.

Adding Filters

You can use openpyxl to add filters and sorts to your spreadsheet. However, when you open the spreadsheet, the data won’t be rearranged according to these sorts and filters.

At first, this might seem like a pretty useless feature, but when you’re programmatically creating a spreadsheet that is going to be sent and used by somebody else, it’s still nice to at least create the filters and allow people to use it afterward.

The code below is an example of how you would add some filters to our existing sample.xlsx spreadsheet:

>>>

>>> # Check the used spreadsheet space using the attribute "dimensions"
>>> sheet.dimensions
'A1:O100'

>>> sheet.auto_filter.ref = "A1:O100"
>>> workbook.save(filename="sample_with_filters.xlsx")

You should now see the filters created when opening the spreadsheet in your editor:

You don’t have to use sheet.dimensions if you know precisely which part of the spreadsheet you want to apply filters to.

Adding Formulas

Formulas (or formulae) are one of the most powerful features of spreadsheets.

They gives you the power to apply specific mathematical equations to a range of cells. Using formulas with openpyxl is as simple as editing the value of a cell.

You can see the list of formulas supported by openpyxl:

>>>

>>> from openpyxl.utils import FORMULAE
>>> FORMULAE
frozenset({'ABS',
           'ACCRINT',
           'ACCRINTM',
           'ACOS',
           'ACOSH',
           'AMORDEGRC',
           'AMORLINC',
           'AND',
           ...
           'YEARFRAC',
           'YIELD',
           'YIELDDISC',
           'YIELDMAT',
           'ZTEST'})

Let’s add some formulas to our sample.xlsx spreadsheet.

Starting with something easy, let’s check the average star rating for the 99 reviews within the spreadsheet:

>>>

>>> # Star rating is column "H"
>>> sheet["P2"] = "=AVERAGE(H2:H100)"
>>> workbook.save(filename="sample_formulas.xlsx")

If you open the spreadsheet now and go to cell P2, you should see that its value is: 4.18181818181818. Have a look in the editor:

You can use the same methodology to add any formulas to your spreadsheet. For example, let’s count the number of reviews that had helpful votes:

>>>

>>> # The helpful votes are counted on column "I"
>>> sheet["P3"] = '=COUNTIF(I2:I100, ">0")'
>>> workbook.save(filename="sample_formulas.xlsx")

You should get the number 21 on your P3 spreadsheet cell like so:

You’ll have to make sure that the strings within a formula are always in double quotes, so you either have to use single quotes around the formula like in the example above or you’ll have to escape the double quotes inside the formula: "=COUNTIF(I2:I100, \">0\")".

There are a ton of other formulas you can add to your spreadsheet using the same procedure you tried above. Give it a go yourself!

Adding Styles

Even though styling a spreadsheet might not be something you would do every day, it’s still good to know how to do it.

Using openpyxl, you can apply multiple styling options to your spreadsheet, including fonts, borders, colors, and so on. Have a look at the openpyxl documentation to learn more.

You can also choose to either apply a style directly to a cell or create a template and reuse it to apply styles to multiple cells.

Let’s start by having a look at simple cell styling, using our sample.xlsx again as the base spreadsheet:

>>>

>>> # Import necessary style classes
>>> from openpyxl.styles import Font, Color, Alignment, Border, Side, colors

>>> # Create a few styles
>>> bold_font = Font(bold=True)
>>> big_red_text = Font(color=colors.RED, size=20)
>>> center_aligned_text = Alignment(horizontal="center")
>>> double_border_side = Side(border_style="double")
>>> square_border = Border(top=double_border_side,
...                        right=double_border_side,
...                        bottom=double_border_side,
...                        left=double_border_side)

>>> # Style some cells!
>>> sheet["A2"].font = bold_font
>>> sheet["A3"].font = big_red_text
>>> sheet["A4"].alignment = center_aligned_text
>>> sheet["A5"].border = square_border
>>> workbook.save(filename="sample_styles.xlsx")

If you open your spreadsheet now, you should see quite a few different styles on the first 5 cells of column A:

There you go. You got:

A2 with the text in bold
A3 with the text in red and bigger font size
A4 with the text centered
A5 with a square border around the text

Note: For the colors, you can also use HEX codes instead by doing Font(color="C70E0F").

You can also combine styles by simply adding them to the cell at the same time:

>>>

>>> # Reusing the same styles from the example above
>>> sheet["A6"].alignment = center_aligned_text
>>> sheet["A6"].font = big_red_text
>>> sheet["A6"].border = square_border
>>> workbook.save(filename="sample_styles.xlsx")

Have a look at cell A6 here:

When you want to apply multiple styles to one or several cells, you can use a NamedStyle class instead, which is like a style template that you can use over and over again. Have a look at the example below:

>>>

>>> from openpyxl.styles import NamedStyle

>>> # Let's create a style template for the header row
>>> header = NamedStyle(name="header")
>>> header.font = Font(bold=True)
>>> header.border = Border(bottom=Side(border_style="thin"))
>>> header.alignment = Alignment(horizontal="center", vertical="center")

>>> # Now let's apply this to all first row (header) cells
>>> header_row = sheet[1]
>>> for cell in header_row:
...     cell.style = header

>>> workbook.save(filename="sample_styles.xlsx")

If you open the spreadsheet now, you should see that its first row is bold, the text is aligned to the center, and there’s a small bottom border! Have a look below:

As you saw above, there are many options when it comes to styling, and it depends on the use case, so feel free to check openpyxl documentation and see what other things you can do.

Conditional Formatting

This feature is one of my personal favorites when it comes to adding styles to a spreadsheet.

It’s a much more powerful approach to styling because it dynamically applies styles according to how the data in the spreadsheet changes.

In a nutshell, conditional formatting allows you to specify a list of styles to apply to a cell (or cell range) according to specific conditions.

For example, a widespread use case is to have a balance sheet where all the negative totals are in red, and the positive ones are in green. This formatting makes it much more efficient to spot good vs bad periods.

Without further ado, let’s pick our favorite spreadsheet—sample.xlsx—and add some conditional formatting.

You can start by adding a simple one that adds a red background to all reviews with less than 3 stars:

>>>

>>> from openpyxl.styles import PatternFill, colors
>>> from openpyxl.styles.differential import DifferentialStyle
>>> from openpyxl.formatting.rule import Rule

>>> red_background = PatternFill(bgColor=colors.RED)
>>> diff_style = DifferentialStyle(fill=red_background)
>>> rule = Rule(type="expression", dxf=diff_style)
>>> rule.formula = ["$H1<3"]
>>> sheet.conditional_formatting.add("A1:O100", rule)
>>> workbook.save("sample_conditional_formatting.xlsx")

Now you’ll see all the reviews with a star rating below 3 marked with a red background:

Code-wise, the only things that are new here are the objects DifferentialStyle and Rule:

DifferentialStyle is quite similar to NamedStyle, which you already saw above, and it’s used to aggregate multiple styles such as fonts, borders, alignment, and so forth.
Rule is responsible for selecting the cells and applying the styles if the cells match the rule’s logic.

Using a Rule object, you can create numerous conditional formatting scenarios.

However, for simplicity sake, the openpyxl package offers 3 built-in formats that make it easier to create a few common conditional formatting patterns. These built-ins are:

ColorScale
IconSet
DataBar

The ColorScale gives you the ability to create color gradients:

>>>

>>> from openpyxl.formatting.rule import ColorScaleRule
>>> color_scale_rule = ColorScaleRule(start_type="min",
...                                   start_color=colors.RED,
...                                   end_type="max",
...                                   end_color=colors.GREEN)

>>> # Again, let's add this gradient to the star ratings, column "H"
>>> sheet.conditional_formatting.add("H2:H100", color_scale_rule)
>>> workbook.save(filename="sample_conditional_formatting_color_scale.xlsx")

Now you should see a color gradient on column H, from red to green, according to the star rating:

You can also add a third color and make two gradients instead:

>>>

>>> from openpyxl.formatting.rule import ColorScaleRule
>>> color_scale_rule = ColorScaleRule(start_type="num",
...                                   start_value=1,
...                                   start_color=colors.RED,
...                                   mid_type="num",
...                                   mid_value=3,
...                                   mid_color=colors.YELLOW,
...                                   end_type="num",
...                                   end_value=5,
...                                   end_color=colors.GREEN)

>>> # Again, let's add this gradient to the star ratings, column "H"
>>> sheet.conditional_formatting.add("H2:H100", color_scale_rule)
>>> workbook.save(filename="sample_conditional_formatting_color_scale_3.xlsx")

This time, you’ll notice that star ratings between 1 and 3 have a gradient from red to yellow, and star ratings between 3 and 5 have a gradient from yellow to green:

The IconSet allows you to add an icon to the cell according to its value:

>>>

>>> from openpyxl.formatting.rule import IconSetRule

>>> icon_set_rule = IconSetRule("5Arrows", "num", [1, 2, 3, 4, 5])
>>> sheet.conditional_formatting.add("H2:H100", icon_set_rule)
>>> workbook.save("sample_conditional_formatting_icon_set.xlsx")

You’ll see a colored arrow next to the star rating. This arrow is red and points down when the value of the cell is 1 and, as the rating gets better, the arrow starts pointing up and becomes green:

The openpyxl package has a full list of other icons you can use, besides the arrow.

Finally, the DataBar allows you to create progress bars:

>>>

>>> from openpyxl.formatting.rule import DataBarRule

>>> data_bar_rule = DataBarRule(start_type="num",
...                             start_value=1,
...                             end_type="num",
...                             end_value="5",
...                             color=colors.GREEN)
>>> sheet.conditional_formatting.add("H2:H100", data_bar_rule)
>>> workbook.save("sample_conditional_formatting_data_bar.xlsx")

You’ll now see a green progress bar that gets fuller the closer the star rating is to the number 5:

As you can see, there are a lot of cool things you can do with conditional formatting.

Here, you saw only a few examples of what you can achieve with it, but check the openpyxl documentation to see a bunch of other options.

Adding Images

Even though images are not something that you’ll often see in a spreadsheet, it’s quite cool to be able to add them. Maybe you can use it for branding purposes or to make spreadsheets more personal.

To be able to load images to a spreadsheet using openpyxl, you’ll have to install Pillow:

$ pip install Pillow

Apart from that, you’ll also need an image. For this example, you can grab the Real Python logo below and convert it from .webp to .png using an online converter such as cloudconvert.com, save the final file as logo.png, and copy it to the root folder where you’re running your examples:

Afterward, this is the code you need to import that image into the hello_word.xlsx spreadsheet:

from openpyxl import load_workbook
from openpyxl.drawing.image import Image

# Let's use the hello_world spreadsheet since it has less data
workbook = load_workbook(filename="hello_world.xlsx")
sheet = workbook.active

logo = Image("logo.png")

# A bit of resizing to not fill the whole spreadsheet with the logo
logo.height = 150
logo.width = 150

sheet.add_image(logo, "A3")
workbook.save(filename="hello_world_logo.xlsx")

You have an image on your spreadsheet! Here it is:

The image’s left top corner is on the cell you chose, in this case, A3.

Adding Pretty Charts

Another powerful thing you can do with spreadsheets is create an incredible variety of charts.

Charts are a great way to visualize and understand loads of data quickly. There are a lot of different chart types: bar chart, pie chart, line chart, and so on. openpyxl has support for a lot of them.

Here, you’ll see only a couple of examples of charts because the theory behind it is the same for every single chart type:

Note: A few of the chart types that openpyxl currently doesn’t have support for are Funnel, Gantt, Pareto, Treemap, Waterfall, Map, and Sunburst.

For any chart you want to build, you’ll need to define the chart type: BarChart, LineChart, and so forth, plus the data to be used for the chart, which is called Reference.

Before you can build your chart, you need to define what data you want to see represented in it. Sometimes, you can use the dataset as is, but other times you need to massage the data a bit to get additional information.

Let’s start by building a new workbook with some sample data:

 1 from openpyxl import Workbook
 2 from openpyxl.chart import BarChart, Reference
 3 
 4 workbook = Workbook()
 5 sheet = workbook.active
 6 
 7 # Let's create some sample sales data
 8 rows = [
 9     ["Product", "Online", "Store"],
10     [1, 30, 45],
11     [2, 40, 30],
12     [3, 40, 25],
13     [4, 50, 30],
14     [5, 30, 25],
15     [6, 25, 35],
16     [7, 20, 40],
17 ]
18 
19 for row in rows:
20     sheet.append(row)

Now you’re going to start by creating a bar chart that displays the total number of sales per product:

22 chart = BarChart()
23 data = Reference(worksheet=sheet,
24                  min_row=1,
25                  max_row=8,
26                  min_col=2,
27                  max_col=3)
28 
29 chart.add_data(data, titles_from_data=True)
30 sheet.add_chart(chart, "E2")
31 
32 workbook.save("chart.xlsx")

There you have it. Below, you can see a very straightforward bar chart showing the difference between online product sales online and in-store product sales:

Like with images, the top left corner of the chart is on the cell you added the chart to. In your case, it was on cell E2.

Note: Depending on whether you’re using Microsoft Excel or an open-source alternative (LibreOffice or OpenOffice), the chart might look slightly different.

Try creating a line chart instead, changing the data a bit:

 1 import random
 2 from openpyxl import Workbook
 3 from openpyxl.chart import LineChart, Reference
 4 
 5 workbook = Workbook()
 6 sheet = workbook.active
 7 
 8 # Let's create some sample sales data
 9 rows = [
10     ["", "January", "February", "March", "April",
11     "May", "June", "July", "August", "September",
12      "October", "November", "December"],
13     [1, ],
14     [2, ],
15     [3, ],
16 ]
17 
18 for row in rows:
19     sheet.append(row)
20 
21 for row in sheet.iter_rows(min_row=2,
22                            max_row=4,
23                            min_col=2,
24                            max_col=13):
25     for cell in row:
26         cell.value = random.randrange(5, 100)

With the above code, you’ll be able to generate some random data regarding the sales of 3 different products across a whole year.

Once that’s done, you can very easily create a line chart with the following code:

28 chart = LineChart()
29 data = Reference(worksheet=sheet,
30                  min_row=2,
31                  max_row=4,
32                  min_col=1,
33                  max_col=13)
34 
35 chart.add_data(data, from_rows=True, titles_from_data=True)
36 sheet.add_chart(chart, "C6")
37 
38 workbook.save("line_chart.xlsx")

Here’s the outcome of the above piece of code:

One thing to keep in mind here is the fact that you’re using from_rows=True when adding the data. This argument makes the chart plot row by row instead of column by column.

In your sample data, you see that each product has a row with 12 values (1 column per month). That’s why you use from_rows. If you don’t pass that argument, by default, the chart tries to plot by column, and you’ll get a month-by-month comparison of sales.

Another difference that has to do with the above argument change is the fact that our Reference now starts from the first column, min_col=1, instead of the second one. This change is needed because the chart now expects the first column to have the titles.

There are a couple of other things you can also change regarding the style of the chart. For example, you can add specific categories to the chart:

cats = Reference(worksheet=sheet,
                 min_row=1,
                 max_row=1,
                 min_col=2,
                 max_col=13)
chart.set_categories(cats)

Add this piece of code before saving the workbook, and you should see the month names appearing instead of numbers:

Code-wise, this is a minimal change. But in terms of the readability of the spreadsheet, this makes it much easier for someone to open the spreadsheet and understand the chart straight away.

Another thing you can do to improve the chart readability is to add an axis. You can do it using the attributes x_axis and y_axis:

chart.x_axis.title = "Months"
chart.y_axis.title = "Sales (per unit)"

This will generate a spreadsheet like the below one:

As you can see, small changes like the above make reading your chart a much easier and quicker task.

There is also a way to style your chart by using Excel’s default ChartStyle property. In this case, you have to choose a number between 1 and 48. Depending on your choice, the colors of your chart change as well:

# You can play with this by choosing any number between 1 and 48
chart.style = 24

With the style selected above, all lines have some shade of orange:

There is no clear documentation on what each style number looks like, but this spreadsheet has a few examples of the styles available.

Here’s the full code used to generate the line chart with categories, axis titles, and style:

import random
from openpyxl import Workbook
from openpyxl.chart import LineChart, Reference

workbook = Workbook()
sheet = workbook.active

# Let's create some sample sales data
rows = [
    ["", "January", "February", "March", "April",
    "May", "June", "July", "August", "September",
     "October", "November", "December"],
    [1, ],
    [2, ],
    [3, ],
]

for row in rows:
    sheet.append(row)

for row in sheet.iter_rows(min_row=2,
                           max_row=4,
                           min_col=2,
                           max_col=13):
    for cell in row:
        cell.value = random.randrange(5, 100)

# Create a LineChart and add the main data
chart = LineChart()
data = Reference(worksheet=sheet,
                           min_row=2,
                           max_row=4,
                           min_col=1,
                           max_col=13)
chart.add_data(data, titles_from_data=True, from_rows=True)

# Add categories to the chart
cats = Reference(worksheet=sheet,
                 min_row=1,
                 max_row=1,
                 min_col=2,
                 max_col=13)
chart.set_categories(cats)

# Rename the X and Y Axis
chart.x_axis.title = "Months"
chart.y_axis.title = "Sales (per unit)"

# Apply a specific Style
chart.style = 24

# Save!
sheet.add_chart(chart, "C6")
workbook.save("line_chart.xlsx")

There are a lot more chart types and customization you can apply, so be sure to check out the package documentation on this if you need some specific formatting.

Convert Python Classes to Excel Spreadsheet

You already saw how to convert an Excel spreadsheet’s data into Python classes, but now let’s do the opposite.

Let’s imagine you have a database and are using some Object-Relational Mapping (ORM) to map DB objects into Python classes. Now, you want to export those same objects into a spreadsheet.

Let’s assume the following data classes to represent the data coming from your database regarding product sales:

from dataclasses import dataclass
from typing import List

@dataclass
class Sale:
    id: str
    quantity: int

@dataclass
class Product:
    id: str
    name: str
    sales: List[Sale]

Now, let’s generate some random data, assuming the above classes are stored in a db_classes.py file:

 1 import random
 2 
 3 # Ignore these for now. You'll use them in a sec ;)
 4 from openpyxl import Workbook
 5 from openpyxl.chart import LineChart, Reference
 6 
 7 from db_classes import Product, Sale
 8 
 9 products = []
10 
11 # Let's create 5 products
12 for idx in range(1, 6):
13     sales = []
14 
15     # Create 5 months of sales
16     for _ in range(5):
17         sale = Sale(quantity=random.randrange(5, 100))
18         sales.append(sale)
19 
20     product = Product(id=str(idx),
21                       name="Product %s" % idx,
22                       sales=sales)
23     products.append(product)

By running this piece of code, you should get 5 products with 5 months of sales with a random quantity of sales for each month.

Now, to convert this into a spreadsheet, you need to iterate over the data and append it to the spreadsheet:

25 workbook = Workbook()
26 sheet = workbook.active
27 
28 # Append column names first
29 sheet.append(["Product ID", "Product Name", "Month 1",
30               "Month 2", "Month 3", "Month 4", "Month 5"])
31 
32 # Append the data
33 for product in products:
34     data = [product.id, product.name]
35     for sale in product.sales:
36         data.append(sale.quantity)
37     sheet.append(data)

That’s it. That should allow you to create a spreadsheet with some data coming from your database.

However, why not use some of that cool knowledge you gained recently to add a chart as well to display that data more visually?

All right, then you could probably do something like this:

38 chart = LineChart()
39 data = Reference(worksheet=sheet,
40                  min_row=2,
41                  max_row=6,
42                  min_col=2,
43                  max_col=7)
44 
45 chart.add_data(data, titles_from_data=True, from_rows=True)
46 sheet.add_chart(chart, "B8")
47 
48 cats = Reference(worksheet=sheet,
49                  min_row=1,
50                  max_row=1,
51                  min_col=3,
52                  max_col=7)
53 chart.set_categories(cats)
54 
55 chart.x_axis.title = "Months"
56 chart.y_axis.title = "Sales (per unit)"
57 
58 workbook.save(filename="oop_sample.xlsx")

Now we’re talking! Here’s a spreadsheet generated from database objects and with a chart and everything:

That’s a great way for you to wrap up your new knowledge of charts!

Bonus: Working With Pandas

Even though you can use Pandas to handle Excel files, there are few things that you either can’t accomplish with Pandas or that you’d be better off just using openpyxl directly.

For example, some of the advantages of using openpyxl are the ability to easily customize your spreadsheet with styles, conditional formatting, and such.

But guess what, you don’t have to worry about picking. In fact, openpyxl has support for both converting data from a Pandas DataFrame into a workbook or the opposite, converting an openpyxl workbook into a Pandas DataFrame.

Note: If you’re new to Pandas, check our course on Pandas DataFrames beforehand.

First things first, remember to install the pandas package:

$ pip install pandas

Then, let’s create a sample DataFrame:

 1 import pandas as pd
 2 
 3 data = {
 4     "Product Name": ["Product 1", "Product 2"],
 5     "Sales Month 1": [10, 20],
 6     "Sales Month 2": [5, 35],
 7 }
 8 df = pd.DataFrame(data)

Now that you have some data, you can use .dataframe_to_rows() to convert it from a DataFrame into a worksheet:

10 from openpyxl import Workbook
11 from openpyxl.utils.dataframe import dataframe_to_rows
12 
13 workbook = Workbook()
14 sheet = workbook.active
15 
16 for row in dataframe_to_rows(df, index=False, header=True):
17     sheet.append(row)
18 
19 workbook.save("pandas.xlsx")

You should see a spreadsheet that looks like this:

If you want to add the DataFrame’s index, you can change index=True, and it adds each row’s index into your spreadsheet.

On the other hand, if you want to convert a spreadsheet into a DataFrame, you can also do it in a very straightforward way like so:

import pandas as pd
from openpyxl import load_workbook

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

values = sheet.values
df = pd.DataFrame(values)

Alternatively, if you want to add the correct headers and use the review ID as the index, for example, then you can also do it like this instead:

import pandas as pd
from openpyxl import load_workbook
from mapping import REVIEW_ID

workbook = load_workbook(filename="sample.xlsx")
sheet = workbook.active

data = sheet.values

# Set the first row as the columns for the DataFrame
cols = next(data)
data = list(data)

# Set the field "review_id" as the indexes for each row
idx = [row[REVIEW_ID] for row in data]

df = pd.DataFrame(data, index=idx, columns=cols)

Using indexes and columns allows you to access data from your DataFrame easily:

>>>

>>> df.columns
Index(['marketplace', 'customer_id', 'review_id', 'product_id',
       'product_parent', 'product_title', 'product_category', 'star_rating',
       'helpful_votes', 'total_votes', 'vine', 'verified_purchase',
       'review_headline', 'review_body', 'review_date'],
      dtype='object')

>>> # Get first 10 reviews' star rating
>>> df["star_rating"][:10]
R3O9SGZBVQBV76    5
RKH8BNC3L5DLF     5
R2HLE8WKZSU3NL    2
R31U3UH5AZ42LL    5
R2SV659OUJ945Y    4
RA51CP8TR5A2L     5
RB2Q7DLDN6TH6     5
R2RHFJV0UYBK3Y    1
R2Z6JOQ94LFHEP    5
RX27XIIWY5JPB     4
Name: star_rating, dtype: int64

>>> # Grab review with id "R2EQL1V1L6E0C9", using the index
>>> df.loc["R2EQL1V1L6E0C9"]
marketplace               US
customer_id         15305006
review_id     R2EQL1V1L6E0C9
product_id        B004LURNO6
product_parent     892860326
review_headline   Five Stars
review_body          Love it
review_date       2015-08-31
Name: R2EQL1V1L6E0C9, dtype: object

There you go, whether you want to use openpyxl to prettify your Pandas dataset or use Pandas to do some hardcore algebra, you now know how to switch between both packages.

Conclusion

Phew, after that long read, you now know how to work with spreadsheets in Python! You can rely on openpyxl, your trustworthy companion, to:

Extract valuable information from spreadsheets in a Pythonic manner
Create your own spreadsheets, no matter the complexity level
Add cool features such as conditional formatting or charts to your spreadsheets

There are a few other things you can do with openpyxl that might not have been covered in this tutorial, but you can always check the package’s official documentation website to learn more about it. You can even venture into checking its source code and improving the package further.

Feel free to leave any comments below if you have any questions, or if there’s any section you’d love to hear more about.

Download Dataset: Click here to download the dataset for the openpyxl exercise you'll be following in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Python vs C++: Selecting the Right Tool for the Job

Comparing Languages: Python vs C++

Compilation vs Virtual Machine

Syntax Differences

Whitespace

Boolean Expressions

Variables and Pointers

Comprehensions

Python’s std::algorithms

Static vs Dynamic Typing

Static Typing

Duck Typing

Templates

Type Checking

Object-Oriented Programming

Similarities

Differences

Operator Overloads vs Dunder Methods

Memory Management

Reference Counting Collector

Generational Garbage Collector

When You Don’t Want Garbage Collection

Threading, Multiprocessing, and Async IO

Threading

Multiprocessing

Async IO

Miscellaneous Issues

Summary: Python vs C++

Conclusion

Absolute vs Relative Imports in Python

MATLAB vs Python: Why and How to Make the Switch

MATLAB vs Python: Comparing Features and Philosophy

Setting Up Your Environment for Python

Getting Python via Anaconda

Getting an Integrated Development Environment

Spyder

Changing the Default Window Layout in Spyder

Running Statements in the Console in Spyder

Running Code in Files in Spyder

Summarizing Your Experience in Spyder

JupyterLab

Learning About Python’s Mathematical Libraries

NumPy (Numerical Python)

SciPy (Scientific Python)

Matplotlib (MATLAB-like Plotting Library)

Other Important Python Libraries

Syntax Differences Between MATLAB® and Python

You Will Probably See This Syntax

Comments Start With # in Python

Whitespace at the Beginning of a Line Is Significant in Python

Conditional Statements Use elif in Python

Calling Functions and Indexing Sequences Use Different Brackets in Python

The First Index in a Sequence Is 0 in Python

The Last Element of a Sequence Has Index -1 in Python

Exponentiation Is Done With ** in Python

The Length of a Sequence Is Found With len() in Python

Console Output Is Shown With print() in Python

You Will Probably See These, but You Can Learn Them When You Need To

Function Definitions Start With def and return Values in Python

Functions Accept Positional and Keyword Arguments in Python

There Are No switch/case Blocks in Python

Namespaces Are One Honking Great Idea in Python

The Most Recent Unassigned Result Is Available as _ in Python

Anonymous Functions Are Created With the lambda Keyword in Python

You Will Only Need These in Specialized Situations

Class Definitions Start With class in Python

There Are No Private Properties or Methods in Python

A Class Refers to Itself as self in Python

There Is One String Type in Python

Libraries Are Not Automatically Reloaded in Python

An Overview of Basic Array Operations

Basic Mathematical Operators Work Element-Wise in NumPy

One-Dimensional Arrays Are Vectors in NumPy

Creating Arrays Is Very Flexible in NumPy

The Colon Operator Is Very Powerful in NumPy

Array Slices Are Views of Arrays in NumPy

Tips and Tricks to Make Your Code Pythonic

You Should Not Use Semicolons to End Lines in Python

You Should Not Import * From a Module in Python

Python’s `std::algorithms`

Comments Start With `#` in Python

Conditional Statements Use `elif` in Python

The Last Element of a Sequence Has Index `-1` in Python

Exponentiation Is Done With `**` in Python

The Length of a Sequence Is Found With `len()` in Python

Console Output Is Shown With `print()` in Python

Function Definitions Start With `def` and `return` Values in Python

There Are No `switch`/`case` Blocks in Python

The Most Recent Unassigned Result Is Available as `_` in Python

Anonymous Functions Are Created With the `lambda` Keyword in Python

Class Definitions Start With `class` in Python

A Class Refers to Itself as `self` in Python

You Should Not Import `*` From a Module in Python

Unpacking With the Asterisk Operators: `*` & `**`