Real Python

Exploring HTTPS With Python

2020-01-08T14:00:00+00:00

Have you ever wondered why it’s okay for you to send your credit card information over the Internet? You may have noticed the https:// on URLs in your browser, but what is it, and how does it keep your information safe? Or perhaps you want to create a Python HTTPS application, but you’re not exactly sure what that means. How can you be sure that your web application is safe?

It may surprise you to know that you don’t have to be an expert in security to answer these questions! In this tutorial, you’ll get a working knowledge of the various factors that combine to keep communications over the Internet safe. You’ll see concrete examples of how a Python HTTPS application keeps information secure.

In this tutorial, you’ll learn how to:

Monitor and analyze network traffic
Apply cryptography to keep data safe
Describe the core concepts of Public Key Infrastructure (PKI)
Create your own Certificate Authority
Build a Python HTTPS application
Identify common Python HTTPS warnings and errors

Free Bonus: Click here to get access to a free Flask + Python video tutorial that shows you how to build Flask web app, step-by-step.

What Is HTTP?

Before you dive into HTTPS and its use in Python, it’s important to understand its parent, HTTP. This acronym stands for HyperText Transfer Protocol, which underpins most of the communications that go on when you’re surfing your favorite websites. More specifically, HTTP is how a user agent, like your web browser, communicates with a web server, like realpython.com. Here’s a simplified diagram of HTTP communications:

This diagram shows a simplified version of how your computer communicates with a server. Here’s the breakdown of each step:

You tell your browser to go to http://someserver.com/link.
Your device and the server set up a TCP connection.
Your browser sends an HTTP request to the server.
The server receives the HTTP request and parses it.
The server responds with an HTTP response.
Your computer receives, parses, and displays the response.

This breakdown captures the basics of HTTP. You make a request to a server, and the server returns a response. While HTTP doesn’t require TCP, it does require a reliable lower-level protocol. In practice, this is almost always TCP over IP (though Google is trying to create a replacement). If you need a refresher, then check out Socket Programming in Python (Guide).

As protocols go, HTTP is one of the simpler ones. It was designed to send content over the Internet, like HTML, videos, images, and so on. This is done with an HTTP request and response. HTTP requests contain the following elements:

The method describes what action the client wants to perform. The method for static content is typically GET, though there are others available, like POST, HEAD, and DELETE.
The path indicates to the server what web page you would like to request. For example, the path of this page is /python-https.
The version is one of several HTTP versions, like 1.0, 1.1, or 2.0. The most common is probably 1.1.
The headers help describe additional information for the server.
The body provides the server with information from the client. Though this field is not required, it’s typical for some methods to have a body, like a POST.

These are the tools your browser uses to communicate with a server. The server responds with an HTTP response. The HTTP response contains the following elements:

The version identifies the HTTP version, which will typically be the same as the request’s version.
The status code indicates whether a request was completed successfully. There are quite a few status codes.
The status message provides a human-readable message that helps describe the status code.
The headers allow the server to respond with additional metadata about the request. These are the same concept as request headers.
The body carries the content. Technically, this is optional, but typically it contains a useful resource.

These are the building blocks for HTTP. If you’re interested in learning more about HTTP, then you can check out an overview page to learn about the protocol in more depth.

What Is HTTPS?

Now that you understand a bit more about HTTP, what is HTTPS? The good news is, you already know this! HTTPS stands for HyperText Transfer Protocol Secure. Fundamentally, HTTPS is the same protocol as HTTP but with the added implication that the communications are secure.

HTTPS doesn’t rewrite any of the HTTP fundamentals on which it’s built. Instead, HTTPS consists of regular HTTP sent over an encrypted connection. Typically, this encrypted connection is provided by either TLS or SSL, which are cryptographic protocols that encrypt the information before it’s sent over a network.

Note: TLS and SSL are extremely similar protocols, though SSL is on its way out, with TLS to take its place. The differences in these protocols are outside the scope of this tutorial. It’s enough to know that TLS is the newer, better version of SSL.

So, why create this separation? Why not just introduce the complexity into the HTTP protocol itself? The answer is portability. Securing communications is an important and hard problem, but HTTP is only one of many protocols that require security. There are countless others across a wide variety of applications:

E-mail
Instant Messaging
VoIP (Voice over IP)

There are others, as well! If each of these protocols had to create their own security mechanism, then the world would be much less secure and much more confusing. TLS, which is often used by the above protocols, provides a common method to secure communications.

Note: This separation of protocols is a common theme in networking, so much so that it has a name. The OSI Model represents communications from physical medium all the way up to the HTML rendered on this page!

Almost all of the information you’ll learn in this tutorial will be applicable to more than just Python HTTPS applications. You’ll be learning the basics of secure communications along with how it applies specifically to HTTPS.

Why Is HTTPS Important?

Secure communications are critical in providing a safe online environment. As more of the world moves online, including banks and healthcare sites, it’s becoming more and more important for developers to create Python HTTPS applications. Again, HTTPS is just HTTP over TLS or SSL. TLS is designed to provide privacy from eavesdroppers. It can also provide authentication of both the client and the server.

In this section, you’ll explore these concepts in depth by doing the following:

Creating a Python HTTPS server
Communicating with your Python HTTPS server
Capturing these communications
Analyzing those messages

Let’s get started!

Creating an Example Application

Suppose you’re the leader of a cool Python club called the Secret Squirrels. The Squirrels, being secret, require a secret message to attend their meetings. As the leader, you choose the secret message, which changes for each meeting. Sometimes, though, it’s hard for you to meet with all the members before the meeting to tell them the secret message! You decide to set up a secret server where members can just see the secret message for themselves.

Note: The example code used in this tutorial is not designed for production. It’s designed to help you learn the basics of HTTP and TLS. Please do not use this code for production. Many of the examples below have terrible security practices. In this tutorial, you’ll learn about TLS, and one way it can help you be more secure.

You’ve followed some tutorials on Real Python and decide to use some dependencies you know:

Flask to build a web application
uWSGI as a production server
requests to exercise your server

To install all of these dependencies, you can use pip:

$ pip install flask uwsgi requests

With your dependencies installed, you begin to write your application. In a file called server.py, you create a Flask application:

# server.py
from flask import Flask

SECRET_MESSAGE = "fluffy tail"
app = Flask(__name__)

@app.route("/")
def get_secret_message():
    return SECRET_MESSAGE

This Flask application will display the secret message whenever someone visits the / path of your server. With that out of the way, you deploy your application on your secret server and run it:

$ uwsgi --http-socket 127.0.0.1:5683 --mount /=server:app

This command starts up a server using the Flask application above. You start it on a weird port because you don’t want people to be able to find it, and pat yourself on the back for being so sneaky! You can confirm that it’s working by visiting http://localhost:5683 in your browser.

Since everyone in the Secret Squirrels knows Python, you decide to help them out. You write a script called client.py that will help them get the secret message:

# client.py
import os
import requests

def get_secret_message():
    url = os.environ["SECRET_URL"]
    response = requests.get(url)
    print(f"The secret message is: {response.text}")

if __name__ == "__main__":
    get_secret_message()

This code will print out the secret message as long as they have the SECRET_URL environment variable set. In this case, the SECRET_URL is 127.0.0.1:5683. So, your plan is to give each club member the secret URL and tell them to keep it secret and safe.

While this might seem okay, rest assured it’s not! In fact, even if you were to put a username and password on this site, it still wouldn’t be safe. But even if your team somehow did manage to keep the URL safe, your secret message still wouldn’t be secure. To demonstrate why you’ll need to know a little bit about monitoring network traffic. To do this, you’ll be using a tool called Wireshark.

Setting Up Wireshark

Wireshark is a widely used tool for network and protocol analysis. What this means is that it can help you see what’s happening over network connections. Installing and setting up Wireshark is optional for this tutorial, but feel free if you’d like to follow along. The download page has several installers available:

macOS 10.12 and later
Windows installer 64-bit
Windows installer 32-bit

If you’re using Windows or Mac, then you should be able to download the appropriate installer and follow the prompts. In the end, you should have a running Wireshark.

If you’re on a Debian-based Linux environment, then the installation is a bit harder, but still possible. You can install Wireshark with the following commands:

$ sudo add-apt-repository ppa:wireshark-dev/stable
$ sudo apt-get update
$ sudo apt-get install wireshark
$ sudo wireshark

You should be met with a screen that looks something like this:

With Wireshark running, it’s time to analyze some traffic!

Seeing That Your Data Is Not Safe

The way your current client and server are running is not secure. HTTP will send everything in the clear for anyone to see. What this means is that even if someone doesn’t have your SECRET_URL, they can still see everything you do as long as they can monitor traffic on any device between you and the server.

This should be relatively scary for you. After all, you don’t want other people showing up for your Secret Squirrel meetings! You can prove that this is happening. First, start up your server if you don’t still have it running:

$ uwsgi --http-socket 127.0.0.1:5683 --mount /=server:app

This will start up your Flask application on port 5683. Next, you’ll start a packet capture in Wireshark. This packet capture will help you see all the traffic going to and from the server. Begin by selecting the Loopback:lo interface on Wireshark:

You can see that the Loopback:lo portion is highlighted. This instructs Wireshark to monitor this port for traffic. You can do better and specify which port and protocol you’d like to capture. You can type port 5683 in the capture filter and http in the display filter:

The green box indicates that Wireshark is happy with the filter you typed. Now you can begin the capture by clicking on the fin in the top left:

Clicking this button will spawn a new window in Wireshark:

This new window is fairly plain, but the message at the bottom says <live capture in progress>, which indicates that it’s working. Don’t worry that nothing is being displayed, as that’s normal. In order for Wireshark to report anything, there has to be some activity on your server. To get some data, try running your client:

$ SECRET_URL="http://127.0.0.1:5683" python client.py
The secret message is: fluffy tail

After executing the client.py code from above, you should now see some entries in Wireshark. If all has gone well, then you’ll see two entries that look something like this:

These two entries represent the two parts of the communication that occurred. The first one is the client’s request to your server. When you click on the first entry, you’ll see a plethora of information:

That’s a lot of information! At the top, you still have your HTTP request and response. Once you select one of these entries, you’ll see the middle and bottom row populate with information.

The middle row provides you with a breakdown of the protocols that Wireshark was able to identify for the selected request. This breakdown allows you to explore what actually happened in your HTTP Request. Here’s a quick summary of the information Wireshark describes in the middle row from top to bottom:

Physical Layer: This row describes the physical interface used to send the request. In your case, this is probably Interface ID 0 (lo) for your loopback interface.
Ethernet Information: This row shows you the Layer-2 Protocol, which includes source and destination MAC addresses.
IPv4: This row displays source and destination IP addresses (127.0.0.1).
TCP: This row includes the required TCP handshake in order to create a reliable pipe of data.
HTTP: This row displays information about the HTTP request itself.

When you expand the Hypertext Transfer Protocol layer, you can see all the information that makes up an HTTP Request:

This image shows you your script’s HTTP request:

Method: GET
Path: /
Version: 1.1
Headers: Host: 127.0.0.1:5683, Connection: keep-alive, and others
Body: No body

The last row you’ll see is a hex dump of the data. You may notice in this hex dump that you can actually see the parts of your HTTP request. That’s because your HTTP request was sent in the open. But what about the reply? If you click on the HTTP response, then you’ll see a similar view:

Again, you have the same three sections. If you look carefully at the hex dump, then you’ll see the secret message in plain text! This is a big problem for the Secret Squirrels. What this means is that anyone with some technical know-how can very easily see this traffic if they’re interested. So, how do you solve this problem? The answer is cryptography.

How Does Cryptography Help?

In this section, you’ll learn one way to keep your data safe by creating your own cryptography keys and using them on both your server and your client. While this won’t be your final step, it will help you get a solid foundation for how to build Python HTTPS applications.

Understanding Cryptography Basics

Cryptography is a way to secure communications from eavesdroppers, or adversaries. Another way to state this is that you’re taking normal information, called plaintext, and converting it to scrambled text, called ciphertext.

Cryptography can be intimidating at first, but the fundamental concepts are pretty accessible. In fact, you’ve probably already practiced cryptography before. If you’ve ever had a secret language with your friends and used it to pass notes in class, then you’ve practiced cryptography. (If you haven’t done that, then don’t worry—you’re about to.)

Somehow, you need to take the string "fluffy tail" and convert it into something unintelligible. One way to do this is to map certain characters onto different characters. An effective way to do this is to shift the characters back one spot in the alphabet. Doing so would look something like this:

This image shows you how to translate from the original alphabet to the new alphabet and back. So, if you had the message ABC, then you would actually send the message ZAB. If you apply this to "fluffy tail", then assuming that spaces stay the same, you get ekteex szhk. While it’s not perfect, it’ll probably look like gibberish to anyone that sees it.

Congratulations! You’ve created what is known in cryptography as a cipher, which describes how to convert plaintext to ciphertext and back. Your cipher, in this case, is described in English. This particular type of cipher is called a substitution cipher. Fundamentally this is the same type of cipher used in the Enigma Machine, albeit a much simpler version.

Now, if you wanted to get a message out to the Secret Squirrels, then you would first need to tell them how many letters to shift and then give them the encoded message. In Python, this might look something like the following:

CIPHER = {"a": "z", "A": "Z", "b": "a"} # And so on

def encrypt(plaintext: str):
    return "".join(CIPHER.get(letter, letter) for letter in plaintext)

Here, you’ve created a function called encrypt(), which will take plaintext and convert it to ciphertext. Imagine you have a dictionary CIPHER that has all of the characters mapped out. Similarly, you could create a decrypt():

DECIPHER = {v: k for k, v in CIPHER.items()}

def decrypt(ciphertext: str):
    return "".join(DECIPHER.get(letter, letter) for letter in ciphertext)

This function is the opposite of encrypt(). It will take ciphertext and convert it to plaintext. In this form of the cipher, you have a special key that users need to know in order to encrypt and decrypt messages. For the above example, that key is 1. That is, the cipher indicates you should shift each letter back by one character. The key is very important to keep secret because anyone with the key can easily decrypt your message.

Note: While you could use this for your encryption, this still isn’t terribly secure. This cipher is quick to break using frequency analysis and is much too primitive for the Secret Squirrels.

In the modern era, cryptography is a lot more advanced. It relies on complex mathematical theory and computer science to be secure. While the math behind these ciphers is outside of the scope of this tutorial, the underlying concepts are still the same. You have a cipher that describes how to take plaintext and convert it into ciphertext.

The only real difference between your substitution cipher and modern ciphers is that modern ciphers are mathematically proven to be impractical to break by an eavesdropper. Now, let’s see how to use your new ciphers.

Using Cryptography in Python HTTPS Applications

Luckily for you, you don’t have to be an expert in mathematics or computer science to use cryptography. Python also has a secrets module that can help you generate cryptographically-secure random data. In this tutorial, you’ll learn about a Python library that’s aptly named cryptography. It’s available on PyPI, so you can install it with pip:

$ pip install cryptography

This will install cryptography into your virtual environment. With cryptography installed, you can now encrypt and decrypt things in a way that’s mathematically secure by using the Fernet method.

Recall that your secret key in your cipher was 1. In the same vein, you need to create a key for Fernet to work correctly:

>>>

>>> from cryptography.fernet import Fernet
>>> key = Fernet.generate_key()
>>> key
b'8jtTR9QcD-k3RO9Pcd5ePgmTu_itJQt9WKQPzqjrcoM='

In this code, you’ve imported Fernet and generated a key. The key is just a bunch of bytes, but it’s incredibly important that you keep this key secret and safe. Just like the substitution example above, anyone with this key can easily decrypt your messages.

Note: In real life, you would keep this key very secure. In these examples, it’s helpful to see the key, but this is bad practice, especially if you’re posting it on a public website! In other words, do not use the exact key you see above for anything you want to be secure.

This key behaves much like the earlier key. It’s required to make the transition to ciphertext and back to plaintext. Now it’s time for the fun part! You can encrypt a message like this:

>>>

>>> my_cipher = Fernet(key)
>>> ciphertext = my_cipher.encrypt(b"fluffy tail")
>>> ciphertext
b'gAAAAABdlW033LxsrnmA2P0WzaS-wk1UKXA1IdyDpmHcV6yrE7H_ApmSK8KpCW-6jaODFaeTeDRKJMMsa_526koApx1suJ4_dQ=='

In this code, you’ve created a Fernet object called my_cipher, which you can then use to encrypt your message. Notice that your secret message "fluffy tail" needs to be a bytes object in order to encrypt it. After the encryption, you can see that the ciphertext is a long stream of bytes.

Thanks to Fernet, this ciphertext can’t be manipulated or read without the key! This type of encryption requires that both the server and the client have access to the key. When both sides require the same key, this is called symmetric encryption. In the next section, you’ll see how to put this symmetric encryption to use to keep your data safe.

Seeing That Your Data Is Safe

Now that you understand some of the basics of cryptography in Python, you can apply this knowledge to your server. Create a new file called symmetric_server.py:

# symmetric_server.py
import os
from flask import Flask
from cryptography.fernet import Fernet

SECRET_KEY = os.environb[b"SECRET_KEY"]
SECRET_MESSAGE = b"fluffy tail"
app = Flask(__name__)

my_cipher = Fernet(SECRET_KEY)

@app.route("/")
def get_secret_message():
    return my_cipher.encrypt(SECRET_MESSAGE)

This code combines your original server code with the Fernet object you used in the previous section. The key is now read as a bytes object from the environment using os.environb. With the server out of the way, you can now focus on the client. Paste the following into symmetric_client.py:

# symmetric_client.py
import os
import requests
from cryptography.fernet import Fernet

SECRET_KEY = os.environb[b"SECRET_KEY"]
my_cipher = Fernet(SECRET_KEY)

def get_secret_message():
    response = requests.get("http://127.0.0.1:5683")

    decrypted_message = my_cipher.decrypt(response.content)
    print(f"The codeword is: {decrypted_message}")

if __name__ == "__main__":
    get_secret_message()

Once again, this is modified code to combine your earlier client with the Fernet encryption mechanism. get_secret_message() does the following:

Make the request to your server.
Take the raw bytes from the response.
Attempt to decrypt the raw bytes.
Print the decrypted message.

If you run both the server and the client, then you’ll see that you’re successfully encrypting and decrypting your secret message:

$ uwsgi --http-socket 127.0.0.1:5683 \
    --env SECRET_KEY="8jtTR9QcD-k3RO9Pcd5ePgmTu_itJQt9WKQPzqjrcoM=" \
    --mount /=symmetric_server:app

In this call, you start the server on port 5683 again. This time, you pass in a SECRET_KEY which must be at least a 32-length base64 encoded string. With your server restarted, you may now query it:

$ SECRET_KEY="8jtTR9QcD-k3RO9Pcd5ePgmTu_itJQt9WKQPzqjrcoM=" python symmetric_client.py
The secret message is: b'fluffy tail'

Woohoo! Your were able to encrypt and decrypt your message. If you try running this with an invalid SECRET_KEY, then you’ll get an error:

$ SECRET_KEY="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" python symmetric_client.py
Traceback (most recent call last):
  File ".../cryptography/fernet.py", line 104, in _verify_signature
    h.verify(data[-32:])
  File ".../cryptography/hazmat/primitives/hmac.py", line 66, in verify
    ctx.verify(signature)
  File ".../cryptography/hazmat/backends/openssl/hmac.py", line 74, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "symmetric_client.py", line 16, in <module>
    get_secret_message()
  File "symmetric_client.py", line 11, in get_secret_message
    decrypted_message = my_cipher.decrypt(response.content)
  File ".../cryptography/fernet.py", line 75, in decrypt
    return self._decrypt_data(data, timestamp, ttl)
  File ".../cryptography/fernet.py", line 117, in _decrypt_data
    self._verify_signature(data)
  File ".../cryptography/fernet.py", line 106, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

So, you know the encryption and decryption are working. But is it secure? Well, yes, it is. To prove this, you can go back to Wireshark and start a new capture with the same filters as before. After you have the capture setup, run the client code again:

$ SECRET_KEY="8jtTR9QcD-k3RO9Pcd5ePgmTu_itJQt9WKQPzqjrcoM=" python symmetric_client.py
The secret message is: b'fluffy tail'

You’ve made another successful HTTP request and response, and once again, you see these messages in Wireshark. Since the secret message only gets transferred in the response, you can click on that to look at the data:

In the middle row of this picture, you can see the data that was actually transferred:

gAAAAABdlXSesekh9LYGDpZE4jkxm4Ai6rZQg2iHaxyDXkPWz1O74AB37V_a4vabF13fEr4kwmCe98Wlr8Zo1XNm-WjAVtSgFQ==

Awesome! This means that the data was encrypted and that eavesdroppers have no clue what the message content actually is. Not only that, but it also means that they could spend an insanely long amount of time trying to brute-force crack this data, and they would almost never be successful.

Your data is safe! But wait a minute—you never had to know anything about a key when you were using Python HTTPS applications before. That’s because HTTPS doesn’t use symmetric encryption exclusively. As it turns out, sharing secrets is a hard problem.

To prove this concept, navigate to http://127.0.0.1:5683 in your browser, and you’ll see the encrypted response text. This is because your browser doesn’t know anything about your secret cipher key. So how do Python HTTPS applications really work? That’s where asymmetric encryption comes into play.

How Are Keys Shared?

In the previous section, you saw how you could use symmetric encryption to keep your data secure as it traverses the Internet. Still, even though symmetric encryption is secure, it isn’t the only encryption technique used by Python HTTPS applications to keep your data safe. Symmetric encryption introduces some fundamental problems that are not so easily solved.

Note: Remember, symmetric encryption requires that you have a shared key between client and server. Unfortunately, security only works as hard as your weakest link, and weak links are especially catastrophic in symmetric encryption. Once one person compromises the key, then every key is compromised. It’s safe to assume that any security system will, at some point, become compromised.

So, how do you change your key? If you only have one server and one client, then this might be a quick task. However, as you get more clients and more servers, there is more and more coordination that needs to happen in order to change the key and keep your secrets safe effectively.

Moreover, you have to choose a new secret every time. In the above example, you saw a randomly generated key. It can be next to impossible for you to try and get people to remember that key. As your client and server numbers grow, you’ll likely use keys that are easier to remember and guess.

If you can deal with changing your key, then you still have one more problem to solve. How do you share your initial key? In the Secret Squirrels example, you solved this problem by having physical access to each of the members. You could give each member the secret in person and tell them to keep it secret, but remember that someone will be the weakest link.

Now, suppose you add a member to the Secret Squirrels from another physical location. How do you share the secret with this member? Do you make them take a plane to you each time the key changes? It would be nice if you could put up the secret key on your server and share it automatically. Unfortunately, this would defeat the whole purpose of encryption, since anyone could get the secret key!

Of course, you could give everyone an initial master key to get the secret message, but now you just have twice as many problems as before. If your head hurts, then don’t worry! You’re not the only one.

What you need is for two parties that have never communicated to have a shared secret. Sounds impossible, right? Luckily, three guys by the names of Ralph Merkle, Whitfield Diffie, and Martin Hellman have your back. They helped demonstrate that public-key cryptography, otherwise known as asymmetric encryption, was possible.

Note: While Whitfield Diffie and Martin Hellman are widely known as the first to discover the scheme, it was revealed in 1997 that three men working for GCHQ named James H. Ellis, Clifford Cocks, and Malcolm J. Williamson had previously demonstrated this capability seven years earlier!

Asymmetric encryption allows for two users who have never communicated before to share a common secret. One of the easiest ways to understand the fundamentals is to use a color analogy. Imagine you have the following scenario:

In this diagram, you’re trying to communicate with a Secret Squirrel you’ve never met before, but a spy can see everything you send. You know about symmetric encryption and would like to use it, but you first need to share a secret. Luckily, both of you have a private key. Unfortunately, you can’t send your private key because the spy will see it. So what do you do?

The first thing you need to do is agree with your partner on a color, like yellow:

Notice here that the spy can see the shared color, as can you and the Secret Squirrel. The shared color is effectively public. Now, both you and the Secret Squirrel combine your private keys with the shared color:

Your colors combine to make green, while the Secret Squirrel’s colors combine to make orange. Both of you are done with the shared color, and now you need to share your combined colors with one another:

You now have your private key and the Secret Squirrel’s combined color. Likewise, the Secret Squirrel has their private key and your combined color. It was pretty quick for you and the Secret Squirrel to combine your colors.

The spy, however, only has these combined colors. Trying to figure out your exact original color is very hard, even given the initial shared color. The spy would have to go to the store and buy lots of different blues to try. Even then, it would be hard to know if they were looking at the right shade of green after the combination! In short, your private key is still private.

But what about you and the Secret Squirrel? You still don’t have a combined secret! This is where your private key comes back. If you combine your private key with the combined color you received from the Secret Squirrel, then you’ll both end up with the same color:

Now, you and the Secret Squirrel have the same shared secret color. You have now successfully shared a secure secret with a complete stranger. This is surprisingly accurate to how public-key cryptography works. Another common name for this sequence of events is the Diffie-Hellman key exchange. The key exchange is made up of the following parts:

The private key is your private color from the examples.
The public key is the combined color that you shared.

The private key is something you always keep private, while the public key can be shared with anyone. These concepts map directly to the real world of Python HTTPS applications. Now that the server and the client have a shared secret, you can use your old pal symmetric encryption to encrypt all further messages!

Note: Public-key cryptography also relies on some math to do color mixing. The Wikipedia page for the Diffie-Hellman key exchange has a good explanation, but an in-depth explanation is outside the scope of this tutorial.

When you’re communicating over a secure website, like this one, your browser and the server set up a secure communication using these same principles:

Your browser requests information from the server.
Your browser and the server exchange public keys.
Your browser and the server generate a shared private key.
Your browser and the server encrypt and decrypt messages using this shared key through symmetric encryption.

Luckily for you, you don’t need to implement any of these details. There are lots of built-in and third-party libraries that can help you keep your client and server communications secure.

What Is HTTPS Like in the Real World?

Given all this information about encryption, let’s zoom out a bit and talk about how Python HTTPS applications actually work in the real world. Encryption is only half of the story. When visiting a secure website, there are two major components needed:

Encryption converts plaintext to ciphertext and back.
Authentication verifies that a person or thing is who they say they are.

You’ve heard extensively about how encryption works, but what about authentication? To understand authentication in the real world, you’ll need to know about Public Key Infrastructure. PKI introduces another important concept into the security ecosystem, called certificates.

Certificates are like passports for the Internet. Like most things in the computer world, they are just chunks of data in a file. Generally speaking, certificates include the following information:

Issued To: identifies who owns the certificate
Issued By: identifies who issued the certificate
Validity Period: identifies the time frame for which the certificate is valid

Just like passports, certificates are only really useful if they’re generated and recognized by some authority. It’s impractical for your browser to know about every single certificate of every site you visit on the Internet. Instead, PKI relies on a concept known as Certificate Authorities (CA).

Certificate Authorities are responsible for issuing certificates. They are considered a trusted third party (TTP) in PKI. Essentially, these entities act as valid authorities for a certificate. Suppose you’d like to visit another country, and you have a passport with all your information on it. How do the immigration officers in the foreign country know that your passport contains valid information?

If you were to fill out all the information yourself and sign it, then each immigration officer in each country you want to visit would need to know you personally and be able to attest that the information there was indeed correct.

Another way to handle this is to send all your information into a Trusted Third Party (TTP). The TTP would do a thorough investigation of the information you provided, verify your claims, and then sign your passport. This turns out to be much more practical because the immigration officers only need to know the trusted third parties.

The TTP scenario is how certificates are handled in practice. The process goes something like this:

Create a Certificate Signing Request (CSR): This is like filling out the information for your visa.
Send the CSR to a Trusted Third Party (TTP): This is like sending your information into a visa application office.
Verify your information: Somehow, the TTP needs to verify the information you provided. As an example, see how Amazon validates ownership.
Generate a Public Key: The TTP signs your CSR. This is equivalent to the TTP signing your visa.
Issue the verified Public Key: This is equivalent to you receiving your visa in the mail.

Note that the CSR is tied cryptographically to your private key. As such, all three pieces of information—public key, private key, and certificate authority—are related in one way or another. This creates what is known as a chain of trust, so you now have a valid certificate that can be used to verify your identity.

Most often, this is only the responsibility of website owners. A website owner will follow all these steps. At the end of this process, their certificate says the following:

From time A to time B I am X according to Y

This sentence is all that a certificate really tells you. The variables can be filled in as follows:

A is the valid start date and time.
B is the valid end date and time.
X is the name of the server.
Y is the name of the Certificate Authority.

Fundamentally, this is all a certificate describes. In other words, having a certificate doesn’t necessarily mean that you are who you say you are, just that you got Y to agree that you are who you say you are. This is where the “trusted” part of trusted third parties come in.

TTPs need to be shared between clients and servers in order for everyone to be happy about the HTTPS handshake. Your browser comes with lots of Certificate Authorities automatically installed. To see them, take the following steps:

Chrome: Go to Settings > Advanced > Privacy and security > Manage certificates > Authorities.
Firefox: Go to Settings > Preferences > Privacy & Security > View Certificates > Authorities.

This covers the infrastructure required to create Python HTTPS applications in the real world. In the next section, you’ll apply these concepts to your own code. You’ll walk through the most common examples and become your own Certificate Authority for the Secret Squirrels!

What Does a Python HTTPS Application Look Like?

Now that you have an understanding of the basic parts required for a making a Python HTTPS application, it’s time to tie all the pieces together one-by-one to your application from before. This will ensure that your communication between server and client is secure.

It’s possible to set up the entire PKI infrastructure on your own machine, and this is exactly what you’ll be doing in this section. It’s not as hard as it sounds, so don’t worry! Becoming a real Certificate Authority is significantly harder than taking the steps below, but what you’ll read is, more or less, all you’d need to run your own CA.

Becoming a Certificate Authority

A Certificate Authority is nothing more than a very important public and private key pair. To become a CA, you just need to generate a public and private key pair.

Note: Becoming a CA that’s meant for use by the public is a very arduous process, though there are many companies that have followed this process. You won’t be one of those companies by the end of this tutorial, however!

Your initial public and private key pair will be a self-signed certificate. You’re generating the initial secret, so if you’re actually going to become a CA, then it’s incredibly important that this private key is safe. If someone gets access to the CA’s public and private key pair, then they can generate a completely valid certificate, and there’s nothing you can do to detect the problem except to stop trusting your CA.

With that warning out of the way, you can generate the certificate in no time. For starters, you’ll need to generate a private key. Paste the following into a file called pki_helpers.py:

 1 # pki_helpers.py
 2 from cryptography.hazmat.backends import default_backend
 3 from cryptography.hazmat.primitives import serialization
 4 from cryptography.hazmat.primitives.asymmetric import rsa
 5 
 6 def generate_private_key(filename: str, passphrase: str):
 7     private_key = rsa.generate_private_key(
 8         public_exponent=65537, key_size=2048, backend=default_backend()
 9     )
10 
11     utf8_pass = passphrase.encode("utf-8")
12     algorithm = serialization.BestAvailableEncryption(utf8_pass)
13 
14     with open(filename, "wb") as keyfile:
15         keyfile.write(
16             private_key.private_bytes(
17                 encoding=serialization.Encoding.PEM,
18                 format=serialization.PrivateFormat.TraditionalOpenSSL,
19                 encryption_algorithm=algorithm,
20             )
21         )
22 
23     return private_key

generate_private_key() generates a private key using RSA. Here’s a breakdown of the code:

Lines 2 to 4 import the libraries required for the function to work.
Lines 7 to 9 use RSA to generate a private key. The magic numbers 65537 and 2048 are just two possible values. You can read more about why or just trust that these numbers are useful.
Lines 11 to 12 set up the encryption algorithm to be used on your private key.
Lines 14 to 21 write your private key to disk at the specified filename. This file is encrypted using the password provided.

The next step in becoming your own CA is to generate a self-signed public key. You can bypass the certificate signing request (CSR) and immediately build a public key. Paste the following into pki_helpers.py:

 1 # pki_helpers.py
 2 from datetime import datetime, timedelta
 3 from cryptography import x509
 4 from cryptography.x509.oid import NameOID
 5 from cryptography.hazmat.primitives import hashes
 6 
 7 def generate_public_key(private_key, filename, **kwargs):
 8     subject = x509.Name(
 9         [
10             x509.NameAttribute(NameOID.COUNTRY_NAME, kwargs["country"]),
11             x509.NameAttribute(
12                 NameOID.STATE_OR_PROVINCE_NAME, kwargs["state"]
13             ),
14             x509.NameAttribute(NameOID.LOCALITY_NAME, kwargs["locality"]),
15             x509.NameAttribute(NameOID.ORGANIZATION_NAME, kwargs["org"]),
16             x509.NameAttribute(NameOID.COMMON_NAME, kwargs["hostname"]),
17         ]
18     )
19 
20     # Because this is self signed, the issuer is always the subject
21     issuer = subject
22 
23     # This certificate is valid from now until 30 days
24     valid_from = datetime.utcnow()
25     valid_to = valid_from + timedelta(days=30)
26 
27     # Used to build the certificate
28     builder = (
29         x509.CertificateBuilder()
30         .subject_name(subject)
31         .issuer_name(issuer)
32         .public_key(private_key.public_key())
33         .serial_number(x509.random_serial_number())
34         .not_valid_before(valid_from)
35         .not_valid_after(valid_to)
36     )
37 
38     # Sign the certificate with the private key
39     public_key = builder.sign(
40         private_key, hashes.SHA256(), default_backend()
41     )
42 
43     with open(filename, "wb") as certfile:
44         certfile.write(public_key.public_bytes(serialization.Encoding.PEM))
45 
46     return public_key

Here you have a new function generate_public_key() that will generate a self-signed public key. Here’s how this code works:

Lines 2 to 5 are imports required for the function to work.
Lines 8 to 18 build up information about the subject of the certificate.
Line 21 uses the same issuer and subject since this is a self-signed certificate.
Lines 24 to 25 indicate the time range during which this public key is valid. In this case, it’s 30 days.
Lines 28 to 36 add all required information to a public key builder object, which then needs to be signed.
Lines 38 to 41 sign the public key with the private key.
Lines 43 to 44 write the public key out to filename.

Using these two functions, you can generate your private and public key pair quite quickly in Python:

>>>

>>> from pki_helpers import generate_private_key, generate_public_key
>>> private_key = generate_private_key("ca-private-key.pem", "secret_password")
>>> private_key
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x7ffbb292bf90>
>>> generate_public_key(
...   private_key,
...   filename="ca-public-key.pem",
...   country="US",
...   state="Maryland",
...   locality="Baltimore",
...   org="My CA Company",
...   hostname="my-ca.com",
... )
<Certificate(subject=<Name(C=US,ST=Maryland,L=Baltimore,O=My CA Company,CN=logan-ca.com)>, ...)>

After importing your helper functions from pki_helpers, you first generate your private key and save it to the file ca-private-key.pem. You then pass that private key into generate_public_key() to generate your public key. In your directory you should now have two files:

$ ls ca*
ca-private-key.pem ca-public-key.pem

Congratulations! You now have the ability to be a Certificate Authority.

Trusting Your Server

The first step to your server becoming trusted is for you to generate a Certificate Signing Request (CSR). In the real world, the CSR would be sent to an actual Certificate Authority like Verisign or Let’s Encrypt. In this example, you’ll use the CA you just created.

Paste the code for generating a CSR into the pki_helpers.py file from above:

 1 # pki_helpers.py
 2 def generate_csr(private_key, filename, **kwargs):
 3     subject = x509.Name(
 4         [
 5             x509.NameAttribute(NameOID.COUNTRY_NAME, kwargs["country"]),
 6             x509.NameAttribute(
 7                 NameOID.STATE_OR_PROVINCE_NAME, kwargs["state"]
 8             ),
 9             x509.NameAttribute(NameOID.LOCALITY_NAME, kwargs["locality"]),
10             x509.NameAttribute(NameOID.ORGANIZATION_NAME, kwargs["org"]),
11             x509.NameAttribute(NameOID.COMMON_NAME, kwargs["hostname"]),
12         ]
13     )
14 
15     # Generate any alternative dns names
16     alt_names = []
17     for name in kwargs.get("alt_names", []):
18         alt_names.append(x509.DNSName(name))
19     san = x509.SubjectAlternativeName(alt_names)
20 
21     builder = (
22         x509.CertificateSigningRequestBuilder()
23         .subject_name(subject)
24         .add_extension(san, critical=False)
25     )
26 
27     csr = builder.sign(private_key, hashes.SHA256(), default_backend())
28 
29     with open(filename, "wb") as csrfile:
30         csrfile.write(csr.public_bytes(serialization.Encoding.PEM))
31 
32     return csr

For the most part this code is identical to how you generated your original public key. The main differences are outlined below:

Lines 16 to 19 set up alternate DNS names, which will be valid for your certificate.
Lines 21 to 25 generate a different builder object, but the same fundamental principle applies as before. You’re building all the required attributes for your CSR.
Line 27 signs your CSR with a private key.
Lines 29 to 30 write your CSR to disk in PEM format.

You’ll notice that, in order to create a CSR, you’ll need a private key first. Luckily, you can use the same generate_private_key() from when you created your CA’s private key. Using the above function and the previous methods defined, you can do the following:

>>>

>>> from pki_helpers import generate_csr, generate_private_key
>>> server_private_key = generate_private_key(
...   "server-private-key.pem", "serverpassword"
... )
>>> server_private_key
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x7f6adafa3050>
>>> generate_csr(
...   server_private_key,
...   filename="server-csr.pem",
...   country="US",
...   state="Maryland",
...   locality="Baltimore",
...   org="My Company",
...   alt_names=["localhost"],
...   hostname="my-site.com",
... )
<cryptography.hazmat.backends.openssl.x509._CertificateSigningRequest object at 0x7f6ad5372210>

After you run these steps in a console, you should end up with two new files:

server-private-key.pem: your server’s private key
server-csr.pem: your server’s CSR

You can view your new CSR and private key from the console:

$ ls server*.pem
server-csr.pem  server-private-key.pem

With these two documents in hand, you can now begin the process of signing your keys. Typically, lots of verification would happen in this step. In the real world, the CA would make sure that you owned my-site.com and ask you to prove it in various ways.

Since you are the CA in this case, you can forego that headache create your very own verified public key. To do that, you’ll add another function to your pki_helpers.py file:

# pki_helpers.py
def sign_csr(csr, ca_public_key, ca_private_key, new_filename):
    valid_from = datetime.utcnow()
    valid_until = valid_from + timedelta(days=30)

    builder = (
        x509.CertificateBuilder()
        .subject_name(csr.subject)
        .issuer_name(ca_public_key.subject)
        .public_key(csr.public_key())
        .serial_number(x509.random_serial_number())
        .not_valid_before(valid_from)
        .not_valid_after(valid_until)
    )

    for extension in csr.extensions:
        builder = builder.add_extension(extension.value, extension.critical)

    public_key = builder.sign(
        private_key=ca_private_key,
        algorithm=hashes.SHA256(),
        backend=default_backend(),
    )

    with open(new_filename, "wb") as keyfile:
        keyfile.write(public_key.public_bytes(serialization.Encoding.PEM))

This code looks very similar to generate_public_key() from the generate_ca.py file. In fact, they’re nearly identical. The major differences are as follows:

Lines 8 to 9 base the subject name on the CSR, while the issuer is based on the Certificate Authority.
Line 10 gets the public key from the CSR this time.
Lines 16 to 17 copy any extensions that were set on the CSR.
Line 20 signs the public key with the CA’s private key.

The next step is to fire up the Python console and use sign_csr(). You’ll need to load your CSR and your CA’s private and public key. Begin by loading your CSR:

>>>

>>> from cryptography import x509
>>> from cryptography.hazmat.backends import default_backend
>>> csr_file = open("server-csr.pem", "rb")
>>> csr = x509.load_pem_x509_csr(csr_file.read(), default_backend())
>>> csr
<cryptography.hazmat.backends.openssl.x509._CertificateSigningRequest object at 0x7f68ae289150>

In this section of code, you’re opening up your server-csr.pem file and using x509.load_pem_x509_csr() to create your csr object. Next up, you’ll need to load your CA’s public key:

>>>

>>> ca_public_key_file = open("ca-public-key.pem", "rb")
>>> ca_public_key = x509.load_pem_x509_certificate(
...   ca_public_key_file.read(), default_backend()
... )
>>> ca_public_key
<Certificate(subject=<Name(C=US,ST=Maryland,L=Baltimore,O=My CA Company,CN=logan-ca.com)>, ...)>

Once again, you’ve created a ca_public_key object which can be used by sign_csr(). The x509 module had the handy load_pem_x509_certificate() to help. The final step is to load your CA’s private key:

>>>

>>> from getpass import getpass
>>> from cryptography.hazmat.primitives import serialization
>>> ca_private_key_file = open("ca-private-key.pem", "rb")
>>> ca_private_key = serialization.load_pem_private_key(
...   ca_private_key_file.read(),
...   getpass().encode("utf-8"),
...   default_backend(),
... )
Password:
>>> private_key
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x7f68a85ade50>

This code will load up your private key. Recall from earlier that your private key was encrypted using the password you specified. With these three components, you can now sign your CSR and generate a verified public key:

>>>

>>> from pki_helpers import sign_csr
>>> sign_csr(csr, ca_public_key, ca_private_key, "server-public-key.pem")

After running this, you should have three server key files in your directory:

$ ls server*.pem
server-csr.pem  server-private-key.pem  server-public-key.pem

Whew! That was quite a lot of work. The good news is that now that you have your private and public key pair, you don’t have to change any server code to start using it.

Using your original server.py file, run the following command to start your brand new Python HTTPS application:

$ uwsgi \
    --master \
    --https localhost:5683,\
            logan-site.com-public-key.pem,\
            logan-site.com-private-key.pem \
    --mount /=server:app

Congratulations! You now have a Python HTTPS-enabled server running with your very own private-public key pair, which was signed by your very own Certificate Authority!

Note: There is another side to the Python HTTPS authentication equation, and that’s the client. It’s possible to set up certificate verification for a client certificate as well. This requires a bit more work and isn’t really seen outside of enterprises. However, client authentication can be a very powerful tool.

Now, all that’s left to do is query your server. First, you’ll need to make some changes to the client.py code:

# client.py
import os
import requests

def get_secret_message():
    response = requests.get("https://localhost:5683")
    print(f"The secret message is {response.text}")

if __name__ == "__main__":
    get_secret_message()

The only change from the previous code is from http to https. If you try to run this code, then you’ll be met with an error:

$ python client.py
...
requests.exceptions.SSLError: \
    HTTPSConnectionPool(host='localhost', port=5683): \
    Max retries exceeded with url: / (Caused by \
    SSLError(SSLCertVerificationError(1, \
    '[SSL: CERTIFICATE_VERIFY_FAILED] \
    certificate verify failed: unable to get local issuer \
    certificate (_ssl.c:1076)')))

That’s quite the nasty error message! The important part here is the message certificate verify failed: unable to get local issuer. These words should be more familiar to you now. Essentially, it’s saying the following:

localhost:5683 gave me a certificate. I checked the issuer of the certificate it gave me, and according to all the Certificate Authorities I know about, that issuer is not one of them.

If you attempt to navigate to your website with your browser, then you’ll get a similar message:

If you want to avoid this message, then you have to tell requests about your Certificate Authority! All you need to do is point requests at the ca-public-key.pem file that you generated earlier:

# client.py
def get_secret_message():
    response = requests.get("http://localhost:5683", verify="ca-public-key.pem")
    print(f"The secret message is {response.text}")

After doing that, you should be able to run the following successfully:

$ python client.py
The secret message is fluffy tail

Nice! You’ve made a fully-functioning Python HTTPS server and queried it successfully. You and the Secret Squirrels now have messages that you can trade back and forth happily and securely!

Conclusion

In this tutorial, you’ve learned some of the core underpinnings of secure communications on the Internet today. Now that you understand these building blocks, you’ll become a better and more secure developer.

Throughout this tutorial, you’ve gained an understanding of several topics:

Cryptography
HTTPS and TLS
Public Key Infrastructure
Certificates

If this information has you interested, then you’re in luck! You’ve barely scratched the surface of all the nuances involved in every layer. The security world is constantly evolving, and new techniques and vulnerabilities are always being discovered. If you still have questions, then feel free to reach out in the comments section below or on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python args and kwargs: Demystified

2020-01-07T14:00:00+00:00

Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: *args and **kwargs. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in main(), then this course is for you! You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.

By the end of the course, you’ll know:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

This course assumes that you already know how to define Python functions and work with lists and dictionaries.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using Pandas and Python to Explore Your Dataset

2020-01-06T14:00:00+00:00

Do you have a large dataset that’s full of interesting insights, but you’re not sure where to start exploring it? Has your boss asked you to generate some statistics from it, but they’re not so easy to extract? These are precisely the use cases where Pandas and Python can help you! With these tools, you’ll be able to slice a large dataset down into manageable parts and glean insight from that information.

In this tutorial, you’ll learn how to:

Calculate metrics about your data
Perform basic queries and aggregations
Discover and handle incorrect data, inconsistencies, and missing values
Visualize your data with plots

You’ll also learn about the differences between the main data structures that Pandas and Python use. To follow along, you can get all of the example code in this tutorial at the link below:

Get Jupyter Notebook: Click here to get the Jupyter Notebook you'll use to explore data with Pandas in this tutorial.

Setting Up Your Environment

There are a few things you’ll need to get started with this tutorial. First is a familiarity with Python’s built-in data structures, especially lists and dictionaries. For more information, check out Lists and Tuples in Python and Dictionaries in Python.

The second thing you’ll need is a working Python environment. You can follow along in any terminal that has Python 3 installed. If you want to see nicer output, especially for the large NBA dataset you’ll be working with, then you might want to run the examples in a Jupyter notebook.

Note: If you don’t have Python installed at all, then check out Python 3 Installation & Setup Guide. You can also follow along online in a try-out Jupyter notebook.

The last thing you’ll need is the Pandas Python library, which you can install with pip:

$ python -m pip install pandas

You can also use the Conda package manager:

$ conda install pandas

If you’re using the Anaconda distribution, then you’re good to go! Anaconda already comes with the Pandas Python library installed.

Note: Have you heard that there are multiple package managers in the Python world and are somewhat confused about which one to pick? pip and conda are both excellent choices, and they each have their advantages.

If you’re going to use Python mainly for data science work, then conda is perhaps the better choice. In the conda ecosystem, you have two main alternatives:

If you want to get a stable data science environment up and running quickly, and you don’t mind downloading 500 MB of data, then check out the Anaconda distribution.
If you prefer a more minimalist setup, then check out the section on installing Miniconda in Setting Up Python for Machine Learning on Windows.

The examples in this tutorial have been tested with Python 3.7 and Pandas 0.25.0, but they should also work in older versions. You can get all the code examples you’ll see in this tutorial in a Jupyter notebook by clicking the link below:

Get Jupyter Notebook: Click here to get the Jupyter Notebook you'll use to explore data with Pandas in this tutorial.

Let’s get started!

Using the Pandas Python Library

Now that you’ve installed Pandas, it’s time to have a look at a dataset. In this tutorial, you’ll analyze NBA results provided by FiveThirtyEight in a 17MB CSV file. Create a script download_nba_all_elo.py to download the data:

import requests

download_url = "https://raw.githubusercontent.com/fivethirtyeight/data/master/nba-elo/nbaallelo.csv"
target_csv_path = "nba_all_elo.csv"

response = requests.get(download_url)
response.raise_for_status()    # Check that the request was successful
with open(target_csv_path, "wb") as f:
    f.write(response.content)
print("Download ready.")

When you execute the script, it will save the file nba_all_elo.csv in your current working directory.

Note: You could also use your web browser to download the CSV file.

However, having a download script has several advantages:

You can tell where you got your data.
You can repeat the download anytime! That’s especially handy if the data is often refreshed.
You don’t need to share the 17MB CSV file with your co-workers. Usually, it’s enough to share the download script.

Now you can use the Pandas Python library to take a look at your data:

>>>

>>> import pandas as pd
>>> nba = pd.read_csv("nba_all_elo.csv")
>>> type(nba)
<class 'pandas.core.frame.DataFrame'>

Here, you follow the convention of importing Pandas in Python with the pd alias. Then, you use .read_csv() to read in your dataset and store it as a DataFrame object in the variable nba.

Note: Is your data not in CSV format? No worries! The Pandas Python library provides several similar functions like read_json(), read_html(), and read_sql_table(). To learn how to work with these file formats, check out Reading and Writing Files With Pandas or consult the docs.

You can see how much data nba contains:

>>>

>>> len(nba)
126314
>>> nba.shape
(126314, 23)

You use the Python built-in function len() to determine the number of rows. You also use the .shape attribute of the DataFrame to see its dimensionality. The result is a tuple containing the number of rows and columns.

Now you know that there are 126,314 rows and 23 columns in your dataset. But how can you be sure the dataset really contains basketball stats? You can have a look at the first five rows with .head():

>>>

>>> nba.head()

If you’re following along with a Jupyter notebook, then you’ll see a result like this:

Unless your screen is quite large, your output probably won’t display all 23 columns. Somewhere in the middle, you’ll see a column of ellipses (...) indicating the missing data. If you’re working in a terminal, then that’s probably more readable than wrapping long rows. However, Jupyter notebooks will allow you to scroll. You can configure Pandas to display all 23 columns like this:

>>>

>>> pd.set_option("display.max.columns", None)

While it’s practical to see all the columns, you probably won’t need six decimal places! Change it to two:

>>>

>>> pd.set_option("display.precision", 2)

To verify that you’ve changed the options successfully, you can execute .head() again, or you can display the last five rows with .tail() instead:

>>>

>>> nba.tail()

Now, you should see all the columns, and your data should show two decimal places:

You can discover some further possibilities of .head() and .tail() with a small exercise. Can you print the last three lines of your DataFrame? Expand the code block below to see the solution:

Here’s how to print the last three lines of nba:

>>>

>>> nba.tail(3)

Your output should look something like this:

You can see the last three lines of your dataset with the options you’ve set above.

Similar to the Python standard library, functions in Pandas also come with several optional parameters. Whenever you bump into an example that looks relevant but is slightly different from your use case, check out the official documentation. The chances are good that you’ll find a solution by tweaking some optional parameters!

Getting to Know Your Data

You’ve imported a CSV file with the Pandas Python library and had a first look at the contents of your dataset. So far, you’ve only seen the size of your dataset and its first and last few rows. Next, you’ll learn how to examine your data more systematically.

Displaying Data Types

The first step in getting to know your data is to discover the different data types it contains. While you can put anything into a list, the columns of a DataFrame contain values of a specific data type. When you compare Pandas and Python data structures, you’ll see that this behavior makes Pandas much faster!

You can display all columns and their data types with .info():

>>>

>>> nba.info()

This will produce the following output:

You’ll see a list of all the columns in your dataset and the type of data each column contains. Here, you can see the data types int64, float64, and object. Pandas uses the NumPy library to work with these types. Later, you’ll meet the more complex categorical data type, which the Pandas Python library implements itself.

The object data type is a special one. According to the Pandas Cookbook, the object data type is “a catch-all for columns that Pandas doesn’t recognize as any other specific type.” In practice, it often means that all of the values in the column are strings.

Although you can store arbitrary Python objects in the object data type, you should be aware of the drawbacks to doing so. Strange values in an object column can harm Pandas’ performance and its interoperability with other libraries. For more information, check out the official getting started guide.

Showing Basics Statistics

Now that you’ve seen what data types are in your dataset, it’s time to get an overview of the values each column contains. You can do this with .describe():

>>>

>>> nba.describe()

This function shows you some basic descriptive statistics for all numeric columns:

.describe() only analyzes numeric columns by default, but you can provide other data types if you use the include parameter:

>>>

>>> import numpy as np
>>> nba.describe(include=np.object)

.describe() won’t try to calculate a mean or a standard deviation for the object columns, since they mostly include text strings. However, it will still display some descriptive statistics:

Take a look at the team_id and fran_id columns. Your dataset contains 104 different team IDs, but only 53 different franchise IDs. Furthermore, the most frequent team ID is BOS, but the most frequent franchise ID Lakers. How is that possible? You’ll need to explore your dataset a bit more to answer this question.

Exploring Your Dataset

Exploratory data analysis can help you answer questions about your dataset. For example, you can examine how often specific values occur in a column:

>>>

>>> nba["team_id"].value_counts()
BOS    5997
NYK    5769
LAL    5078
...
SDS      11
>>> nba["fran_id"].value_counts()
Name: team_id, Length: 104, dtype: int64
Lakers          6024
Celtics         5997
Knicks          5769
...
Huskies           60
Name: fran_id, dtype: int64

It seems that a team named "Lakers" played 6024 games, but only 5078 of those were played by the Los Angeles Lakers. Find out who the other "Lakers" team is:

>>>

>>> nba.loc[nba["fran_id"] == "Lakers", "team_id"].value_counts()
LAL    5078
MNL     946
Name: team_id, dtype: int64

Indeed, the Minneapolis Lakers ("MNL") played 946 games. You can even find out when they played those games:

>>>

>>> nba.loc[nba["team_id"] == "MNL", "date_game"].min()
'1/1/1949'
>>> nba.loc[nba["team_id"] == "MNL", "date_game"].max()
'4/9/1959'
>>> nba.loc[nba["team_id"] == "MNL", "date_game"].agg(("min", "max"))
min    1/1/1949
max    4/9/1959
Name: date_game, dtype: object

It looks like the Minneapolis Lakers played between the years of 1949 and 1959. That explains why you might not recognize this team!

You’ve also found out why the Boston Celtics team "BOS" played the most games in the dataset. Let’s analyze their history also a little bit. Find out how many points the Boston Celtics have scored during all matches contained in this dataset. Expand the code block below for the solution:

Similar to the .min() and .max() aggregate functions, you can also use .sum():

>>>

>>> nba.loc[nba["team_id"] == "BOS", "pts"].sum()
626484

The Boston Celtics scored a total of 626,484 points.

You’ve got a taste for the capabilities of a Pandas DataFrame. In the following sections, you’ll expand on the techniques you’ve just used, but first, you’ll zoom in and learn how this powerful data structure works.

Getting to Know Pandas’ Data Structures

While a DataFrame provides functions that can feel quite intuitive, the underlying concepts are a bit trickier to understand. For this reason, you’ll set aside the vast NBA DataFrame and build some smaller Pandas objects from scratch.

Understanding Series Objects

Python’s most basic data structure is the list, which is also a good starting point for getting to know pandas.Series objects. Create a new Series object based on a list:

>>>

>>> revenues = pd.Series([5555, 7000, 1980])
>>> revenues
0    5555
1    7000
2    1980
dtype: int64

You’ve used the list [5555, 7000, 1980] to create a Series object called revenues. A Series object wraps two components:

A sequence of values
A sequence of identifiers, which is the index

You can access these components with .values and .index, respectively:

>>>

>>> revenues.values
array([5555, 7000, 1980])
>>> revenues.index
RangeIndex(start=0, stop=3, step=1)

revenues.values returns the values in the Series, whereas revenues.index returns the positional index.

Note: If you’re familiar with NumPy, then it might be interesting for you to note that the values of a Series object are actually n-dimensional arrays:

>>>

>>> type(revenues.values)
<class 'numpy.ndarray'>

If you’re not familiar with NumPy, then there’s no need to worry! You can explore the ins and outs of your dataset with the Pandas Python library alone. However, if you’re curious about what Pandas does behind the scenes, then check out Look Ma, No For-Loops: Array Programming With NumPy.

While Pandas builds on NumPy, a significant difference is in their indexing. Just like a NumPy array, a Pandas Series also has an integer index that’s implicitly defined. This implicit index indicates the element’s position in the Series.

However, a Series can also have an arbitrary type of index. You can think of this explicit index as labels for a specific row:

>>>

>>> city_revenues = pd.Series(
...     [4200, 8000, 6500],
...     index=["Amsterdam", "Toronto", "Tokyo"]
... )
>>> city_revenues
Amsterdam    4200
Toronto      8000
Tokyo        6500
dtype: int64

Here, the index is a list of city names represented by strings. You may have noticed that Python dictionaries use string indices as well, and this is a handy analogy to keep in mind! You can use the code blocks above to distinguish between two types of Series:

revenues: This Series behaves like a Python list because it only has a positional index.
city_revenues: This Series acts like a Python dictionary because it features both a positional and a label index.

Here’s how to construct a Series with a label index from a Python dictionary:

>>>

>>> city_employee_count = pd.Series({"Amsterdam": 5, "Tokyo": 8})
>>> city_employee_count
Amsterdam    5
Tokyo        8
dtype: int64

The dictionary keys become the index, and the dictionary values are the Series values.

Just like dictionaries, Series also support .keys() and the in keyword:

>>>

>>> city_employee_count.keys()
Index(['Amsterdam', 'Tokyo'], dtype='object')
>>> "Tokyo" in city_employee_count
True
>>> "New York" in city_employee_count
False

You can use these methods to answer questions about your dataset quickly.

Understanding DataFrame Objects

While a Series is a pretty powerful data structure, it has its limitations. For example, you can only store one attribute per key. As you’ve seen with the nba dataset, which features 23 columns, the Pandas Python library has more to offer with its DataFrame. This data structure is a sequence of Series objects that share the same index.

If you’ve followed along with the Series examples, then you should already have two Series objects with cities as keys:

city_revenues
city_employee_count

You can combine these objects into a DataFrame by providing a dictionary in the constructor. The dictionary keys will become the column names, and the values should contain the Series objects:

>>>

>>> city_data = pd.DataFrame({
...     "revenue": city_revenues,
...     "employee_count": city_employee_count
... })
>>> city_data
           revenue  employee_count
Amsterdam     4200             5.0
Tokyo         6500             8.0
Toronto       8000             NaN

Note how Pandas replaced the missing employee_count value for Toronto with NaN.

The new DataFrame index is the union of the two Series indices:

>>>

>>> city_data.index
Index(['Amsterdam', 'Tokyo', 'Toronto'], dtype='object')

Just like a Series, a DataFrame also stores its values in a NumPy array:

>>>

>>> city_data.values
array([[4.2e+03, 5.0e+00],
       [6.5e+03, 8.0e+00],
       [8.0e+03,     nan]])

You can also refer to the 2 dimensions of a DataFrame as axes:

>>>

>>> city_data.axes
[Index(['Amsterdam', 'Tokyo', 'Toronto'], dtype='object'),
 Index(['revenue', 'employee_count'], dtype='object')]
>>> city_data.axes[0]
 Index(['Amsterdam', 'Tokyo', 'Toronto'], dtype='object')
>>> city_data.axes[1]
 Index(['revenue', 'employee_count'], dtype='object')

The axis marked with 0 is the row index, and the axis marked with 1 is the column index. This terminology is important to know because you’ll encounter several DataFrame methods that accept an axis parameter.

A DataFrame is also a dictionary-like data structure, so it also supports .keys() and the in keyword. However, for a DataFrame these don’t relate to the index, but to the columns:

>>>

>>> city_data.keys()
Index(['revenue', 'employee_count'], dtype='object')
>>> "Amsterdam" in city_data
False
>>> "revenue" in city_data
True

You can see these concepts in action with the bigger NBA dataset. Does it contain a column called "points", or was it called "pts"? To answer this question, display the index and the axes of the nba dataset, then expand the code block below for the solution:

Because you didn’t specify an index column when you read in the CSV file, Pandas has assigned a RangeIndex to the DataFrame:

>>>

>>> nba.index
RangeIndex(start=0, stop=126314, step=1)

nba, like all DataFrame objects, has two axes:

>>>

>>> nba.axes
[RangeIndex(start=0, stop=126314, step=1),
 Index(['gameorder', 'game_id', 'lg_id', '_iscopy', 'year_id', 'date_game',
        'seasongame', 'is_playoffs', 'team_id', 'fran_id', 'pts', 'elo_i',
        'elo_n', 'win_equiv', 'opp_id', 'opp_fran', 'opp_pts', 'opp_elo_i',
        'opp_elo_n', 'game_location', 'game_result', 'forecast', 'notes'],
       dtype='object')]

You can check the existence of a column with .keys():

>>>

>>> "points" in nba.keys()
False
>>> "pts" in nba.keys()
True

The column is called "pts", not "points".

As you use these methods to answer questions about your dataset, be sure to keep in mind whether you’re working with a Series or a DataFrame so that your interpretation is accurate.

Accessing Series Elements

In the section above, you’ve created a Pandas Series based on a Python list and compared the two data structures. You’ve seen how a Series object is similar to lists and dictionaries in several ways. A further similarity is that you can use the indexing operator ([]) for Series as well.

You’ll also learn how to use two Pandas-specific access methods:

.loc
.iloc

You’ll see that these data access methods can be much more readable than the indexing operator.

Using the Indexing Operator

Recall that a Series has two indices:

A positional or implicit index, which is always a RangeIndex
A label or explicit index, which can contain any hashable objects

Next, revisit the city_revenues object:

>>>

>>> city_revenues
Amsterdam    4200
Toronto      8000
Tokyo        6500
dtype: int64

You can conveniently access the values in a Series with both the label and positional indices:

>>>

>>> city_revenues["Toronto"]
8000
>>> city_revenues[1]
8000

You can also use negative indices and slices, just like you would for a list:

>>>

>>> city_revenues[-1]
6500
>>> city_revenues[1:]
Toronto    8000
Tokyo      6500
dtype: int64
>>> city_revenues["Toronto":]
Toronto    8000
Tokyo      6500
dtype: int64

If you want to learn more about the possibilities of the indexing operator, then check out Lists and Tuples in Python.

Using `.loc` and `.iloc`

The indexing operator ([]) is convenient, but there’s a caveat. What if the labels are also numbers? Say you have to work with a Series object like this:

>>>

>>> colors = pd.Series(
...     ["red", "purple", "blue", "green", "yellow"],
...     index=[1, 2, 3, 5, 8]
... )
>>> colors
1       red
2    purple
3      blue
5     green
8    yellow
dtype: object

What will colors[1] return? For a positional index, colors[1] is "purple". However, if you go by the label index, then colors[1] is referring to "red".

The good news is, you don’t have to figure it out! Instead, to avoid confusion, the Pandas Python library provides two data access methods:

.loc refers to the label index.
.iloc refers to the positional index.

These data access methods are much more readable:

>>>

>>> colors.loc[1]
'red'
>>> colors.iloc[1]
'purple'

colors.loc[1] returned "red", the element with the label 1. colors.iloc[1] returned "purple", the element with the index 1.

The following figure shows which elements .loc and .iloc refer to:

Again, .loc points to the label index on the right-hand side of the image. Meanwhile, .iloc points to the positional index on the left-hand side of the picture.

It’s easier to keep in mind the distinction between .loc and .iloc than it is to figure out what the indexing operator will return. Even if you’re familiar with all the quirks of the indexing operator, it can be dangerous to assume that everybody who reads your code has internalized those rules as well!

Note: In addition to being confusing for Series with numeric labels, the Python indexing operator has some performance drawbacks. It’s perfectly okay to use it in interactive sessions for ad-hoc analysis, but for production code, the .loc and .iloc data access methods are preferable. For further details, check out the Pandas User Guide section on indexing and selecting data.

.loc and .iloc also support the features you would expect from indexing operators, like slicing. However, these data access methods have an important difference. While .iloc excludes the closing element, .loc includes it. Take a look at this code block:

>>>

>>> # Return the elements with the implicit index: 1, 2
>>> colors.iloc[1:3]
2    purple
3      blue
dtype: object

If you compare this code with the image above, then you can see that colors.iloc[1:3] returns the elements with the positional indices of 1 and 2. The closing item "green" with a positional index of 3 is excluded.

On the other hand, .loc includes the closing element:

>>>

>>> # Return the elements with the explicit index between 3 and 8
>>> colors.loc[3:8]
3      blue
5     green
8    yellow
dtype: object

This code block says to return all elements with a label index between 3 and 8. Here, the closing item "yellow" has a label index of 8 and is included in the output.

You can also pass a negative positional index to .iloc:

>>>

>>> colors.iloc[-2]
'green'

You start from the end of the Series and return the second element.

Note: There used to be an .ix indexer, which tried to guess whether it should apply positional or label indexing depending on the data type of the index. Because it caused a lot of confusion, it has been deprecated since Pandas version 0.20.0.

It’s highly recommended that you do not use .ix for indexing. Instead, always use .loc for label indexing and .iloc for positional indexing. For further details, check out the Pandas User Guide.

You can use the code blocks above to distinguish between two Series behaviors:

You can use .iloc on a Series similar to using [] on a list.
You can use .loc on a Series similar to using [] on a dictionary.

Be sure to keep these distinctions in mind as you access elements of your Series objects.

Accessing DataFrame Elements

Since a DataFrame consists of Series objects, you can use the very same tools to access its elements. The crucial difference is the additional dimension of the DataFrame. You’ll use the indexing operator for the columns and the access methods .loc and .iloc on the rows.

Using the Indexing Operator

If you think of a DataFrame as a dictionary whose values are Series, then it makes sense that you can access its columns with the indexing operator:

>>>

>>> city_data["revenue"]
Amsterdam    4200
Tokyo        6500
Toronto      8000
Name: revenue, dtype: int64
>>> type(city_data["revenue"])
pandas.core.series.Series

Here, you use the indexing operator to select the column labeled "revenue".

If the column name is a string, then you can use attribute-style accessing with dot notation as well:

>>>

>>> city_data.revenue
Amsterdam    4200
Tokyo        6500
Toronto      8000
Name: revenue, dtype: int64

city_data["revenue"] and city_data.revenue return the same output.

There’s one situation where accessing DataFrame elements with dot notation may not work or may lead to surprises. This is when a column name coincides with a DataFrame attribute or method name:

>>>

>>> toys = pd.DataFrame([
...     {"name": "ball", "shape": "sphere"},
...     {"name": "Rubik's cube", "shape": "cube"}
... ])
>>> toys["shape"]
0    sphere
1      cube
Name: shape, dtype: object
>>> toys.shape
(2, 2)

The indexing operation toys["shape"] returns the correct data, but the attribute-style operation toys.shape still returns the shape of the DataFrame. You should only use attribute-style accessing in interactive sessions or for read operations. You shouldn’t use it for production code or for manipulating data (such as defining new columns).

Using `.loc` and `.iloc`

Similar to Series, a DataFrame also provides .loc and .iloc data access methods. Remember, .loc uses the label and .iloc the positional index:

>>>

>>> city_data.loc["Amsterdam"]
revenue           4200.0
employee_count       5.0
Name: Amsterdam, dtype: float64
>>> city_data.loc["Tokyo": "Toronto"]
        revenue employee_count
Tokyo   6500    8.0
Toronto 8000    NaN
>>> city_data.iloc[1]
revenue           6500.0
employee_count       8.0
Name: Tokyo, dtype: float64

Each line of code selects a different row from city_data:

city_data.loc["Amsterdam"] selects the row with the label index "Amsterdam".
city_data.loc["Tokyo": "Toronto"] selects the rows with label indices from "Tokyo" to "Toronto". Remember, .loc is inclusive.
city_data.iloc[1] selects the row with the positional index 1, which is "Tokyo".

Alright, you’ve used .loc and .iloc on small data structures. Now, it’s time to practice with something bigger! Use a data access method to display the second-to-last row of the nba dataset. Then, expand the code block below to see a solution:

The second-to-last row is the row with the positional index of -2. You can display it with .iloc:

>>>

>>> nba.iloc[-2]
gameorder               63157
game_id          201506170CLE
lg_id                     NBA
_iscopy                     0
year_id                  2015
date_game           6/16/2015
seasongame                102
is_playoffs                 1
team_id                   CLE
fran_id             Cavaliers
pts                        97
elo_i                 1700.74
elo_n                 1692.09
win_equiv             59.2902
opp_id                    GSW
opp_fran             Warriors
opp_pts                   105
opp_elo_i             1813.63
opp_elo_n             1822.29
game_location               H
game_result                 L
forecast              0.48145
notes                     NaN
Name: 126312, dtype: object

You’ll see the output as a Series object.

For a DataFrame, the data access methods .loc and .iloc also accept a second parameter. While the first parameter selects rows based on the indices, the second parameter selects the columns. You can use these parameters together to select a subset of rows and columns from your DataFrame:

>>>

>>> city_data.loc["Amsterdam": "Tokyo", "revenue"]
Amsterdam    4200
Tokyo        6500
Name: revenue, dtype: int64

Note that you separate the parameters with a comma (,). The first parameter, "Amsterdam" : "Tokyo," says to select all rows between those two labels. The second parameter comes after the comma and says to select the "revenue" column.

It’s time to see the same construct in action with the bigger nba dataset. Select all games between the labels 5555 and 5559. You’re only interested in the names of the teams and the scores, so select those elements as well. Expand the code block below to see a solution:

First, define which rows you want to see, then list the relevant columns:

>>>

>>> nba.loc[5555:5559, ["fran_id", "opp_fran", "pts", "opp_pts"]]

You use .loc for the label index and a comma (,) to separate your two parameters.

You should see a small part of your quite huge dataset:

The output is much easier to read!

With data access methods like .loc and .iloc, you can select just the right subset of your DataFrame to help you answer questions about your dataset.

Querying Your Dataset

You’ve seen how to access subsets of a huge dataset based on its indices. Now, you’ll select rows based on the values in your dataset’s columns to query your data. For example, you can create a new DataFrame that contains only games played after 2010:

>>>

>>> current_decade = nba[nba["year_id"] > 2010]
>>> current_decade.shape
(12658, 23)

You still have all 23 columns, but your new DataFrame only consists of rows where the value in the "year_id" column is greater than 2010.

You can also select the rows where a specific field is not null:

>>>

>>> games_with_notes = nba[nba["notes"].notnull()]
>>> games_with_notes.shape
(5424, 23)

This can be helpful if you want to avoid any missing values in a column. You can also use .notna() to achieve the same goal.

You can even access values of the object data type as str and perform string methods on them:

>>>

>>> ers = nba[nba["fran_id"].str.endswith("ers")]
>>> ers.shape
(27797, 23)

You use .str.endswith() to filter your dataset and find all games where the home team’s name ends with "ers".

You can combine multiple criteria and query your dataset as well. To do this, be sure to put each one in parentheses and use the logical operators | and & to separate them.

Note: The operators and, or, &&, and || won’t work here. If you’re curious as to why then check out the section on how the Pandas Python library uses Boolean operators in Python Pandas: Tricks & Features You May Not Know.

Do a search for Baltimore games where both teams scored over 100 points. In order to see each game only once, you’ll need to exclude duplicates:

>>>

>>> nba[
...     (nba["_iscopy"] == 0) &
...     (nba["pts"] > 100) &
...     (nba["opp_pts"] > 100) &
...     (nba["team_id"] == "BLB")
... ]

Here, you use nba["_iscopy"] == 0 to include only the entries that aren’t copies.

Your output should contain five eventful games:

Try to build another query with multiple criteria. In the spring of 1992, both teams from Los Angeles had to play a home game at another court. Query your dataset to find those two games. Both teams have an ID starting with "LA". Expand the code block below to see a solution:

You can use .str to find the team IDs that start with "LA", and you can assume that such an unusual game would have some notes:

>>>

>>> nba[
...     (nba["_iscopy"] == 0) &
...     (nba["team_id"].str.startswith("LA")) &
...     (nba["year_id"]==1992) &
...     (nba["notes"].notnull())
... ]

Your output should show two games on the day 5/3/1992:

Nice find!

When you know how to query your dataset with multiple criteria, you’ll be able to answer more specific questions about your dataset.

Grouping and Aggregating Your Data

You may also want to learn other features of your dataset, like the sum, mean, or average value of a group of elements. Luckily, the Pandas Python library offers grouping and aggregation functions to help you accomplish this task.

A Series has more than twenty different methods for calculating descriptive statistics. Here are some examples:

>>>

>>> city_revenues.sum()
18700
>>> city_revenues.max()
8000

The first method returns the total of city_revenues, while the second returns the max value. There are other methods you can use, like .min() and .mean().

Remember, a column of a DataFrame is actually a Series object. For this reason, you can use these same functions on the columns of nba:

>>>

>>> points = nba["pts"]
>>> type(points)
<class 'pandas.core.series.Series'>
>>> points.sum()
12976235

A DataFrame can have multiple columns, which introduces new possibilities for aggregations, like grouping:

>>>

>>> nba.groupby("fran_id", sort=False)["pts"].sum()
fran_id
Huskies           3995
Knicks          582497
Stags            20398
Falcons           3797
Capitols         22387
...

By default, Pandas sorts the group keys during the call to .groupby(). If you don’t want to sort, then pass sort=False. This parameter can lead to performance gains.

You can also group by multiple columns:

>>>

>>> nba[
...     (nba["fran_id"] == "Spurs") &
...     (nba["year_id"] > 2010)
... ].groupby(["year_id", "game_result"])["game_id"].count()
year_id  game_result
2011     L              25
         W              63
2012     L              20
         W              60
2013     L              30
         W              73
2014     L              27
         W              78
2015     L              31
         W              58
Name: game_id, dtype: int64

You can practice these basics with an exercise. Take a look at the Golden State Warriors’ 2014-15 season (year_id: 2015). How many wins and losses did they score during the regular season and the playoffs? Expand the code block below for the solution:

First, you can group by the "is_playoffs" field, then by the result:

>>>

>>> nba[
...     (nba["fran_id"] == "Warriors") &
...     (nba["year_id"] == 2015)
... ].groupby(["is_playoffs", "game_result"])["game_id"].count()
is_playoffs  game_result
0            L              15
             W              67
1            L               5
             W              16

is_playoffs=0 shows the results for the regular season, and is_playoffs=1 shows the results for the playoffs.

In the examples above, you’ve only scratched the surface of the aggregation functions that are available to you in the Pandas Python library. To see more examples of how to use them, check out Pandas GroupBy: Your Guide to Grouping Data in Python.

Manipulating Columns

You’ll need to know how to manipulate your dataset’s columns in different phases of the data analysis process. You can add and drop columns as part of the initial data cleaning phase, or later based on the insights of your analysis.

Create a copy of your original DataFrame to work with:

>>>

>>> df = nba.copy()
>>> df.shape
(126314, 23)

You can define new columns based on the existing ones:

>>>

>>> df["difference"] = df.pts - df.opp_pts
>>> df.shape
(126314, 24)

Here, you used the "pts" and "opp_pts" columns to create a new one called "difference". This new column has the same functions as the old ones:

>>>

>>> df["difference"].max()
68

Here, you used an aggregation function .max() to find the largest value of your new column.

You can also rename the columns of your dataset. It seems that "game_result" and "game_location" are too verbose, so go ahead and rename them now:

>>>

>>> renamed_df = df.rename(
...     columns={"game_result": "result", "game_location": "location"}
... )
>>> renamed_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 126314 entries, 0 to 126313
Data columns (total 24 columns):
gameorder      126314 non-null int64
...
location       126314 non-null object
result         126314 non-null object
forecast       126314 non-null float64
notes          5424 non-null object
difference     126314 non-null int64
dtypes: float64(6), int64(8), object(10)
memory usage: 23.1+ MB

Note that there’s a new object, renamed_df. Like several other data manipulation methods, .rename() returns a new DataFrame by default. If you want to manipulate the original DataFrame directly, then .rename() also provides an inplace parameter that you can set to True.

Your dataset might contain columns that you don’t need. For example, Elo ratings may be a fascinating concept to some, but you won’t analyze them in this tutorial. You can delete the four columns related to Elo:

>>>

>>> df.shape
(126314, 24)
>>> elo_columns = ["elo_i", "elo_n", "opp_elo_i", "opp_elo_n"]
>>> df.drop(elo_columns, inplace=True, axis=1)
>>> df.shape
(126314, 20)

Remember, you added the new column "difference" in a previous example, bringing the total number of columns to 24. When you remove the four Elo columns, the total number of columns drops to 20.

Specifying Data Types

When you create a new DataFrame, either by calling a constructor or reading a CSV file, Pandas assigns a data type to each column based on its values. While it does a pretty good job, it’s not perfect. If you choose the right data type for your columns upfront, then you can significantly improve your code’s performance.

Take another look at the columns of the nba dataset:

>>>

>>> df.info()

You’ll see the same output as before:

Ten of your columns have the data type object. Most of these object columns contain arbitrary text, but there are also some candidates for data type conversion. For example, take a look at the date_game column:

>>>

>>> df["date_game"] = pd.to_datetime(df["date_game"])

Here, you use .to_datetime() to specify all game dates as datetime objects.

Other columns contain text that are a bit more structured. The game_location column can have only three different values:

>>>

>>> df["game_location"].nunique()
3
>>> df["game_location"].value_counts()
A    63138
H    63138
N       38
Name: game_location, dtype: int64

Which data type would you use in a relational database for such a column? You would probably not use a varchar type, but rather an enum. Pandas provides the categorical data type for the same purpose:

>>>

>>> df["game_location"] = pd.Categorical(df["game_location"])
>>> df["game_location"].dtype
CategoricalDtype(categories=['A', 'H', 'N'], ordered=False)

categorical data has a few advantages over unstructured text. When you specify the categorical data type, you make validation easier and save a ton of memory, as Pandas will only use the unique values internally. The higher the ratio of total values to unique values, the more space savings you’ll get.

Run df.info() again. You should see that changing the game_location data type from object to categorical has decreased the memory usage.

Note: The categorical data type also gives you access to additional methods through the .cat accessor. To learn more, check out the official docs.

You’ll often encounter datasets with too many text columns. An essential skill for data scientists to have is the ability to spot which columns they can convert to a more performant data type.

Take a moment to practice this now. Find another column in the nba dataset that has a generic data type and convert it to a more specific one. You can expand the code block below to see one potential solution:

game_result can take only two different values:

>>>

>>> df["game_result"].nunique()
2
>>> df["game_result"].value_counts()
L    63157
W    63157

To improve performance, you can convert it into a categorical column:

>>>

>>> df["game_result"] = pd.Categorical(df["game_result"])

You can use df.info() to check the memory usage.

As you work with more massive datasets, memory savings becomes especially crucial. Be sure to keep performance in mind as you continue to explore your datasets.

Cleaning Data

You may be surprised to find this section so late in the tutorial! Usually, you’d take a critical look at your dataset to fix any issues before you move on to a more sophisticated analysis. However, in this tutorial, you’ll rely on the techniques that you’ve learned in the previous sections to clean your dataset.

Missing Values

Have you ever wondered why .info() shows how many non-null values a column contains? The reason why is that this is vital information. Null values often indicate a problem in the data-gathering process. They can make several analysis techniques, like different types of machine learning, difficult or even impossible.

When you inspect the nba dataset with nba.info(), you’ll see that it’s quite neat. Only the column notes contains null values for the majority of its rows:

This output shows that the notes column has only 5424 non-null values. That means that over 120,000 rows of your dataset have null values in this column.

Sometimes, the easiest way to deal with records containing missing values is to ignore them. You can remove all the rows with missing values using .dropna():

>>>

>>> rows_without_missing_data = nba.dropna()
>>> rows_without_missing_data.shape
(5424, 23)

Of course, this kind of data cleanup doesn’t make sense for your nba dataset, because it’s not a problem for a game to lack notes. But if your dataset contains a million valid records and a hundred where relevant data is missing, then dropping the incomplete records can be a reasonable solution.

You can also drop problematic columns if they’re not relevant for your analysis. To do this, use .dropna() again and provide the axis=1 parameter:

>>>

>>> data_without_missing_columns = nba.dropna(axis=1)
>>> data_without_missing_columns.shape
(126314, 22)

Now, the resulting DataFrame contains all 126,314 games, but not the sometimes empty notes column.

If there’s a meaningful default value for your use case, then you can also replace the missing values with that:

>>>

>>> data_with_default_notes = nba.copy()
>>> data_with_default_notes["notes"].fillna(
...     value="no notes at all",
...     inplace=True
... )
>>> data_with_default_notes["notes"].describe()
count              126314
unique                232
top       no notes at all
freq               120890
Name: notes, dtype: object

Here, you fill the empty notes rows with the string "no notes at all".

Invalid Values

Invalid values can be even more dangerous than missing values. Often, you can perform your data analysis as expected, but the results you get are peculiar. This is especially important if your dataset is enormous or used manual entry. Invalid values are often more challenging to detect, but you can implement some sanity checks with queries and aggregations.

One thing you can do is validate the ranges of your data. For this, .describe() is quite handy. Recall that it returns the following output:

The year_id varies between 1947 and 2015. That sounds plausible.

What about pts? How can the minimum be 0? Let’s have a look at those games:

>>>

>>> nba[nba["pts"] == 0]

This query returns a single row:

It seems the game was forfeited. Depending on your analysis, you may want to remove it from the dataset.

Inconsistent Values

Sometimes a value would be entirely realistic in and of itself, but it doesn’t fit with the values in the other columns. You can define some query criteria that are mutually exclusive and verify that these don’t occur together.

In the NBA dataset, the values of the fields pts, opp_pts and game_result should be consistent with each other. You can check this using the .empty attribute:

>>>

>>> nba[(nba["pts"] > nba["opp_pts"]) & (nba["game_result"] != 'W')].empty
True
>>> nba[(nba["pts"] < nba["opp_pts"]) & (nba["game_result"] != 'L')].empty
True

Fortunately, both of these queries return an empty DataFrame.

Be prepared for surprises whenever you’re working with raw datasets, especially if they were gathered from different sources or through a complex pipeline. You might see rows where a team scored more points than their opponent, but still didn’t win—at least, according to your dataset! To avoid situations like this, make sure you add further data cleaning techniques to your Pandas and Python arsenal.

Combining Multiple Datasets

In the previous section, you’ve learned how to clean a messy dataset. Another aspect of real-world data is that it often comes in multiple pieces. In this section, you’ll learn how to grab those pieces and combine them into one dataset that’s ready for analysis.

Earlier, you combined two Series objects into a DataFrame based on their indices. Now, you’ll take this one step further and use .concat() to combine city_data with another DataFrame. Say you’ve managed to gather some data on two more cities:

>>>

>>> further_city_data = pd.DataFrame(
...     {"revenue": [7000, 3400], "employee_count":[2, 2]},
...     index=["New York", "Barcelona"]
... )

This second DataFrame contains info on the cities "New York" and "Barcelona".

You can add these cities to city_data using .concat():

>>>

>>> all_city_data = pd.concat([city_data, further_city_data], sort=False)
>>> all_city_data
Amsterdam   4200    5.0
Tokyo       6500    8.0
Toronto     8000    NaN
New York    7000    2.0
Barcelona   3400    2.0

Now, the new variable all_city_data contains the values from both DataFrame objects.

Note: As of Pandas version 0.25.0, the sort parameter’s default value is True, but this will change to False soon. It’s good practice to provide an explicit value for this parameter to ensure that your code works consistently in different Pandas and Python versions. For more info, consult the Pandas User Guide.

By default, concat() combines along axis=0. In other words, it appends rows. You can also use it to append columns by supplying the parameter axis=1:

>>>

>>> city_countries = pd.DataFrame({
...     "country": ["Holland", "Japan", "Holland", "Canada", "Spain"],
...     "capital": [1, 1, 0, 0, 0]},
...     index=["Amsterdam", "Tokyo", "Rotterdam", "Toronto", "Barcelona"]
... )
>>> cities = pd.concat([all_city_data, city_countries], axis=1, sort=False)
>>> cities
           revenue  employee_count  country  capital
Amsterdam   4200.0             5.0  Holland      1.0
Tokyo       6500.0             8.0    Japan      1.0
Toronto     8000.0             NaN   Canada      0.0
New York    7000.0             2.0      NaN      NaN
Barcelona   3400.0             2.0    Spain      0.0
Rotterdam      NaN             NaN  Holland      0.0

Note how Pandas added NaN for the missing values. If you want to combine only the cities that appear in both DataFrame objects, then you can set the join parameter to inner:

>>>

>>> pd.concat([all_city_data, city_countries], axis=1, join="inner")
           revenue  employee_count  country  capital
Amsterdam     4200             5.0  Holland        1
Tokyo         6500             8.0    Japan        1
Toronto       8000             NaN   Canada        0
Barcelona     3400             2.0    Spain        0

While it’s most straightforward to combine data based on the index, it’s not the only possibility. You can use .merge() to implement a join operation similar to the one from SQL:

>>>

>>> countries = pd.DataFrame({
...     "population_millions": [17, 127, 37],
...     "continent": ["Europe", "Asia", "North America"]
... }, index= ["Holland", "Japan", "Canada"])
>>> pd.merge(cities, countries, left_on="country", right_index=True)

Here, you pass the parameter left_on="country" to .merge() to indicate what column you want to join on. The result is a bigger DataFrame that contains not only city data, but also the population and continent of the respective countries:

Note that the result contains only the cities where the country is known and appears in the joined DataFrame.

.merge() performs an inner join by default. If you want to include all cities in the result, then you need to provide the how parameter:

>>>

>>> pd.merge(
...     cities,
...     countries,
...     left_on="country",
...     right_index=True,
...     how="left"
... )

With this left join, you’ll see all the cities, including those without country data:

Welcome back, New York & Barcelona!

Visualizing Your Pandas DataFrame

Data visualization is one of the things that works much better in a Jupyter notebook than in a terminal, so go ahead and fire one up. If you need help getting started, then check out Jupyter Notebook: An Introduction. You can also access the Jupyter notebook that contains the examples from this tutorial by clicking the link below:

Get Jupyter Notebook: Click here to get the Jupyter Notebook you'll use to explore data with Pandas in this tutorial.

Include this line to show plots directly in the notebook:

>>>

>>> %matplotlib inline

Both Series and DataFrame objects have a .plot() method, which is a wrapper around matplotlib.pyplot.plot(). By default, it creates a line plot. Visualize how many points the Knicks scored throughout the seasons:

>>>

>>> nba[nba["fran_id"] == "Knicks"].groupby("year_id")["pts"].sum().plot()

This shows a line plot with several peaks and two notable valleys around the years 2000 and 2010:

You can also create other types of plots, like a bar plot:

>>>

>>> nba["fran_id"].value_counts().head(10).plot(kind="bar")

This will show the franchises with the most games played:

The Lakers are leading the Celtics by a minimal edge, and there are six further teams with a game count above 5000.

Now try a more complicated exercise. In 2013, the Miami Heat won the championship. Create a pie plot showing the count of their wins and losses during that season. Then, expand the code block to see a solution:

First, you define a criteria to include only the Heat’s games from 2013. Then, you create a plot in the same way as you’ve seen above:

>>>

>>> nba[
...     (nba["fran_id"] == "Heat") &
...     (nba["year_id"] == 2013)
... ]["game_result"].value_counts().plot(kind="pie")

Here’s what a champion pie looks like:

The slice of wins is significantly larger than the slice of losses!

Sometimes, the numbers speak for themselves, but often a chart helps a lot with communicating your insights. To learn more about visualizing your data, check out Interactive Data Visualization in Python With Bokeh.

Conclusion

In this tutorial, you’ve learned how to start exploring a dataset with the Pandas Python library. You saw how you could access specific rows and columns to tame even the largest of datasets. Speaking of taming, you’ve also seen multiple techniques to prepare and clean your data, by specifying the data type of columns, dealing with missing values, and more. You’ve even created queries, aggregations, and plots based on those.

Now you can:

Work with Series and DataFrame objects
Subset your data with .loc, .iloc, and the indexing operator
Answer questions with queries, grouping, and aggregation
Handle missing, invalid, and inconsistent data
Visualize your dataset in a Jupyter notebook

This journey using the NBA stats only scratches the surface of what you can do with the Pandas Python library. You can power up your project with Pandas tricks, learn techniques to speed up Pandas in Python, and even dive deep to see how Pandas works behind the scenes. There are many more features for you to discover, so get out there and tackle those datasets!

You can get all the code examples you saw in this tutorial by clicking the link below:

Get Jupyter Notebook: Click here to get the Jupyter Notebook you'll use to explore data with Pandas in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sorting Data With Python

2019-12-31T14:00:00+00:00

All programmers will have to write code to sort items or data at some point. Sorting can be critical to the user experience in your application, whether it’s ordering a user’s most recent activity by timestamp, or putting a list of email recipients in alphabetical order by last name. Python sorting functionality offers robust features to do basic sorting or customize ordering at a granular level.

In this course, you’ll learn how to sort various types of data in different data structures, customize the order, and work with two different methods of sorting in Python.

By the end of this tutorial, you’ll know how to:

Implement basic Python sorting and ordering on data structures
Differentiate between sorted() and .sort()
Customize a complex sort order in your code based on unique requirements

For this course, you’ll need a basic understanding of lists and tuples as well as sets. Those data structures will be used in this course, and some basic operations will be performed on them. Also, this course uses Python 3, so example output might vary slightly from what you’d see if you were using Python 2.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Timer Functions: Three Ways to Monitor Your Code

2019-12-30T14:00:00+00:00

While many developers recognize Python as an effective programming language, pure Python programs may run slower than their counterparts in compiled languages like C, Rust, and Java. Throughout this tutorial, you’ll see how to use a Python timer to monitor how fast your programs are running.

In this tutorial, you’ll learn how to use:

time.perf_counter() to measure time in Python
Classes to keep state
Context managers to work with a block of code
Decorators to customize a function

You’ll also gain background knowledge into how classes, context managers, and decorators work. As you see examples of each concept, you’ll be inspired to use one or several of them in your code, both for timing code execution and other applications. Each method has its advantages, and you’ll learn which to use depending on the situation. Plus, you’ll have a working Python timer that you can use to monitor your programs!

Decorators Q&A Transcript: Click here to get access to a 25-page chat log from our recent Python decorators Q&A session in the Real Python Community Slack where we discussed common decorator questions.

Python Timers

First, you’ll take a look at some example code that you’ll use throughout the tutorial. Later, you’ll add a Python timer to this code to monitor its performance. You’ll also see some of the simplest ways to measure the running time of this example.

Python Timer Functions

If you look at the built in time module in Python, then you’ll notice several functions that can measure time:

Python 3.7 introduced several new functions, like thread_time(), as well as nanosecond versions of all the functions above, named with an _ns suffix. For example, perf_counter_ns() is the nanosecond version of perf_counter(). You’ll learn more about these functions later. For now, note what the documentation has to say about perf_counter():

Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. (Source)

First, you’ll use perf_counter() to create a Python timer. Later, you’ll compare this with other Python timer functions and learn why perf_counter() is usually the best choice.

Example: Download Tutorials

To better compare the different ways you can add a Python timer to your code, you’ll apply different Python timer functions to the same code example throughout this tutorial. If you already have code you’d like to measure, then feel free to follow the examples with that instead.

The example you’ll see in this tutorial is a short function that uses the realpython-reader package to download the latest tutorials available here on Real Python. To learn more about the Real Python Reader and how it works, check out How to Publish an Open-Source Python Package to PyPI. You can install realpython-reader on your system with pip:

$ python -m pip install realpython-reader

Then, you can import the package as reader.

You’ll store the example in a file named latest_tutorial.py. The code consists of one function that downloads and prints the latest tutorial from Real Python:

 1 # latest_tutorial.py
 2 
 3 from reader import feed
 4 
 5 def main():
 6     """Download and print the latest tutorial from Real Python"""
 7     tutorial = feed.get_article(0)
 8     print(tutorial)
 9 
10 if __name__ == "__main__":
11     main()

realpython-reader handles most of the hard work:

Line 3 imports feed from realpython-reader. This module contains functionality for downloading tutorials from the Real Python feed.
Line 7 downloads the latest tutorial from Real Python. The number 0 is an offset, where 0 means the most recent tutorial, 1 is the previous tutorial, and so on.
Line 8 prints the tutorial to the console.
Line 11 calls main() when you run the script.

When you run this example, your output will typically look something like this:

$ python latest_tutorial.py
# Python Timer Functions: Three Ways to Monitor Your Code

While many developers recognize Python as an effective programming language,
pure Python programs may run slower than their counterparts in compiled
languages like C, Rust, and Java. Throughout this tutorial, you’ll see how to
use a Python timer to monitor how fast your programs are running.

[ ... The full text of the tutorial ... ]

The code may take a little while to run depending on the network, so you might want to use a Python timer to monitor the performance of the script.

Your First Python Timer

Let’s add a bare-bones Python timer to the example with time.perf_counter(). Again, this is a performance counter that’s well-suited for timing parts of your code.

perf_counter() measures the time in seconds from some unspecified moment in time, which means that the return value of a single call to the function isn’t useful. However, when you look at the difference between two calls to perf_counter(), you can figure out how many seconds passed between the two calls:

>>>

>>> import time
>>> time.perf_counter()
32311.48899951

>>> time.perf_counter()  # A few seconds later
32315.261320793

In this example, you made two calls to perf_counter() almost 4 seconds apart. You can confirm this by calculating the difference between the two outputs: 32315.26 - 32311.49 = 3.77.

You can now add a Python timer to the example code:

 1 # latest_tutorial.py
 2 
 3 import time
 4 from reader import feed
 5 
 6 def main():
 7     """Print the latest tutorial from Real Python"""
 8     tic = time.perf_counter()
 9     tutorial = feed.get_article(0)
10     toc = time.perf_counter()
11     print(f"Downloaded the tutorial in {toc - tic:0.4f} seconds")
12 
13     print(tutorial)
14 
15 if __name__ == "__main__":
16     main()

Note that you call perf_counter() both before and after downloading the tutorial. You then print the time it took to download the tutorial by calculating the difference between the two calls.

Note: In line 11, the f before the string indicates that this is an f-string, which is a convenient way to format a text string. :0.4f is a format specifier that says the number, toc - tic, should be printed as a decimal number with four decimals.

f-strings are only available in Python 3.6 and later. For more information, check out Python 3’s f-Strings: An Improved String Formatting Syntax.

Now, when you run the example, you’ll see the elapsed time before the tutorial:

$ python latest_tutorial.py
Downloaded the tutorial in 0.67 seconds
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]

That’s it! You’ve covered the basics of timing your own Python code. In the rest of the tutorial, you’ll learn how you can wrap a Python timer into a class, a context manager, and a decorator to make it more consistent and convenient to use.

A Python Timer Class

Look back at how you added the Python timer to the example above. Note that you need at least one variable (tic) to store the state of the Python timer before you download the tutorial. After staring at the code a little, you might also note that the three highlighted lines are added only for timing purposes! Now, you’ll create a class that does the same as your manual calls to perf_counter(), but in a more readable and consistent manner.

Throughout this tutorial, you’ll create and update Timer, a class that you can use to time your code in several different ways. The final code is also available on PyPI under the name codetiming. You can install this to your system like so:

$ python -m pip install codetiming

You can find more information about codetiming later on in this tutorial, in the section named The Python Timer Code.

Understanding Classes in Python

Classes are the main building blocks of object-oriented programming. A class is essentially a template that you can use to create objects. While Python doesn’t force you to program in an object-oriented manner, classes are everywhere in the language. For a quick proof, let’s investigate the time module:

>>>

>>> import time
>>> type(time)
<class 'module'>

>>> time.__class__
<class 'module'>

type() returns the type of an object. Here you can see that modules are in fact objects created from a module class. The special attribute .__class__ can be used to get access to the class that defines an object. In fact, almost everything in Python is a class:

>>>

>>> type(3)
<class 'int'>

>>> type(None)
<class 'NoneType'>

>>> type(print)
<class 'builtin_function_or_method'>

>>> type(type)
<class 'type'>

In Python, classes are great when you need to model something that needs to keep track of a particular state. In general, a class is a collection of properties (called attributes) and behaviors (called methods). For more background on classes and object-oriented programming, check out Object-Oriented Programming (OOP) in Python 3 or the official docs.

Creating a Python Timer Class

Classes are good for tracking state. In a Timer class, you want to keep track of when a timer starts and how much time has passed since then. For the first implementation of Timer, you’ll add a ._start_time attribute, as well as .start() and .stop() methods. Add the following code to a file named timer.py:

 1 # timer.py
 2 
 3 import time
 4 
 5 class TimerError(Exception):
 6     """A custom exception used to report errors in use of Timer class"""
 7 
 8 class Timer:
 9     def __init__(self):
10         self._start_time = None
11 
12     def start(self):
13         """Start a new timer"""
14         if self._start_time is not None:
15             raise TimerError(f"Timer is running. Use .stop() to stop it")
16 
17         self._start_time = time.perf_counter()
18 
19     def stop(self):
20         """Stop the timer, and report the elapsed time"""
21         if self._start_time is None:
22             raise TimerError(f"Timer is not running. Use .start() to start it")
23 
24         elapsed_time = time.perf_counter() - self._start_time
25         self._start_time = None
26         print(f"Elapsed time: {elapsed_time:0.4f} seconds")

A few different things are happening here, so let’s walk through the code step by step.

In line 5, you define a TimerError class. The (Exception) notation means that TimerError inherits from another class called Exception. Python uses this built-in class for error handling. You don’t need to add any attributes or methods to TimerError. However, having a custom error will give you more flexibility to handle problems inside Timer. For more information, check out Python Exceptions: An Introduction.

The definition of Timer itself starts on line 8. When you first create or instantiate an object from a class, your code calls the special method .__init__(). In this first version of Timer, you only initialize the ._start_time attribute, which you’ll use to track the state of your Python timer. It has the value None when the timer isn’t running. Once the timer is running, ._start_time keeps track of when the timer started.

Note: The underscore prefix of ._start_time is a Python convention. It signals that ._start_time is an internal attribute that should not be manipulated by users of the Timer class.

When you call .start() to start a new Python timer, you first check that the timer isn’t already running. Then you store the current value of perf_counter() in ._start_time. On the other hand, when you call .stop(), you first check that the Python timer is running. If it is, then you calculate the elapsed time as the difference between the current value of perf_counter() and the one you stored in ._start_time. Finally, you reset ._start_time so that the timer can be restarted, and print the elapsed time.

Here’s how you use Timer:

>>>

>>> from timer import Timer
>>> t = Timer()
>>> t.start()

>>> t.stop()  # A few seconds later
Elapsed time: 3.8191 seconds

Compare this to the earlier example where you used perf_counter() directly. The structure of the code is fairly similar, but now the code is more clear, and this is one of the benefits of using classes. By carefully choosing your class, method, and attribute names, you can make your code very descriptive!

Using the Python Timer Class

Let’s apply Timer to latest_tutorial.py. You only need to make a few changes to your previous code:

# latest_tutorial.py

from timer import Timer
from reader import feed

def main():
    """Print the latest tutorial from Real Python"""
    t = Timer()
    t.start()
    tutorial = feed.get_article(0)
    t.stop()

    print(tutorial)

if __name__ == "__main__":
    main()

Notice that the code is very similar to what you saw earlier. In addition to making the code more readable, Timer takes care of printing the elapsed time to the console, which makes the logging of time spent more consistent. When you run the code, you’ll see pretty much the same output:

$ python latest_tutorial.py
Elapsed time: 0.64 seconds
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]

Printing the elapsed time from Timer may be consistent, but it seems that this approach is not very flexible. In the next section, you’ll see how to customize your class.

Adding More Convenience and Flexibility

So far, you’ve seen that classes are suitable for when you want to encapsulate state and ensure consistent behavior in your code. In this section, you’ll add more convenience and flexibility to your Python timer:

Use adaptable text and formatting when reporting the time spent
Apply flexible logging, either to the screen, to a log file, or other parts of your program
Create a Python timer that can accumulate over several invocations
Build an informative representation of a Python timer

First, let’s see how you can customize the text used to report the time spent. In the previous code, the text f"Elapsed time: {elapsed_time:0.4f} seconds" is hard-coded into .stop(). You can add flexibility to classes using instance variables. Their values are normally passed as arguments to .__init__() and stored as self attributes. For convenience, you can also provide reasonable default values.

To add .text as a Timer instance variable, you’ll do something like this:

def __init__(self, text="Elapsed time: {:0.4f} seconds"):
    self._start_time = None
    self.text = text

Note that the default text, "Elapsed time: {:0.4f} seconds", is given as a regular string, not as an f-string. You can’t use an f-string here because they evaluate immediately, and when you instantiate Timer, your code has not yet calculated the elapsed time.

Note: If you want to use an f-string to specify .text, then you need to use double curly braces to escape the curly braces that the actual elapsed time will replace.

One example would be f"Finished {task} in {{:0.4f}} seconds". If the value of task is "reading", then this f-string would be evaluated as "Finished reading in {:0.4f} seconds".

In .stop(), you use .text as a template and .format() to populate the template:

def stop(self):
    """Stop the timer, and report the elapsed time"""
    if self._start_time is None:
        raise TimerError(f"Timer is not running. Use .start() to start it")

    elapsed_time = time.perf_counter() - self._start_time
    self._start_time = None
    print(self.text.format(elapsed_time))

After this update to timer.py, you can change the text as follows:

>>>

>>> from timer import Timer
>>> t = Timer(text="You waited {:.1f} seconds")
>>> t.start()

>>> t.stop()  # A few seconds later
You waited 4.1 seconds

Next, assume that you don’t just want to print a message to the console. Maybe you want to save your time measurements so you can store them in a database. You can do this by returning the value of elapsed_time from .stop(). Then, the calling code can choose to either ignore that return value or save it for later processing.

Perhaps you want to integrate Timer into your logging routines. To support logging or other outputs from Timer you need to change the call to print() so that the user can supply their own logging function. This can be done similar to how you customized the text earlier:

def __init__(self, text="Elapsed time: {:0.4f} seconds", logger=print):
    self._start_time = None
    self.text = text
    self.logger = logger

def stop(self):
    """Stop the timer, and report the elapsed time"""
    if self._start_time is None:
        raise TimerError(f"Timer is not running. Use .start() to start it")

    elapsed_time = time.perf_counter() - self._start_time
    self._start_time = None

    if self.logger:
        self.logger(self.text.format(elapsed_time))

    return elapsed_time

Instead of using print() directly, you create another instance variable, self.logger, that should refer to a function that takes a string as an argument. In addition to print(), you can use functions like logging.info() or .write() on file objects. Also note the if test, which allows you to turn off printing completely by passing logger=None.

Here are two examples that show the new functionality in action:

>>>

>>> from timer import Timer
>>> import logging
>>> t = Timer(logger=logging.warning)
>>> t.start()

>>> t.stop()  # A few seconds later
WARNING:root:Elapsed time: 3.1610 seconds
3.1609658249999484

>>> t = Timer(logger=None)
>>> t.start()

>>> value = t.stop()  # A few seconds later
>>> value
4.710851433001153

When you run these examples in an interactive shell, Python prints the return value automatically.

The third improvement you’ll add is the ability to accumulate time measurements. You may want to do this, for instance, when you’re calling a slow function in a loop. You’ll add a bit more functionality in the form of named timers with a dictionary that keeps track of every Python timer in your code.

Assume that you’re expanding latest_tutorial.py to a latest_tutorials.py script that downloads and prints the ten latest tutorials from Real Python. The following is one possible implementation:

# latest_tutorials.py

from timer import Timer
from reader import feed

def main():
    """Print the 10 latest tutorials from Real Python"""
    t = Timer(text="Downloaded 10 tutorials in {:0.2f} seconds")
    t.start()
    for tutorial_num in range(10):
        tutorial = feed.get_article(tutorial_num)
        print(tutorial)
    t.stop()

if __name__ == "__main__":
    main()

The code loops over the numbers from 0 to 9 and uses those as offset arguments to feed.get_article(). When you run the script, you’ll see a lot of information printed to your console:

$ python latest_tutorials.py
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of ten tutorials ... ]
Downloaded 10 tutorials in 0.67 seconds

One subtle issue with this code is that you’re not only measuring the time it takes to download the tutorials, but also the time Python spends printing the tutorials to your screen. This might not be that important since the time spent printing should be negligible compared to the time spent downloading. Still, it would be good to have a way to precisely time what you’re after in these kinds of situations.

Note: The time spent downloading ten tutorials is about the same as the time spent downloading one tutorial. This is not a bug in your code! Instead, reader caches the Real Python feed the first time get_article() is called, and reuses the information on later invocations.

There are several ways you can work around this without changing the current implementation of Timer. However, supporting this use case will be quite useful, and can be done with just a few lines of code.

First, you’ll introduce a dictionary called .timers as a class variable on Timer, which means that all instances of Timer will share it. You implement it by defining it outside any methods:

class Timer:
    timers = dict()

Class variables can be accessed either directly on the class, or through an instance of the class:

>>>

>>> from timer import Timer
>>> Timer.timers
{}

>>> t = Timer()
>>> t.timers
{}

>>> Timer.timers is t.timers
True

In both cases, the code returns the same empty class dictionary.

Next, you’ll add optional names to your Python timer. You can use the name for two different purposes:

Looking up the elapsed time later in your code
Accumulating timers with the same name

To add names to your Python timer, you need to make two more changes to timer.py. First, Timer should accept the name as a parameter. Second, the elapsed time should be added to .timers when a timer stops:

class Timer:
    timers = dict()

    def __init__(
        self,
        name=None,
        text="Elapsed time: {:0.4f} seconds",
        logger=print,
    ):
        self._start_time = None
        self.name = name
        self.text = text
        self.logger = logger

        # Add new named timers to dictionary of timers
        if name:
            self.timers.setdefault(name, 0)

    # Other methods are unchanged

    def stop(self):
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")

        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None

        if self.logger:
            self.logger(self.text.format(elapsed_time))
        if self.name:
            self.timers[self.name] += elapsed_time

        return elapsed_time

Note that you use .setdefault() when adding the new Python timer to .timers. This is a great feature that only sets the value if name is not already defined in the dictionary. If name is already used in .timers, then the value is left untouched. This allows you to accumulate several timers:

>>>

>>> from timer import Timer
>>> t = Timer("accumulate")
>>> t.start()

>>> t.stop()  # A few seconds later
Elapsed time: 3.7036 seconds
3.703554293999332

>>> t.start()

>>> t.stop()  # A few seconds later
Elapsed time: 2.3449 seconds
2.3448921170001995

>>> Timer.timers
{'accumulate': 6.0484464109995315}

You can now revisit latest_tutorials.py and make sure only the time spent on downloading the tutorials is measured:

# latest_tutorials.py

from timer import Timer
from reader import feed

def main():
    """Print the 10 latest tutorials from Real Python"""
    t = Timer("download", logger=None)
    for tutorial_num in range(10):
        t.start()
        tutorial = feed.get_article(tutorial_num)
        t.stop()
        print(tutorial)

    download_time = Timer.timers["download"]
    print(f"Downloaded 10 tutorials in {download_time:0.2f} seconds")

if __name__ == "__main__":
    main()

Rerunning the script will give similar output as earlier, although now you are only timing the actual download of the tutorials:

$ python latest_tutorials.py
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of ten tutorials ... ]
Downloaded 10 tutorials in 0.65 seconds

The final improvement that you’ll make to Timer is to make it more informative when you’re working with it interactively. Try the following:

>>>

>>> from timer import Timer
>>> t = Timer()
>>> t
<timer.Timer object at 0x7f0578804320>

That last line is the default way Python represents objects. While you can glean some information from it, it’s usually not very useful. Instead, it would be nice to see things like the name of the Timer, or how it will report on the timings.

In Python 3.7, data classes were added to the standard library. These provide several conveniences to your classes, including a more informative representation string.

Note: Data classes are included in Python only for version 3.7 and later. However, there is a backport available on PyPI for Python 3.6.

You can install it using pip:

$ python -m pip install dataclasses

See The Ultimate Guide to Data Classes in Python 3.7 for more information.

You convert your Python timer to a data class using the @dataclass decorator. You’ll learn more about decorators later in this tutorial. For now, you can think of this as a notation that tells Python that Timer is a data class:

 1 from dataclasses import dataclass, field
 2 from typing import Any, ClassVar
 3 
 4 @dataclass
 5 class Timer:
 6     timers: ClassVar = dict()
 7     name: Any = None
 8     text: Any = "Elapsed time: {:0.4f} seconds"
 9     logger: Any = print
10     _start_time: Any = field(default=None, init=False, repr=False)
11 
12     def __post_init__(self):
13         """Initialization: add timer to dict of timers"""
14         if self.name:
15             self.timers.setdefault(self.name, 0)
16 
17     # The rest of the code is unchanged

This code replaces your earlier .__init__() method. Note how data classes use syntax that looks similar to the class variable syntax you saw earlier for defining all variables. In fact, .__init__() is created automatically for data classes, based on annotated variables in the definition of the class.

You need to annotate your variables to use a data class. You can use this to add type hints to your code. If you don’t want to use type hints, then you can instead annotate all variables with Any, just like you did above. You’ll soon see how to add actual type hints to your data class.

Here are a few notes about the Timer data class:

Line 4: The @dataclass decorator defines Timer to be a data class.
Line 6: The special ClassVar annotation is necessary for data classes to specify that .timers is a class variable.
Lines 7 to 9: .name, .text, and .logger will be defined as attributes on Timer, whose values can be specified when creating Timer instances. They all have the given default values.
Line 10: Recall that ._start_time is a special attribute that’s used to keep track of the state of the Python timer, but should be hidden from the user. Using dataclasses.field() you say that ._start_time should be removed from .__init__() and the representation of Timer.
Lines 12 to 15: You can use the special .__post_init__() method for any initialization you need to do apart from setting the instance attributes. Here, you use it to add named timers to .timers.

Your new Timer data class works just like your previous regular class, except that it now has a nice representation:

>>>

>>> from timer import Timer
>>> t = Timer()
>>> t
Timer(name=None, text='Elapsed time: {:0.4f} seconds',
      logger=<built-in function print>)

>>> t.start()

>>> t.stop()  # A few seconds later
Elapsed time: 6.7197 seconds
6.719705373998295

Now you have a pretty neat version of Timer that’s consistent, flexible, convenient, and informative! Many of the improvements you’ve seen in this section can be applied to other types of classes in your projects as well.

Before ending this section, let’s have a look at the complete source code of Timer as it currently stands. You’ll notice the addition of type hints to the code for extra documentation:

# timer.py

from dataclasses import dataclass, field
import time
from typing import Callable, ClassVar, Dict, Optional

class TimerError(Exception):
    """A custom exception used to report errors in use of Timer class"""

@dataclass
class Timer:
    timers: ClassVar[Dict[str, float]] = dict()
    name: Optional[str] = None
    text: str = "Elapsed time: {:0.4f} seconds"
    logger: Optional[Callable[[str], None]] = print
    _start_time: Optional[float] = field(default=None, init=False, repr=False)

    def __post_init__(self) -> None:
        """Add timer to dict of timers after initialization"""
        if self.name is not None:
            self.timers.setdefault(self.name, 0)

    def start(self) -> None:
        """Start a new timer"""
        if self._start_time is not None:
            raise TimerError(f"Timer is running. Use .stop() to stop it")

        self._start_time = time.perf_counter()

    def stop(self) -> float:
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")

        # Calculate elapsed time
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None

        # Report elapsed time
        if self.logger:
            self.logger(self.text.format(elapsed_time))
        if self.name:
            self.timers[self.name] += elapsed_time

        return elapsed_time

Using a class to create a Python timer has several benefits:

Readability: Your code will read more naturally if you carefully choose class and method names.
Consistency: Your code will be easier to use if you encapsulate properties and behaviors into attributes and methods.
Flexibility: Your code will be reusable if you use attributes with default values instead of hardcoded values.

This class is very flexible, and you can use it in almost any situation where you want to monitor the time it takes for code to run. However, in the next sections, you’ll learn about using context managers and decorators, which will be more convenient for timing code blocks and functions.

A Python Timer Context Manager

Your Python Timer class has come a long way! Compared with the first Python timer you created, your code has gotten quite powerful. However, there’s still a bit of boilerplate code necessary to use your Timer:

First, instantiate the class.
Call .start() before the code block that you want to time.
Call .stop() after the code block.

Luckily, Python has a unique construct for calling functions before and after a block of code: the context manager. In this section, you’ll learn what context managers are and how you can create your own. Then you’ll see how to expand Timer so that it can work as a context manager as well. Finally, you’ll see how using Timer as a context manager can simplify your code.

Understanding Context Managers in Python

Context managers have been a part of Python for a long time. They were introduced by PEP 343 in 2005, and first implemented in Python 2.5. You can recognize context managers in code by the use of the with keyword:

with EXPRESSION as VARIABLE:
    BLOCK

EXPRESSION is some Python expression that returns a context manager. The context manager is optionally bound to the name VARIABLE. Finally, BLOCK is any regular Python code block. The context manager will guarantee that your program calls some code before BLOCK and some other code after BLOCK executes. The latter will happen, even if BLOCK raises an exception.

The most common use of context managers is probably handling different resources, like files, locks, and database connections. The context manager is then used to free and clean up the resource after you’ve used it. The following example reveals the fundamental structure of timer.py by only printing lines that contain a colon. More importantly, it shows the common idiom for opening a file in Python:

>>>

>>> with open("timer.py") as fp:
...     print("".join(ln for ln in fp if ":" in ln))
...
class TimerError(Exception):
class Timer:
    timers: ClassVar[Dict[str, float]] = dict()
    name: Optional[str] = None
    text: str = "Elapsed time: {:0.4f} seconds"
    logger: Optional[Callable[[str], None]] = print
    _start_time: Optional[float] = field(default=None, init=False, repr=False)
    def __post_init__(self) -> None:
        if self.name is not None:
    def start(self) -> None:
        if self._start_time is not None:
    def stop(self) -> float:
        if self._start_time is None:
        if self.logger:
        if self.name:

Note that fp, the file pointer, is never explicitly closed because you used open() as a context manager. You can confirm that fp has closed automatically:

>>>

>>> fp.closed
True

In this example, open("timer.py") is an expression that returns a context manager. That context manager is bound to the name fp. The context manager is in effect during the execution of print(). This one-line code block executes in the context of fp.

What does it mean that fp is a context manager? Technically, it means that fp implements the context manager protocol. There are many different protocols underlying the Python language. You can think of a protocol as a contract that states what specific methods your code must implement.

The context manager protocol consists of two methods:

Call .__enter__() when entering the context related to the context manager.
Call .__exit__() when exiting the context related to the context manager.

In other words, to create a context manager yourself, you need to write a class that implements .__enter__() and .__exit__(). No more, no less. Let’s try a Hello, World! context manager example:

# greeter.py

class Greeter:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print(f"Hello {self.name}")
        return self

    def __exit__(self, exc_type, exc_value, exc_tb):
        print(f"See you later, {self.name}")

Greeter is a context manager because it implements the context manager protocol. You can use it like this:

>>>

>>> from greeter import Greeter
>>> with Greeter("Nick"):
...     print("Doing stuff ...")
...
Hello Nick
Doing stuff ...
See you later, Nick

First, note how .__enter__() is called before you’re doing stuff, while .__exit__() is called after. In this simplified example, you’re not referencing the context manager. In such cases, you don’t need to give the context manager a name with as.

Next, notice how .__enter__() returns self. The return value of .__enter__() is what is bound by as. You usually want to return self from .__enter__() when creating context managers. You can use that return value as follows:

>>>

>>> from greeter import Greeter
>>> with Greeter("Emily") as grt:
...     print(f"{grt.name} is doing stuff ...")
...
Hello Emily
Emily is doing stuff ...
See you later, Emily

Finally, .__exit__() takes three arguments: exc_type, exc_value, and exc_tb. These are used for error handling within the context manager, and mirror the return values of sys.exc_info(). If an exception happens while the block is being executed, then your code calls .__exit__() with the type of the exception, an exception instance, and a traceback object. Often, you can ignore these in your context manager, in which case .__exit__() is called before the exception is reraised:

>>>

>>> from greeter import Greeter
>>> with Greeter("Rascal") as grt:
...     print(f"{grt.age} does not exist")
...
Hello Rascal
See you later, Rascal
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
AttributeError: 'Greeter' object has no attribute 'age'

You can see that "See you later, Rascal" is printed, even though there is an error in the code.

Now you know what context managers are and how you can create your own. If you want to dive deeper, then check out contextlib in the standard library. It includes convenient ways for defining new context managers, as well as ready-made context managers that can be used to close objects, suppress errors, or even do nothing! For even more information, check out Python Context Managers and the “with” Statement and the accompanying tutorial.

Creating a Python Timer Context Manager

You’ve seen how context managers work in general, but how can they help with timing code? If you can run certain functions before and after a block of code, then you can simplify how your Python timer works. So far, you’ve needed to call .start() and .stop() explicitly when timing your code, but a context manager can do this automatically.

Again, for Timer to work as a context manager, it needs to adhere to the context manager protocol. In other words, it must implement .__enter__() and .__exit__() to start and stop the Python timer. All the necessary functionality is already available, so there’s not much new code you need to write. Just add the following methods to your Timer class:

def __enter__(self):
    """Start a new timer as a context manager"""
    self.start()
    return self

def __exit__(self, *exc_info):
    """Stop the context manager timer"""
    self.stop()

Timer is now a context manager. The important part of the implementation is that .__enter__() calls .start() to start a Python timer when the context is entered, and .__exit__() uses .stop() to stop the Python timer when the code leaves the context. Try it out:

>>>

>>> from timer import Timer
>>> import time
>>> with Timer():
...     time.sleep(0.7)
...
Elapsed time: 0.7012 seconds

You should also note two more subtle details:

.__enter__() returns self, the Timer instance, which allows the user to bind the Timer instance to a variable using as. For example, with Timer() as t: will create the variable t pointing to the Timer object.
.__exit__() expects a triple of arguments with information about any exception that occurred during the execution of the context. In your code, these arguments are packed into a tuple called exc_info and then ignored, which means that Timer will not attempt any exception handling.

.__exit__() doesn’t do any error handling in this case. Still, one of the great features of context managers is that they’re guaranteed to call .__exit__(), no matter how the context exits. In the following example, you purposely create an error by dividing by zero:

>>>

>>> from timer import Timer
>>> with Timer():
...     for num in range(-3, 3):
...         print(f"1 / {num} = {1 / num:.3f}")
...
1 / -3 = -0.333
1 / -2 = -0.500
1 / -1 = -1.000
Elapsed time: 0.0001 seconds
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ZeroDivisionError: division by zero

Note that Timer prints out the elapsed time, even though the code crashed. It’s possible to inspect and suppress errors in .__exit__(). See the documentation for more information.

Using the Python Timer Context Manager

Let’s see how to use the Timer context manager to time the download of Real Python tutorials. Recall how you used Timer earlier:

# latest_tutorial.py

from timer import Timer
from reader import feed

def main():
    """Print the latest tutorial from Real Python"""
    t = Timer()
    t.start()
    tutorial = feed.get_article(0)
    t.stop()

    print(tutorial)

if __name__ == "__main__":
    main()

You’re timing the call to feed.get_article(). You can use the context manager to make the code shorter, simpler, and more readable:

# latest_tutorial.py

from timer import Timer
from reader import feed

def main():
    """Print the latest tutorial from Real Python"""
    with Timer():
        tutorial = feed.get_article(0)

    print(tutorial)

if __name__ == "__main__":
    main()

This code does virtually the same as the code above. The main difference is that you don’t define the extraneous variable t, which keeps your namespace cleaner.

Running the script should give a familiar result:

$ python latest_tutorial.py
Elapsed time: 0.71 seconds
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]

There are a few advantages to adding context manager capabilities to your Python timer class:

Low effort: You only need one extra line of code to time the execution of a block of code.
Readability: Invoking the context manager is readable, and you can more clearly visualize the code block you’re timing.

Using Timer as a context manager is almost as flexible as using .start() and .stop() directly, while it has less boilerplate code. In the next section, you’ll see how Timer can be used as a decorator as well. This will make it easier to monitor the runtime of complete functions.

A Python Timer Decorator

Your Timer class is now very versatile. However, there’s one use case where it could be even more streamlined. Say that you want to track the time spent inside one given function in your codebase. Using a context manager, you have essentially two different options:

Use Timer every time you call the function:
```
with Timer("some_name"):
    do_something()
```
If you call do_something() in many places, then this will become cumbersome and hard to maintain.
Wrap the code in your function inside a context manager:
```
def do_something():
    with Timer("some_name"):
        ...
```
The Timer only needs to be added in one place, but this adds a level of indentation to the whole definition of do_something().

A better solution is to use Timer as a decorator. Decorators are powerful constructs that you use to modify the behavior of functions and classes. In this section, you’ll learn a little about how decorators work, how Timer can be extended to be a decorator, and how that will simplify timing functions. For a more in-depth explanation of decorators, see Primer on Python Decorators.

Understanding Decorators in Python

A decorator is a function that wraps another function to modify its behavior. This technique is possible because functions are first-class objects in Python. In other words, functions can be assigned to variables and used as arguments to other functions, just like any other object. This gives you a lot of flexibility and is the basis for several of Python’s more powerful features.

As a first example, you’ll create a decorator that does nothing:

def turn_off(func):
    return lambda *args, **kwargs: None

First, note that turn_off() is just a regular function. What makes this a decorator is that it takes a function as its only argument and returns a function. You can use this to modify other functions like this:

>>>

>>> print("Hello")
Hello

>>> print = turn_off(print)
>>> print("Hush")
>>> # Nothing is printed

The line print = turn_off(print) decorates the print statement with the turn_off() decorator. Effectively, it replaces print() with lambda *args, **kwargs: None returned by turn_off(). The lambda statement represents an anonymous function that does nothing except return None.

For you to define more interesting decorators, you need to know about inner functions. An inner function is a function defined inside another function. One common use of inner functions is to create function factories:

def create_multiplier(factor):
    def multiplier(num):
        return factor * num
    return multiplier

multiplier() is an inner function, defined inside create_multiplier(). Note that you have access to factor inside multiplier(), while multiplier() is not defined outside create_multiplier():

>>>

>>> multiplier
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'multiplier' is not defined

Instead you use create_multiplier() to create new multiplier functions, each based on a different factor:

>>>

>>> double = create_multiplier(factor=2)
>>> double(3)
6

>>> quadruple = create_multiplier(factor=4)
>>> quadruple(7)
28

Similarly, you can use inner functions to create decorators. Remember, a decorator is a function that returns a function:

 1 def triple(func):
 2     def wrapper_triple(*args, **kwargs):
 3         print(f"Tripled {func.__name__!r}")
 4         value = func(*args, **kwargs)
 5         return value * 3
 6     return wrapper_triple

triple() is a decorator, because it’s a function that expects a function as it’s only argument, func(), and returns another function, wrapper_triple(). Note the structure of triple() itself:

Line 1 starts the definition of triple() and expects a function as an argument.
Lines 2 to 5 define the inner function wrapper_triple().
Line 6 returns wrapper_triple().

This pattern is prevalent for defining decorators. The interesting parts are those happening inside the inner function:

Line 2 starts the definition of wrapper_triple(). This function will replace whichever function triple() decorates. The parameters are *args and **kwargs, which collect whichever positional and keyword arguments you pass to the function. This gives you the flexibility to use triple() on any function.
Line 3 prints out the name of the decorated function, and note that triple() has been applied to it.
Line 4 calls func(), the function that has been decorated by triple(). It passes on all arguments passed to wrapper_triple().
Line 5 triples the return value of func() and returns it.

Let’s try it out! knock() is a function that returns the word Penny. See what happens if it’s tripled:

>>>

>>> def knock():
...     return "Penny! "
...
>>> knock = triple(knock)
>>> result = knock()
Tripled 'knock'

>>> result
'Penny! Penny! Penny! '

Multiplying a text string by a number is a form of repetition, so Penny repeats three times. The decoration happens at knock = triple(knock).

It feels a bit clunky to keep repeating knock. Instead, PEP 318 introduced a more convenient syntax for applying decorators. The following definition of knock() does the same as the one above:

>>>

>>> @triple
... def knock():
...     return "Penny! "
...
>>> result = knock()
Tripled 'knock'

>>> result
'Penny! Penny! Penny! '

The @ symbol is used to apply decorators. In this case, @triple means that triple() is applied to the function defined just after it.

One of the few decorators defined in the standard library is @functools.wraps. This one is quite helpful when defining your own decorators. Since decorators effectively replace one function with another, they create a subtle issue with your functions:

>>>

>>> knock
<function triple.<locals>.wrapper_triple at 0x7fa3bfe5dd90>

@triple decorates knock(), which is then replaced by the wrapper_triple() inner function, as the output above confirms. This will also replace the name, docstring, and other metadata. Often, this will not have much effect, but it can make introspection difficult.

Sometimes, decorated functions must have correct metadata. @functools.wraps fixes exactly this issue:

import functools

def triple(func):
    @functools.wraps(func)
    def wrapper_triple(*args, **kwargs):
        print(f"Tripled {func.__name__!r}")
        value = func(*args, **kwargs)
        return value * 3
    return wrapper_triple

With this new definition of @triple, metadata are preserved:

>>>

>>> @triple
... def knock():
...     return "Penny! "
...
>>> knock
<function knock at 0x7fa3bfe5df28>

Note that knock() now keeps its proper name, even after being decorated. It’s good form to use @functools.wraps whenever you define a decorator. A blueprint you can use for most of your decorators is the following:

import functools

def decorator(func):
    @functools.wraps(func)
    def wrapper_decorator(*args, **kwargs):
        # Do something before
        value = func(*args, **kwargs)
        # Do something after
        return value
    return wrapper_decorator

To see more examples of how to define decorators, check out the examples listed in Primer on Python Decorators.

Creating a Python Timer Decorator

In this section, you’ll learn how to extend your Python timer so that you can use it as a decorator as well. However, as a first exercise, let’s create a Python timer decorator from scratch.

Based on the blueprint above, you only need to decide what to do before and after you call the decorated function. This is similar to the considerations about what to do when entering and exiting the context manager. You want to start a Python timer before calling the decorated function, and stop the Python timer after the call finishes. A @timer decorator can be defined as follows:

import functools
import time

def timer(func):
    @functools.wraps(func)
    def wrapper_timer(*args, **kwargs):
        tic = time.perf_counter()
        value = func(*args, **kwargs)
        toc = time.perf_counter()
        elapsed_time = toc - tic
        print(f"Elapsed time: {elapsed_time:0.4f} seconds")
        return value
    return wrapper_timer

Note how much wrapper_timer() resembles the early pattern you established for timing Python code. You can apply @timer as follows:

>>>

>>> @timer
... def latest_tutorial():
...     tutorial = feed.get_article(0)
...     print(tutorial)
...
>>> latest_tutorial()
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]
Elapsed time: 0.5414 seconds

Recall that you can also apply a decorator to a previously defined function:

>>>

>>> feed.get_article = timer(feed.get_article)

Since @ applies when functions are defined, you need to use the more basic form in these cases. One advantage of using a decorator is that you only need to apply it once, and it’ll time the function every time:

>>>

>>> tutorial = feed.get_article(0)
Elapsed time: 0.5512 seconds

@timer does the job. However, in a sense, you’re back to square one, since @timer does not have any of the flexibility or convenience of Timer. Can you also make your Timer class act like a decorator?

So far, you’ve used decorators as functions applied to other functions, but that’s not entirely correct. Decorators must be callables. There are many callable types in Python. You can make your own objects callable by defining the special .__call__() method in their class. The following function and class behave similarly:

>>>

>>> def square(num):
...     return num ** 2
...
>>> square(4)
16

>>> class Squarer:
...     def __call__(self, num):
...         return num ** 2
...
>>> square = Squarer()
>>> square(4)
16

Here, square is an instance that is callable and can square numbers, just like the square() function in the first example.

This gives you a way of adding decorator capabilities to the existing Timer class:

def __call__(self, func):
    """Support using Timer as a decorator"""
    @functools.wraps(func)
    def wrapper_timer(*args, **kwargs):
        with self:
            return func(*args, **kwargs)

    return wrapper_timer

.__call__() uses the fact that Timer is already a context manager to take advantage of the conveniences you’ve already defined there. Make sure you also import functools at the top of timer.py.

You can now use Timer as a decorator:

>>>

>>> @Timer(text="Downloaded the tutorial in {:.2f} seconds")
... def latest_tutorial():
...     tutorial = feed.get_article(0)
...     print(tutorial)
...
>>> latest_tutorial()
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]
Downloaded the tutorial in 0.72 seconds

Before rounding out this section, know that there’s a more straightforward way of turning your Python timer into a decorator. You’ve already seen some of the similarities between context managers and decorators. They’re both typically used to do something before and after executing some given code.

Based on these similarities, there’s a mixin class defined in the standard library called ContextDecorator. You can add decorator abilities to your context manager classes simply by inheriting ContextDecorator:

from contextlib import ContextDecorator

class Timer(ContextDecorator):
    # Implementation of Timer is unchanged

When you use ContextDecorator this way, there’s no need to implement .__call__() yourself, so you can safely delete it from the Timer class.

Using the Python Timer Decorator

Let’s redo the latest_tutorial.py example one last time, using the Python timer as a decorator:

 1 # latest_tutorial.py
 2 
 3 from timer import Timer
 4 from reader import feed
 5 
 6 @Timer()
 7 def main():
 8     """Print the latest tutorial from Real Python"""
 9     tutorial = feed.get_article(0)
10     print(tutorial)
11 
12 if __name__ == "__main__":
13     main()

If you compare this implementation with the original implementation without any timing, then you’ll notice that the only differences are the import of Timer on line 3 and the application of @Timer() on line 6. A significant advantage of using decorators is that they’re usually straightforward to apply, as you see here.

However, the decorator still applies to the whole function. This means your code is taking into account the time it takes to print the tutorial, in addition to the time it takes to download. Let’s run the script one final time:

$ python latest_tutorial.py
# Python Timer Functions: Three Ways to Monitor Your Code

[ ... The full text of the tutorial ... ]
Elapsed time: 0.69 seconds

The location of the elapsed time output is a tell-tale sign that your code is considering the time it takes to print time as well. As you see here, your code prints the elapsed time after the tutorial.

When you use Timer as a decorator, you’ll see similar advantages as you did with context managers:

Low effort: You only need one extra line of code to time the execution of a function.
Readability: When you add the decorator, you can note more clearly that your code will time the function.
Consistency: You only need to add the decorator when the function is defined. Your code will consistently time it every time it’s called.

However, decorators are not as flexible as context managers. You can only apply them to complete functions. It’s possible to add decorators to already defined functions, but this is a bit clunky and less common.

The Python Timer Code

You can expand the code block below to view the final source code for your Python timer:

# timer.py

from contextlib import ContextDecorator
from dataclasses import dataclass, field
import time
from typing import Any, Callable, ClassVar, Dict, Optional

class TimerError(Exception):
    """A custom exception used to report errors in use of Timer class"""

@dataclass
class Timer(ContextDecorator):
    """Time your code using a class, context manager, or decorator"""

    timers: ClassVar[Dict[str, float]] = dict()
    name: Optional[str] = None
    text: str = "Elapsed time: {:0.4f} seconds"
    logger: Optional[Callable[[str], None]] = print
    _start_time: Optional[float] = field(default=None, init=False, repr=False)

    def __post_init__(self) -> None:
        """Initialization: add timer to dict of timers"""
        if self.name:
            self.timers.setdefault(self.name, 0)

    def start(self) -> None:
        """Start a new timer"""
        if self._start_time is not None:
            raise TimerError(f"Timer is running. Use .stop() to stop it")

        self._start_time = time.perf_counter()

    def stop(self) -> float:
        """Stop the timer, and report the elapsed time"""
        if self._start_time is None:
            raise TimerError(f"Timer is not running. Use .start() to start it")

        # Calculate elapsed time
        elapsed_time = time.perf_counter() - self._start_time
        self._start_time = None

        # Report elapsed time
        if self.logger:
            self.logger(self.text.format(elapsed_time))
        if self.name:
            self.timers[self.name] += elapsed_time

        return elapsed_time

    def __enter__(self) -> "Timer":
        """Start a new timer as a context manager"""
        self.start()
        return self

    def __exit__(self, *exc_info: Any) -> None:
        """Stop the context manager timer"""
        self.stop()

The code is also available in the codetiming repository on GitHub.

You can use the code yourself by saving it to a file named timer.py and importing it into your program:

>>>

>>> from timer import Timer

Timer is also available on PyPI, so an even easier option is to install it using pip:

$ python -m pip install codetiming

Note that the package name on PyPI is codetiming. You’ll need to use this name both when you install the package and when you import Timer:

>>>

>>> from codetiming import Timer

Apart from this, codetiming.Timer works exactly as timer.Timer. To summarize, you can use Timer in three different ways:

As a class:

t = Timer(name="class")
t.start()
# Do something
t.stop()

As a context manager:

with Timer(name="context manager"):
    # Do something

As a decorator:

@Timer(name="decorator")
def stuff():
    # Do something

This kind of Python timer is mainly useful for monitoring the time your code spends at individual key code blocks or functions. In the next section, you’ll get a quick overview of alternatives you can use if you’re looking to optimize your code.

Other Python Timer Functions

There are many options for timing your code with Python. In this tutorial, you’ve learned how to create a flexible and convenient class that you can use in several different ways. A quick search on PyPI shows that there are already many projects available that offer Python timer solutions.

In this section, you’ll first learn more about the different functions available in the standard library for measuring time, and why perf_counter() is preferable. Then, you’ll see alternatives for optimizing your code, for which Timer is not well-suited.

Using Alternative Python Timer Functions

You’ve been using perf_counter() throughout this tutorial to do the actual time measurements, but Python’s time library comes with several other functions that also measure time. Here are some alternatives:

One of the reasons why there are several functions is that Python represents time as a float. Floating-point numbers are inaccurate by nature. You may have seen results like these before:

>>>

>>> 0.1 + 0.1 + 0.1
0.30000000000000004

>>> 0.1 + 0.1 + 0.1 == 0.3
False

Python’s float follows the IEEE 754 Standard for Floating-Point Arithmetic, which tries to represent all floating-point numbers in 64 bits. Since there are infinitely many floating-point numbers, you can’t express them all with a finite number of bits.

IEEE 754 prescribes a system where the density of numbers that you can represent varies. The closer you are to 1, the more numbers you can represent. For larger numbers, there’s more space between the numbers that you can express. This has some consequences when you use a float to represent time.

Consider time(). The main purpose of this function is to represent the actual time right now. It does this as the number of seconds since a given point in time, called the epoch. The number returned by time() is quite big, which means that there are fewer numbers available, and the resolution suffers. Specifically, time() is not able to measure nanosecond differences:

>>>

>>> import time
>>> t = time.time()
>>> t
1564342757.0654016

>>> t + 1e-9
1564342757.0654016

>>> t == t + 1e-9
True

A nanosecond is one-billionth of a second. Note that adding a nanosecond to t does not affect the result. perf_counter(), on the other hand, uses some undefined point in time as its epoch, allowing it to work with smaller numbers and therefore obtain a better resolution:

>>>

>>> import time
>>> p = time.perf_counter()
>>> p
11370.015653846

>>> p + 1e-9
11370.015653847

>>> p == p + 1e-9
False

Here, you see that adding a nanosecond to p actually affects the outcome. For more information about how to work with time(), see A Beginner’s Guide to the Python time Module.

The challenges with representing time as a float are well known, so Python 3.7 introduced a new option. Each time measurement function now has a corresponding _ns function that returns the number of nanoseconds as an int instead of the number of seconds as a float. For instance, time() now has a nanosecond counterpart called time_ns():

>>>

>>> import time
>>> time.time_ns()
1564342792866601283

Integers are unbounded in Python, so this allows time_ns() to give nanosecond resolution for all eternity. Similarly, perf_counter_ns() is a nanosecond variant of perf_counter():

>>>

>>> import time
>>> time.perf_counter()
13580.153084446

>>> time.perf_counter_ns()
13580765666638

Since perf_counter() already provides nanosecond resolution, there are fewer advantages to using perf_counter_ns().

Note: perf_counter_ns() is only available in Python 3.7 and later. In this tutorial, you’ve used perf_counter() in your Timer class. That way, Timer can be used on older Python versions as well.

For more information about the _ns functions in time, check out Cool New Features in Python 3.7.

There are two functions in time that do not measure the time spent sleeping. These are process_time() and thread_time(), which are useful in some settings. However, for Timer, you typically want to measure the full time spent. The final function in the list above is monotonic(). The name alludes to this function being a monotonic timer, which is a Python timer that can never move backward.

All these functions are monotonic except time(), which can go backward if the system time is adjusted. On some systems, monotonic() is the same function as perf_counter(), and you can use them interchangeably. However, this is not always the case. You can use time.get_clock_info() to get more information about a Python timer function. Using Python 3.7 on Linux I get the following information:

>>>

>>> import time
>>> time.get_clock_info("monotonic")
namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)',
          monotonic=True, resolution=1e-09)

>>> time.get_clock_info("perf_counter")
namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)',
          monotonic=True, resolution=1e-09)

The results could be different on your system.

PEP 418 describes some of the rationale behind introducing these functions. It includes the following short descriptions:

time.monotonic(): timeout and scheduling, not affected by system clock updates

time.perf_counter(): benchmarking, most precise clock for short period

time.process_time(): profiling, CPU time of the process (Source)

As you can see, it’s usually the best choice for you to use perf_counter() for your Python timer.

Estimating Running Time With `timeit`

Say you’re trying to squeeze the last bit of performance out of your code, and you’re wondering about the most effective way to convert a list to a set. You want to compare using set() and the set literal, {...}. You can use your Python timer for this:

>>>

>>> from timer import Timer
>>> numbers = [7, 6, 1, 4, 1, 8, 0, 6]
>>> with Timer(text="{:.8f}"):
...     set(numbers)
...
{0, 1, 4, 6, 7, 8}
0.00007373

>>> with Timer(text="{:.8f}"):
...     {*numbers}
...
{0, 1, 4, 6, 7, 8}
0.00006204

This test seems to indicate that the set literal might be slightly faster. However, these results are quite uncertain, and if you rerun the code, you might get wildly different results. That’s because you’re only trying the code once. You could, for instance, get unlucky and run the script just as your computer is becoming busy with other tasks.

A better way is to use the timeit standard library. It’s designed precisely to measure the execution time of small code snippets. While you can import and call timeit.timeit() from Python as a regular function, it is usually more convenient to use the command-line interface. You can time the two variants as follows:

$ python -m timeit --setup "nums = [7, 6, 1, 4, 1, 8, 0, 6]" "set(nums)"
2000000 loops, best of 5: 163 nsec per loop

$ python -m timeit --setup "nums = [7, 6, 1, 4, 1, 8, 0, 6]" "{*nums}"
2000000 loops, best of 5: 121 nsec per loop

timeit automatically calls your code many times to average out noisy measurements. The results from timeit confirm that the set literal is faster than set(). You can find more information about this particular issue at Michael Bassili’s blog.

Note: Be careful when you’re using timeit on code that can download files or access databases. Since timeit automatically calls your program several times, you could unintentionally end up spamming the server with requests!

Finally, the IPython interactive shell and the Jupyter notebook have extra support for this functionality with the %timeit magic command:

>>>

In [1]: numbers = [7, 6, 1, 4, 1, 8, 0, 6]

In [2]: %timeit set(numbers)
171 ns ± 0.748 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [3]: %timeit {*numbers}
147 ns ± 2.62 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Again, the measurements indicate that using a set literal is faster. In Jupyter Notebooks you can also use the %%timeit cell-magic to measure the time of running a whole cell.

Finding Bottlenecks in Your Code With Profilers

timeit is excellent for benchmarking a particular snippet of code. However, it would be very cumbersome to use it to check all parts of your program and locate which sections take the most time. Instead, you can use a profiler.

cProfile is a profiler that you can access at any time from the standard library. You can use it in several ways, although it’s usually most straightforward to use it as a command-line tool:

$ python -m cProfile -o latest_tutorial.prof latest_tutorial.py

This command runs latest_tutorial.py with profiling turned on. You save the output from cProfile in latest_tutorial.prof, as specified by the -o option. The output data is in a binary format that needs a dedicated program to make sense of it. Again, Python has an option right in the standard library! Runnin the pstats module on your .prof file opens an interactive profile statistics browser:

$ python -m pstats latest_tutorial.prof
Welcome to the profile statistics browser.
latest_tutorial.prof% help

Documented commands (type help <topic>):
========================================
EOF  add  callees  callers  help  quit  read  reverse  sort  stats  strip

To use pstats you type commands at the prompt. Here you can see the integrated help system. Typically you’ll use the sort and stats commands. To get a cleaner output, strip can be useful:

latest_tutorial.prof% strip
latest_tutorial.prof% sort cumtime
latest_tutorial.prof% stats 10
         1393801 function calls (1389027 primitive calls) in 0.586 seconds

 Ordered by: cumulative time
 List reduced from 1443 to 10 due to restriction <10>

 ncalls tottime percall cumtime percall filename:lineno(function)
  144/1   0.001   0.000   0.586   0.586 {built-in method builtins.exec}
      1   0.000   0.000   0.586   0.586 latest_tutorial.py:3(<module>)
      1   0.000   0.000   0.521   0.521 contextlib.py:71(inner)
      1   0.000   0.000   0.521   0.521 latest_tutorial.py:6(read_latest_tutorial)
      1   0.000   0.000   0.521   0.521 feed.py:28(get_article)
      1   0.000   0.000   0.469   0.469 feed.py:15(_feed)
      1   0.000   0.000   0.469   0.469 feedparser.py:3817(parse)
      1   0.000   0.000   0.271   0.271 expatreader.py:103(parse)
      1   0.000   0.000   0.271   0.271 xmlreader.py:115(parse)
     13   0.000   0.000   0.270   0.021 expatreader.py:206(feed)

This output shows that the total runtime was 0.586 seconds. It also lists the ten functions where your code spent most of its time. Here you’ve sorted by cumulative time (cumtime), which means that your code counts time when the given function has called another function.

You can see that your code spends virtually all its time inside the latest_tutorial module, and in particular, inside read_latest_tutorial(). While this might be useful confirmation of what you already know, it’s often more interesting to find where your code actually spends time.

The total time (tottime) column indicates how much time your code spent inside a function, excluding time in sub-functions. You can see that none of the functions above really spend any time doing this. To find where the code spent most of its time, issue another sort command:

latest_tutorial.prof% sort tottime
latest_tutorial.prof% stats 10
         1393801 function calls (1389027 primitive calls) in 0.586 seconds

 Ordered by: internal time
 List reduced from 1443 to 10 due to restriction <10>

 ncalls tottime percall cumtime percall filename:lineno(function)
     59   0.091   0.002   0.091   0.002 {method 'read' of '_ssl._SSLSocket'}
 114215   0.070   0.000   0.099   0.000 feedparser.py:308(__getitem__)
 113341   0.046   0.000   0.173   0.000 feedparser.py:756(handle_data)
      1   0.033   0.033   0.033   0.033 {method 'do_handshake' of '_ssl._SSLSocket'}
      1   0.029   0.029   0.029   0.029 {method 'connect' of '_socket.socket'}
     13   0.026   0.002   0.270   0.021 {method 'Parse' of 'pyexpat.xmlparser'}
 113806   0.024   0.000   0.123   0.000 feedparser.py:373(get)
   3455   0.023   0.000   0.024   0.000 {method 'sub' of 're.Pattern'}
 113341   0.019   0.000   0.193   0.000 feedparser.py:2033(characters)
    236   0.017   0.000   0.017   0.000 {method 'translate' of 'str'}

You can now see that latest_tutorial.py actually spends most of its time working with sockets or handling data inside feedparser. The latter is one of the dependencies of the Real Python Reader that’s used to parse the tutorial feed.

You can use pstats to get some idea on where your code is spending most of its time and see if you can optimize any bottlenecks you find. You can also use the tool to understand the structure of your code better. For instance, the commands callees and callers will show you which functions call and are called by a given function.

You can also investigate certain functions. Let’s see how much overhead Timer causes by filtering the results with the phrase timer:

latest_tutorial.prof% stats timer
         1393801 function calls (1389027 primitive calls) in 0.586 seconds

 Ordered by: internal time
 List reduced from 1443 to 8 due to restriction <'timer'>

 ncalls tottime percall cumtime percall filename:lineno(function)
      1   0.000   0.000   0.000   0.000 timer.py:13(Timer)
      1   0.000   0.000   0.000   0.000 timer.py:35(stop)
      1   0.000   0.000   0.003   0.003 timer.py:3(<module>)
      1   0.000   0.000   0.000   0.000 timer.py:28(start)
      1   0.000   0.000   0.000   0.000 timer.py:9(TimerError)
      1   0.000   0.000   0.000   0.000 timer.py:23(__post_init__)
      1   0.000   0.000   0.000   0.000 timer.py:57(__exit__)
      1   0.000   0.000   0.000   0.000 timer.py:52(__enter__)

Luckily, Timer causes only minimal overhead. Use quit to leave the pstats browser when you’re done investigating.

For a more powerful interface into profile data, check out KCacheGrind. It uses its own data format, but you can convert data from cProfile using pyprof2calltree:

$ pyprof2calltree -k -i latest_tutorial.prof

This command will convert latest_tutorial.prof and open KCacheGrind to analyze the data.

The last option you’ll see here for timing your code is line_profiler. cProfile can tell you which functions your code spends the most time in, but it won’t give you insights into which lines inside that function are the slowest. That’s where line_profiler can help you.

Note: You can also profile the memory consumption of your code. This falls outside the scope of this tutorial. However, you can have a look at memory-profiler if you need to monitor the memory consumption of your programs.

Note that line profiling takes time and adds a fair bit of overhead to your runtime. A more standard workflow is first to use cProfile to identify which functions to look at and then run line_profiler on those functions. line_profiler is not part of the standard library, so you should first follow the installation instructions to set it up.

Before you run the profiler, you need to tell it which functions to profile. You do this by adding a @profile decorator inside your source code. For example, to profile Timer.stop() you add the following inside timer.py:

@profile
def stop(self) -> float:
    # The rest of the code is unchanged

Note that you don’t import profile anywhere. Instead, it’s automatically added to the global namespace when you run the profiler. You need to delete the line when you’re done profiling, though. Otherwise, you’ll get a NameError.

Next, run the profiler using kernprof, which is part of the line_profiler package:

$ kernprof -l latest_tutorial.py

This command automatically saves the profiler data in a file called latest_tutorial.py.lprof. You can see those results using line_profiler:

$ python -m line_profiler latest_tutorial.py.lprof
Timer unit: 1e-06 s

Total time: 1.6e-05 s
File: /home/realpython/timer.py
Function: stop at line 35

# Hits Time PrHit %Time Line Contents
=====================================
35                      @profile
36                      def stop(self) -> float:
37                          """Stop the timer, and report the elapsed time"""
38  1   1.0   1.0   6.2     if self._start_time is None:
39                              raise TimerError(f"Timer is not running. ...")
40
41                          # Calculate elapsed time
42  1   2.0   2.0  12.5     elapsed_time = time.perf_counter() - self._start_time
43  1   0.0   0.0   0.0     self._start_time = None
44
45                          # Report elapsed time
46  1   0.0   0.0   0.0     if self.logger:
47  1  11.0  11.0  68.8         self.logger(self.text.format(elapsed_time))
48  1   1.0   1.0   6.2     if self.name:
49  1   1.0   1.0   6.2         self.timers[self.name] += elapsed_time
50
51  1   0.0   0.0   0.0     return elapsed_time

First, note that the time unit in this report is microseconds (1e-06 s). Usually, the most accessible number to look at is %Time, which tells you the percentage of the total time your code spends inside a function at each line. In this example, you can see that your code spends almost 70% of the time on line 47, which is the line that formats and prints the result of the timer.

Conclusion

In this tutorial, you’ve seen several different approaches to adding a Python timer to your code:

You used a class to keep state and add a user-friendly interface. Classes are very flexible, and using Timer directly gives you full control over how and when to invoke the timer.
You used a context manager to add features to a block of code and, if necessary, to clean up afterward. Context managers are straightforward to use, and adding with Timer() can help you more clearly distinguish your code visually.
You used a decorator to add behavior to a function. Decorators are concise and compelling, and using @Timer() is a quick way to monitor your code’s runtime.

You’ve also seen why you should prefer time.perf_counter() over time.time() when benchmarking code, as well as what other alternatives are useful when you’re optimizing your code.

Now you can add Python timer functions to your own code! Keeping track of how fast your program runs in your logs will help you monitor your scripts. Do you have ideas for other use cases where classes, context managers, and decorators play well together? Leave a comment down below!

Resources

For a deeper dive into Python timer functions, check out these resources:

codetiming is the Python timer available on PyPI.
time.perf_counter() is a performance counter for precise timings.
timeit is a tool for comparing the runtimes of code snippets.
cProfile is a profiler for finding bottlenecks in scripts and programs.
pstats is a command-line tool for looking at profiler data.
KCachegrind is a GUI for looking at profiler data.
line_profiler is a profiler for measuring individual lines of code.
memory-profiler is a profiler for monitoring memory usage.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Dictionary Iteration: Advanced Tips & Tricks

2019-12-24T14:00:00+00:00

Dictionaries are one of the most important and useful data structures in Python. They can help you solve a wide variety of programming problems. This course will take you on a deep dive into how to iterate through a dictionary in Python.

By the end of this course, you’ll know:

What dictionaries are, as well as some of their main features and implementation details
How to iterate through a dictionary in Python by using the basic tools the language offers
What kind of real-world tasks you can perform by iterating through a dictionary in Python
How to use some more advanced techniques and strategies to iterate through a dictionary in Python

For more information on dictionaries, you can check out the following resources:

Dictionaries in Python
Itertools in Python 3, By Example
The documentation for map() and filter()

Ready? Let’s go!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

NumPy, SciPy, and Pandas: Correlation With Python

2019-12-23T14:00:00+00:00

Correlation coefficients quantify the association between variables or features of a dataset. These statistics are of high importance for science and technology, and Python has great tools that you can use to calculate them. SciPy, NumPy, and Pandas correlation methods are fast, comprehensive, and well-documented.

In this tutorial, you’ll learn:

What Pearson, Spearman, and Kendall correlation coefficients are
How to use SciPy, NumPy, and Pandas correlation functions
How to visualize data, regression lines, and correlation matrices with Matplotlib

You’ll start with an explanation of correlation, then see three quick introductory examples, and finally dive into details of NumPy, SciPy and Pandas correlation.

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

Correlation

Statistics and data science are often concerned about the relationships between two or more variables (or features) of a dataset. Each data point in the dataset is an observation, and the features are the properties or attributes of those observations.

Every dataset you work with uses variables and observations. For example, you might be interested in understanding the following:

How the height of basketball players is correlated to their shooting accuracy
Whether there’s a relationship between employee work experience and salary
What mathematical dependence exists between the population density and the gross domestic product of different countries

In the examples above, the height, shooting accuracy, years of experience, salary, population density, and gross domestic product are the features or variables. The data related to each player, employee, and each country are the observations.

When data is represented in the form of a table, the rows of that table are usually the observations, while the columns are the features. Take a look at this employee table:

Name	Years of Experience	Annual Salary
Ann	30	120,000
Rob	21	105,000
Tom	19	90,000
Ivy	10	82,000

In this table, each row represents one observation, or the data about one employee (either Ann, Rob, Tom, or Ivy). Each column shows one property or feature (name, experience, or salary) for all the employees.

If you analyze any two features of a dataset, then you’ll find some type of correlation between those two features. Consider the following figures:

Each of these plots shows one of three different forms of correlation:

Negative correlation (red dots): In the plot on the left, the y values tend to decrease as the x values increase. This shows strong negative correlation, which occurs when large values of one feature correspond to small values of the other, and vice versa.
Weak or no correlation (green dots): The plot in the middle shows no obvious trend. This is a form of weak correlation, which occurs when an association between two features is not obvious or is hardly observable.
Positive correlation (blue dots): In the plot on the right, the y values tend to increase as the x values increase. This illustrates strong positive correlation, which occurs when large values of one feature correspond to large values of the other, and vice versa.

The next figure represents the data from the employee table above:

The correlation between experience and salary is positive because higher experience corresponds to a larger salary and vice versa.

Note: When you’re analyzing correlation, you should always have in mind that correlation does not indicate causation. It quantifies the strength of the relationship between the features of a dataset. Sometimes, the association is caused by a factor common to several features of interest.

Correlation is tightly connected to other statistical quantities like the mean, standard deviation, variance, and covariance. If you want to learn more about these quantities and how to calculate them with Python, then check out Descriptive Statistics with Python.

There are several statistics that you can use to quantify correlation. In this tutorial, you’ll learn about three correlation coefficients:

Pearson’s coefficient measures linear correlation, while the Spearman and Kendall coefficients compare the ranks of data. There are several NumPy, SciPy, and Pandas correlation functions and methods that you can use to calculate these coefficients. You can also use Matplotlib to conveniently illustrate the results.

Example: NumPy Correlation Calculation

NumPy has many statistics routines, including np.corrcoef(), that return a matrix of Pearson correlation coefficients. You can start by importing NumPy and defining two NumPy arrays. These are instances of the class ndarray. Call them x and y:

>>>

>>> import numpy as np
>>> x = np.arange(10, 20)
>>> x
array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])
>>> y = np.array([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> y
array([ 2,  1,  4,  5,  8, 12, 18, 25, 96, 48])

Here, you use np.arange() to create an array x of integers between 10 (inclusive) and 20 (exclusive). Then you use np.array() to create a second array y containing arbitrary integers.

Once you have two arrays of the same length, you can call np.corrcoef() with both arrays as arguments:

>>>

>>> r = np.corrcoef(x, y)
>>> r
array([[1.        , 0.75864029],
       [0.75864029, 1.        ]])
>>> r[0, 1]
0.7586402890911867
>>> r[1, 0]
0.7586402890911869

corrcoef() returns the correlation matrix, which is a two-dimensional array with the correlation coefficients. Here’s a simplified version of the correlation matrix you just created:

        x       y

x    1.00    0.76
y    0.76    1.00

The values on the main diagonal of the correlation matrix (upper left and lower right) are equal to 1. The upper left value corresponds to the correlation coefficient for x and x, while the lower right value is the correlation coefficient for y and y. They are always equal to 1.

However, what you usually need are the lower left and upper right values of the correlation matrix. These values are equal and both represent the Pearson correlation coefficient for x and y. In this case, it’s approximately 0.76.

This figure shows the data points and the correlation coefficients for the above example:

The red squares are the data points. As you can see, the figure also shows the values of the three correlation coefficients.

Example: SciPy Correlation Calculation

SciPy also has many statistics routines contained in scipy.stats. You can use the following methods to calculate the three correlation coefficients you saw earlier:

Here’s how you would use these functions in Python:

>>>

>>> import numpy as np
>>> import scipy.stats
>>> x = np.arange(10, 20)
>>> y = np.array([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> scipy.stats.pearsonr(x, y)    # Pearson's r
(0.7586402890911869, 0.010964341301680832)
>>> scipy.stats.spearmanr(x, y)   # Spearman's rho
SpearmanrResult(correlation=0.9757575757575757, pvalue=1.4675461874042197e-06)
>>> scipy.stats.kendalltau(x, y)  # Kendall's tau
KendalltauResult(correlation=0.911111111111111, pvalue=2.9761904761904762e-05)

Note that these functions return objects that contain two values:

The correlation coefficient
The p-value

You use the p-value in statistical methods when you’re testing a hypothesis. The p-value is an important measure that requires in-depth knowledge of probability and statistics to interpret. To learn more about them, you can read about the basics or check out a data scientist’s explanation of p-values.

You can extract the p-values and the correlation coefficients with their indices, as the items of tuples:

>>>

>>> scipy.stats.pearsonr(x, y)[0]    # Pearson's r
0.7586402890911869
>>> scipy.stats.spearmanr(x, y)[0]   # Spearman's rho
0.9757575757575757
>>> scipy.stats.kendalltau(x, y)[0]  # Kendall's tau
0.911111111111111

You could also use dot notation for the Spearman and Kendall coefficients:

>>>

>>> scipy.stats.spearmanr(x, y).correlation   # Spearman's rho
0.9757575757575757
>>> scipy.stats.kendalltau(x, y).correlation  # Kendall's tau
0.911111111111111

The dot notation is longer, but it’s also more readable and more self-explanatory.

If you want to get the Pearson correlation coefficient and p-value at the same time, then you can unpack the return value:

>>>

>>> r, p = scipy.stats.pearsonr(x, y)
>>> r
0.7586402890911869
>>> p
0.010964341301680829

This approach exploits Python unpacking and the fact that pearsonr() returns a tuple with these two statistics. You can also use this technique with spearmanr() and kendalltau(), as you’ll see later on.

Example: Pandas Correlation Calculation

Pandas is, in some cases, more convenient than NumPy and SciPy for calculating statistics. It offers statistical methods for Series and DataFrame instances. For example, given two Series objects with the same number of items, you can call .corr() on one of them with the other as the first argument:

>>>

>>> import pandas as pd
>>> x = pd.Series(range(10, 20))
>>> x
0    10
1    11
2    12
3    13
4    14
5    15
6    16
7    17
8    18
9    19
dtype: int64
>>> y = pd.Series([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> y
0     2
1     1
2     4
3     5
4     8
5    12
6    18
7    25
8    96
9    48
dtype: int64
>>> x.corr(y)                     # Pearson's r
0.7586402890911867
>>> y.corr(x)
0.7586402890911869
>>> x.corr(y, method='spearman')  # Spearman's rho
0.9757575757575757
>>> x.corr(y, method='kendall')   # Kendall's tau
0.911111111111111

Here, you use .corr() to calculate all three correlation coefficients. You define the desired statistic with the parameter method, which can take on one of several values:

'pearson'
'spearman'
'kendall'
a callable

The callable can be any function, method, or object with .__call__() that accepts two one-dimensional arrays and returns a floating-point number.

Linear Correlation

Linear correlation measures the proximity of the mathematical relationship between variables or dataset features to a linear function. If the relationship between the two features is closer to some linear function, then their linear correlation is stronger and the absolute value of the correlation coefficient is higher.

Pearson Correlation Coefficient

Consider a dataset with two features: x and y. Each feature has n values, so x and y are n-tuples. Say that the first value x₁ from x corresponds to the first value y₁ from y, the second value x₂ from x to the second value y₂ from y, and so on. Then, there are n pairs of corresponding values: (x₁, y₁), (x₂, y₂), and so on. Each of these x-y pairs represents a single observation.

The Pearson (product-moment) correlation coefficient is a measure of the linear relationship between two features. It’s the ratio of the covariance of x and y to the product of their standard deviations. It’s often denoted with the letter r and called Pearson’s r. You can express this value mathematically with this equation:

r = Σᵢ((xᵢ − mean(x))(yᵢ − mean(y))) (√Σᵢ(xᵢ − mean(x))² √Σᵢ(yᵢ − mean(y))²)⁻¹

Here, i takes on the values 1, 2, …, n. The mean values of x and y are denoted with mean(x) and mean(y). This formula shows that if larger x values tend to correspond to larger y values and vice versa, then r is positive. On the other hand, if larger x values are mostly associated with smaller y values and vice versa, then r is negative.

Here are some important facts about the Pearson correlation coefficient:

The Pearson correlation coefficient can take on any real value in the range −1 ≤ r ≤ 1.
The maximum value r = 1 corresponds to the case when there’s a perfect positive linear relationship between x and y. In other words, larger x values correspond to larger y values and vice versa.
The value r > 0 indicates positive correlation between x and y.
The value r = 0 corresponds to the case when x and y are independent.
The value r < 0 indicates negative correlation between x and y.
The minimal value r = −1 corresponds to the case when there’s a perfect negative linear relationship between x and y. In other words, larger x values correspond to smaller y values and vice versa.

The above facts can be summed up in the following table:

Pearson’s r Value	Correlation Between x and y
equal to 1	perfect positive linear relationship
greater than 0	positive correlation
equal to 0	independent
less than 0	negative correlation
equal to -1	perfect negative linear relationship

In short, a larger absolute value of r indicates stronger correlation, closer to a linear function. A smaller absolute value of r indicates weaker correlation.

Linear Regression: SciPy Implementation

Linear regression is the process of finding the linear function that is as close as possible to the actual relationship between features. In other words, you determine the linear function that best describes the association between the features. This linear function is also called the regression line.

You can implement linear regression with SciPy. You’ll get the linear function that best approximates the relationship between two arrays, as well as the Pearson correlation coefficient. To get started, you first need to import the libraries and prepare some data to work with:

>>>

>>> import numpy as np
>>> import scipy.stats
>>> x = np.arange(10, 20)
>>> y = np.array([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])

Here, you import numpy and scipy.stats and define the variables x and y.

You can use scipy.stats.linregress() to perform linear regression for two arrays of the same length. You should provide the arrays as the arguments and get the outputs by using dot notation:

>>>

>>> result = scipy.stats.linregress(x, y)
>>> result.slope
7.4363636363636365
>>> result.intercept
-85.92727272727274
>>> result.rvalue
0.7586402890911869
>>> result.pvalue
0.010964341301680825
>>> result.stderr
2.257878767543913

That’s it! You’ve completed the linear regression and gotten the following results:

.slope: the slope of the regression line
.intercept: the intercept of the regression line
.pvalue: the p-value
.stderr: the standard error of the estimated gradient

You’ll learn how to visualize these results in a later section.

You can also provide a single argument to linregress(), but it must be a two-dimensional array with one dimension of length two:

>>>

>>> xy = np.array([[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
...                [2, 1, 4, 5, 8, 12, 18, 25, 96, 48]])
>>> scipy.stats.linregress(xy)
LinregressResult(slope=7.4363636363636365, intercept=-85.92727272727274, rvalue=0.7586402890911869, pvalue=0.010964341301680825, stderr=2.257878767543913)

The result is exactly the same as the previous example because xy contains the same data as x and y together. linregress() took the first row of xy as one feature and the second row as the other feature.

Note: In the example above, scipy.stats.linregress() considers the rows as features and columns as observations. That’s because there are two rows.

The usual practice in machine learning is the opposite: rows are observations and columns are features. Many machine learning libraries, like Pandas, Scikit-Learn, Keras, and others, follow this convention.

You should be careful to note how the observations and features are indicated whenever you’re analyzing correlation in a dataset.

linregress() will return the same result if you provide the transpose of xy, or a NumPy array with 10 rows and two columns. In NumPy, you can transpose a matrix in many ways:

Here’s how you might transpose xy:

>>>

>>> xy.T
array([[10,  2],
       [11,  1],
       [12,  4],
       [13,  5],
       [14,  8],
       [15, 12],
       [16, 18],
       [17, 25],
       [18, 96],
       [19, 48]])

Now that you know how to get the transpose, you can pass one to linregress(). The first column will be one feature and the second column the other feature:

>>>

>>> scipy.stats.linregress(xy.T)
LinregressResult(slope=7.4363636363636365, intercept=-85.92727272727274, rvalue=0.7586402890911869, pvalue=0.010964341301680825, stderr=2.257878767543913)

Here, you use .T to get the transpose of xy. linregress() works the same way with xy and its transpose. It extracts the features by splitting the array along the dimension with length two.

You should also be careful to note whether or not your dataset contains missing values. In data science and machine learning, you’ll often find some missing or corrupted data. The usual way to represent it in Python, NumPy, SciPy, and Pandas is by using NaN or Not a Number values. But if your data contains nan values, then you won’t get a useful result with linregress():

>>>

>>> scipy.stats.linregress(np.arange(3), np.array([2, np.nan, 5]))
LinregressResult(slope=nan, intercept=nan, rvalue=nan, pvalue=nan, stderr=nan)

In this case, your resulting object returns all nan values. In Python, nan is a special floating-point value that you can get by using any of the following:

You can also check whether a variable corresponds to nan with math.isnan() or numpy.isnan().

Pearson Correlation: NumPy and SciPy Implementation

You’ve already seen how to get the Pearson correlation coefficient with corrcoef() and pearsonr():

>>>

>>> r, p = scipy.stats.pearsonr(x, y)
>>> r
0.7586402890911869
>>> p
0.010964341301680829
>>> np.corrcoef(x, y)
array([[1.        , 0.75864029],
       [0.75864029, 1.        ]])

Note that if you provide an array with a nan value to pearsonr(), you’ll get a ValueError.

There are few additional details worth considering. First, recall that np.corrcoef() can take two NumPy arrays as arguments. Instead, you can pass a single two-dimensional array with the same values as the argument:

>>>

>>> np.corrcoef(xy)
array([[1.        , 0.75864029],
       [0.75864029, 1.        ]])

The results are the same in this and previous examples. Again, the first row of xy represents one feature, while the second row represents the other.

If you want to get the correlation coefficients for three features, then you just provide a numeric two-dimensional array with three rows as the argument:

>>>

>>> xyz = np.array([[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
...                 [2, 1, 4, 5, 8, 12, 18, 25, 96, 48],
...                 [5, 3, 2, 1, 0, -2, -8, -11, -15, -16]])
>>> np.corrcoef(xyz)
array([[ 1.        ,  0.75864029, -0.96807242],
       [ 0.75864029,  1.        , -0.83407922],
       [-0.96807242, -0.83407922,  1.        ]])

You’ll obtain the correlation matrix again, but this one will be larger than previous ones:

         x        y        z

x     1.00     0.76    -0.97
y     0.76     1.00    -0.83
z    -0.97    -0.83     1.00

This is because corrcoef() considers each row of xyz as one feature. The value 0.76 is the correlation coefficient for the first two features of xyz. This is the same as the coefficient for x and y in previous examples. -0.97 represents Pearson’s r for the first and third features, while -0.83 is Pearson’s r for the last two features.

Here’s an interesting example of what happens when you pass nan data to corrcoef():

>>>

>>> arr_with_nan = np.array([[0, 1, 2, 3],
...                          [2, 4, 1, 8],
...                          [2, 5, np.nan, 2]])
>>> np.corrcoef(arr_with_nan)
array([[1.        , 0.62554324,        nan],
       [0.62554324, 1.        ,        nan],
       [       nan,        nan,        nan]])

In this example, the first two rows (or features) of arr_with_nan are okay, but the third row [2, 5, np.nan, 2] contains a nan value. Everything that doesn’t include the feature with nan is calculated well. The results that depend on the last row, however, are nan.

By default, numpy.corrcoef() considers the rows as features and the columns as observations. If you want the opposite behavior, which is widely used in machine learning, then use the argument rowvar=False:

>>>

>>> xyz.T
array([[ 10,   2,   5],
       [ 11,   1,   3],
       [ 12,   4,   2],
       [ 13,   5,   1],
       [ 14,   8,   0],
       [ 15,  12,  -2],
       [ 16,  18,  -8],
       [ 17,  25, -11],
       [ 18,  96, -15],
       [ 19,  48, -16]])
>>> np.corrcoef(xyz.T, rowvar=False)
array([[ 1.        ,  0.75864029, -0.96807242],
       [ 0.75864029,  1.        , -0.83407922],
       [-0.96807242, -0.83407922,  1.        ]])

This array is identical to the one you saw earlier. Here, you apply a different convention, but the result is the same.

Pearson Correlation: Pandas Implementation

So far, you’ve used Series and DataFrame object methods to calculate correlation coefficients. Let’s explore these methods in more detail. First, you need to import Pandas and create some instances of Series and DataFrame:

>>>

>>> import pandas as pd
>>> x = pd.Series(range(10, 20))
>>> x
0    10
1    11
2    12
3    13
4    14
5    15
6    16
7    17
8    18
9    19
dtype: int64
>>> y = pd.Series([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> y
0     2
1     1
2     4
3     5
4     8
5    12
6    18
7    25
8    96
9    48
dtype: int64
>>> z = pd.Series([5, 3, 2, 1, 0, -2, -8, -11, -15, -16])
>>> z
0     5
1     3
2     2
3     1
4     0
5    -2
6    -8
7   -11
8   -15
9   -16
dtype: int64
>>> xy = pd.DataFrame({'x-values': x, 'y-values': y})
>>> xy
   x-values  y-values
0        10         2
1        11         1
2        12         4
3        13         5
4        14         8
5        15        12
6        16        18
7        17        25
8        18        96
9        19        48
>>> xyz = pd.DataFrame({'x-values': x, 'y-values': y, 'z-values': z})
>>> xyz
   x-values  y-values  z-values
0        10         2         5
1        11         1         3
2        12         4         2
3        13         5         1
4        14         8         0
5        15        12        -2
6        16        18        -8
7        17        25       -11
8        18        96       -15
9        19        48       -16

You now have three Series objects called x, y, and z. You also have two DataFrame objects, xy and xyz.

Note: When you work with DataFrame instances, you should be aware that the rows are observations and the columns are features. This is consistent with the usual practice in machine learning.

You’ve already learned how to use .corr() with Series objects to get the Pearson correlation coefficient:

>>>

>>> x.corr(y)
0.7586402890911867

Here, you call .corr() on one object and pass the other as the first argument.

If you provide a nan value, then .corr() will still work, but it will exclude observations that contain nan values:

>>>

>>> u, u_with_nan = pd.Series([1, 2, 3]), pd.Series([1, 2, np.nan, 3])
>>> v, w = pd.Series([1, 4, 8]), pd.Series([1, 4, 154, 8])
>>> u.corr(v)
0.9966158955401239
>>> u_with_nan.corr(w)
0.9966158955401239

You get the same value of the correlation coefficient in these two examples. That’s because .corr() ignores the pair of values (np.nan, 154) that has a missing value.

You can also use .corr() with DataFrame objects. You can use it to get the correlation matrix for their columns:

>>>

>>> corr_matrix = xy.corr()
>>> corr_matrix
          x-values  y-values
x-values   1.00000   0.75864
y-values   0.75864   1.00000

The resulting correlation matrix is a new instance of DataFrame and holds the correlation coefficients for the columns xy['x-values'] and xy['y-values']. Such labeled results are usually very convenient to work with because you can access them with either their labels or their integer position indices:

>>>

>>> corr_matrix.at['x-values', 'y-values']
0.7586402890911869
>>> corr_matrix.iat[0, 1]
0.7586402890911869

This example shows two ways of accessing values:

Use .at[] to access a single value by row and column labels.
Use .iat[] to access a value by the positions of its row and column.

You can apply .corr() the same way with DataFrame objects that contain three or more columns:

>>>

>>> xyz.corr()
          x-values  y-values  z-values
x-values  1.000000  0.758640 -0.968072
y-values  0.758640  1.000000 -0.834079
z-values -0.968072 -0.834079  1.000000

You’ll get a correlation matrix with the following correlation coefficients:

0.758640 for x-values and y-values
-0.968072 for x-values and z-values
-0.834079 for y-values and z-values

Another useful method is .corrwith(), which allows you to calculate the correlation coefficients between the rows or columns of one DataFrame object and another Series or DataFrame object passed as the first argument:

>>>

>>> xy.corrwith(z)
x-values   -0.968072
y-values   -0.834079
dtype: float64

In this case, the result is a new Series object with the correlation coefficient for the column xy['x-values'] and the values of z, as well as the coefficient for xy['y-values'] and z.

.corrwith() has the optional parameter axis that specifies whether columns or rows represent the features. The default value of axis is 0, and it also defaults to columns representing features. There’s also a drop parameter, which indicates what to do with missing values.

Both .corr() and .corrwith() have the optional parameter method to specify the correlation coefficient that you want to calculate. The Pearson correlation coefficient is returned by default, so you don’t need to provide it in this case.

Rank Correlation

Rank correlation compares the ranks or the orderings of the data related to two variables or dataset features. If the orderings are similar, then the correlation is strong, positive, and high. However, if the orderings are close to reversed, then the correlation is strong, negative, and low. In other words, rank correlation is concerned only with the order of values, not with the particular values from the dataset.

To illustrate the difference between linear and rank correlation, consider the following figure:

The left plot has a perfect positive linear relationship between x and y, so r = 1. The central plot shows positive correlation and the right one shows negative correlation. However, neither of them is a linear function, so r is different than −1 or 1.

When you look only at the orderings or ranks, all three relationships are perfect! The left and central plots show the observations where larger x values always correspond to larger y values. This is perfect positive rank correlation. The right plot illustrates the opposite case, which is perfect negative rank correlation.

Spearman Correlation Coefficient

The Spearman correlation coefficient between two features is the Pearson correlation coefficient between their rank values. It’s calculated the same way as the Pearson correlation coefficient but takes into account their ranks instead of their values. It’s often denoted with the Greek letter rho (ρ) and called Spearman’s rho.

Say you have two n-tuples, x and y, where (x₁, y₁), (x₂, y₂), … are the observations as pairs of corresponding values. You can calculate the Spearman correlation coefficient ρ the same way as the Pearson coefficient. You’ll use the ranks instead of the actual values from x and y.

Here are some important facts about the Pearson correlation coefficient:

It can take a real value in the range −1 ≤ ρ ≤ 1.
Its maximum value ρ = 1 corresponds to the case when there’s a monotonically increasing function between x and y. In other words, larger x values correspond to larger y values and vice versa.
Its minimum value ρ = −1 corresponds to the case when there’s a monotonically decreasing function between x and y. In other words, larger x values correspond to smaller y values and vice versa.

You can calculate Spearman’s rho in Python in a very similar way as you would Pearson’s r.

Kendall Correlation Coefficient

Let’s start again by considering two n-tuples, x and y. Each of the x-y pairs (x₁, y₁), (x₂, y₂), … is a single observation. A pair of observations (xᵢ, yᵢ) and (xⱼ, yⱼ), where i < j, will be one of three things:

concordant if either (xᵢ > xⱼ and yᵢ > yⱼ) or (xᵢ < xⱼ and yᵢ < yⱼ)
discordant if either (xᵢ < xⱼ and yᵢ > yⱼ) or (xᵢ > xⱼ and yᵢ < yⱼ)
neither if there’s a tie in x (xᵢ = xⱼ) or a tie in y (yᵢ = yⱼ)

The Kendall correlation coefficient compares the number of concordant and discordant pairs of data. This coefficient is based on the difference in the counts of concordant and discordant pairs relative to the number of x-y pairs. It’s often denoted with the Greek letter tau (τ) and called Kendall’s tau.

According to the scipy.stats official docs, the Kendall correlation coefficient is calculated as τ = (n⁺ − n⁻) / √((n⁺ + n⁻ + nˣ)(n⁺ + n⁻ + nʸ)), where:

n⁺ is the number of concordant pairs
n⁻ is the number of discordant pairs
nˣ is the number of ties only in x
nʸ is the number of ties only in y

If a tie occurs in both x and y, then it’s not included in either nˣ or nʸ.

The Wikipedia page on Kendall rank correlation coefficient gives the following expression: τ = (2 / (n(n − 1))) Σᵢⱼ(sign(xᵢ − xⱼ) sign(yᵢ − yⱼ)) for i < j, where i = 1, 2, …, n − 1 and j = 2, 3, …, n. The sign function sign(z) is −1 if z < 0, 0 if z = 0, and 1 if z > 0. n(n − 1) / 2 is the total number of x-y pairs.

Some important facts about the Kendall correlation coefficient are as follows:

It can take a real value in the range −1 ≤ τ ≤ 1.
Its maximum value τ = 1 corresponds to the case when the ranks of the corresponding values in x and y are the same. In other words, all pairs are concordant.
Its minimum value τ = −1 corresponds to the case when the rankings in x are the reverse of the rankings in y. In other words, all pairs are discordant.

You can calculate Kendall’s tau in Python similarly to how you would calculate Pearson’s r.

Rank: SciPy Implementation

You can use scipy.stats to determine the rank for each value in an array. First, you’ll import the libraries and create NumPy arrays:

>>>

>>> import numpy as np
>>> import scipy.stats
>>> x = np.arange(10, 20)
>>> y = np.array([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> z = np.array([5, 3, 2, 1, 0, -2, -8, -11, -15, -16])

Now that you’ve prepared data, you can determine the rank of each value in a NumPy array with scipy.stats.rankdata():

>>>

>>> scipy.stats.rankdata(x)
array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.])
>>> scipy.stats.rankdata(y)
array([ 2.,  1.,  3.,  4.,  5.,  6.,  7.,  8., 10.,  9.])
>>> scipy.stats.rankdata(z)
array([10.,  9.,  8.,  7.,  6.,  5.,  4.,  3.,  2.,  1.])

The arrays x and z are monotonic, so their ranks are monotonic as well. The smallest value in y is 1 and it corresponds to the rank 1. The second smallest is 2, which corresponds to the rank 2. The largest value is 96, which corresponds to the largest rank 10 since there are 10 items in the array.

rankdata() has the optional parameter method. This tells Python what to do if there are ties in the array (if two or more values are equal). By default, it assigns them the average of the ranks:

>>>

>>> scipy.stats.rankdata([8, 2, 0, 2])
array([4. , 2.5, 1. , 2.5])

There are two elements with a value of 2 and they have the ranks 2.0 and 3.0. The value 0 has rank 1.0 and the value 8 has rank 4.0. Then, both elements with the value 2 will get the same rank 2.5.

rankdata() treats nan values as if they were large:

>>>

>>> scipy.stats.rankdata([8, np.nan, 0, 2])
array([3., 4., 1., 2.])

In this case, the value np.nan corresponds to the largest rank 4.0. You can also get ranks with np.argsort():

>>>

>>> np.argsort(y) + 1
array([ 2,  1,  3,  4,  5,  6,  7,  8, 10,  9])

argsort() returns the indices that the array items would have in the sorted array. These indices are zero-based, so you’ll need to add 1 to all of them.

Rank Correlation: NumPy and SciPy Implementation

You can calculate the Spearman correlation coefficient with scipy.stats.spearmanr():

>>>

>>> result = scipy.stats.spearmanr(x, y)
>>> result
SpearmanrResult(correlation=0.9757575757575757, pvalue=1.4675461874042197e-06)
>>> result.correlation
0.9757575757575757
>>> result.pvalue
1.4675461874042197e-06
>>> rho, p = scipy.stats.spearmanr(x, y)
>>> rho
0.9757575757575757
>>> p
1.4675461874042197e-06

spearmanr() returns an object that contains the value of the Spearman correlation coefficient and p-value. As you can see, you can access particular values in two ways:

Using dot notation (result.correlation and result.pvalue)
Using Python unpacking (rho, p = scipy.stats.spearmanr(x, y))

You can get the same result if you provide the two-dimensional array xy that contains the same data as x and y to spearmanr():

>>>

>>> xy = np.array([[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
...                [2, 1, 4, 5, 8, 12, 18, 25, 96, 48]])
>>> rho, p = scipy.stats.spearmanr(xy, axis=1)
>>> rho
0.9757575757575757
>>> p
1.4675461874042197e-06

The first row of xy is one feature, while the second row is the other feature. You can modify this. The optional parameter axis determines whether columns (axis=0) or rows (axis=1) represent the features. The default behavior is that the rows are observations and the columns are features.

Another optional parameter nan_policy defines how to handle nan values. It can take one of three values:

'propagate' returns nan if there’s a nan value among the inputs. This is the default behavior.
'raise' raises a ValueError if there’s a nan value among the inputs.
'omit' ignores the observations with nan values.

If you provide a two-dimensional array with more than two features, then you’ll get the correlation matrix and the matrix of the p-values:

>>>

>>> xyz = np.array([[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
...                 [2, 1, 4, 5, 8, 12, 18, 25, 96, 48],
...                 [5, 3, 2, 1, 0, -2, -8, -11, -15, -16]])
>>> corr_matrix, p_matrix = scipy.stats.spearmanr(xyz, axis=1)
>>> corr_matrix
array([[ 1.        ,  0.97575758, -1.        ],
       [ 0.97575758,  1.        , -0.97575758],
       [-1.        , -0.97575758,  1.        ]])
>>> p_matrix
array([[6.64689742e-64, 1.46754619e-06, 6.64689742e-64],
       [1.46754619e-06, 6.64689742e-64, 1.46754619e-06],
       [6.64689742e-64, 1.46754619e-06, 6.64689742e-64]])

The value -1 in the correlation matrix shows that the first and third features have a perfect negative rank correlation, that is that larger values in the first row always correspond to smaller values in the third.

You can obtain the Kendall correlation coefficient with kendalltau():

>>>

>>> result = scipy.stats.kendalltau(x, y)
>>> result
KendalltauResult(correlation=0.911111111111111, pvalue=2.9761904761904762e-05)
>>> result.correlation
0.911111111111111
>>> result.pvalue
2.9761904761904762e-05
>>> tau, p = scipy.stats.kendalltau(x, y)
>>> tau
0.911111111111111
>>> p
2.9761904761904762e-05

kendalltau() works much like spearmanr(). It takes two one-dimensional arrays, has the optional parameter nan_policy, and returns an object with the values of the correlation coefficient and p-value.

However, if you provide only one two-dimensional array as an argument, then kendalltau() will raise a TypeError. If you pass two multi-dimensional arrays of the same shape, then they’ll be flattened before the calculation.

Rank Correlation: Pandas Implementation

You can calculate the Spearman and Kendall correlation coefficients with Pandas. Just like before, you start by importing pandas and creating some Series and DataFrame instances:

>>>

>>> import pandas as pd
>>> x, y, z = pd.Series(x), pd.Series(y), pd.Series(z)
>>> xy = pd.DataFrame({'x-values': x, 'y-values': y})
>>> xyz = pd.DataFrame({'x-values': x, 'y-values': y, 'z-values': z})

Now that you have these Pandas objects, you can use .corr() and .corrwith() just like you did when you calculated the Pearson correlation coefficient. You just need to specify the desired correlation coefficient with the optional parameter method, which defaults to 'pearson'.

To calculate Spearman’s rho, pass method=spearman:

>>>

>>> x.corr(y, method='spearman')
0.9757575757575757
>>> xy.corr(method='spearman')
          x-values  y-values
x-values  1.000000  0.975758
y-values  0.975758  1.000000
>>> xyz.corr(method='spearman')
          x-values  y-values  z-values
x-values  1.000000  0.975758 -1.000000
y-values  0.975758  1.000000 -0.975758
z-values -1.000000 -0.975758  1.000000
>>> xy.corrwith(z, method='spearman')
x-values   -1.000000
y-values   -0.975758
dtype: float64

If you want Kendall’s tau, then you use method=kendall:

>>>

>>> x.corr(y, method='kendall')
0.911111111111111
>>> xy.corr(method='kendall')
          x-values  y-values
x-values  1.000000  0.911111
y-values  0.911111  1.000000
>>> xyz.corr(method='kendall')
          x-values  y-values  z-values
x-values  1.000000  0.911111 -1.000000
y-values  0.911111  1.000000 -0.911111
z-values -1.000000 -0.911111  1.000000
>>> xy.corrwith(z, method='kendall')
x-values   -1.000000
y-values   -0.911111
dtype: float64

As you can see, unlike with SciPy, you can use a single two-dimensional data structure (a dataframe).

Visualization of Correlation

Data visualization is very important in statistics and data science. It can help you better understand your data and give you a better insight into the relationships between features. In this section, you’ll learn how to visually represent the relationship between two features with an x-y plot. You’ll also use heatmaps to visualize a correlation matrix.

You’ll learn how to prepare data and get certain visual representations, but you won’t cover many other explanations. To learn more about Matplotlib in-depth, check out Python Plotting With Matplotlib (Guide). You can also take a look at the official documentation and Anatomy of Matplotlib.

To get started, first import matplotlib.pyplot:

>>>

>>> import matplotlib.pyplot as plt
>>> plt.style.use('ggplot')

Here, you use plt.style.use('ggplot') to set the style of the plots. Feel free to skip this line if you want.

You’ll use the arrays x, y, z, and xyz from the previous sections. You can create them again to cut down on scrolling:

>>>

>>> import numpy as np
>>> import scipy.stats
>>> x = np.arange(10, 20)
>>> y = np.array([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])
>>> z = np.array([5, 3, 2, 1, 0, -2, -8, -11, -15, -16])
>>> xyz = np.array([[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
...                 [2, 1, 4, 5, 8, 12, 18, 25, 96, 48],
...                 [5, 3, 2, 1, 0, -2, -8, -11, -15, -16]])

Now that you’ve got your data, you’re ready to plot.

X-Y Plots With a Regression Line

First, you’ll see how to create an x-y plot with the regression line, its equation, and the Pearson correlation coefficient. You can get the slope and the intercept of the regression line, as well as the correlation coefficient, with linregress():

>>>

>>> slope, intercept, r, p, stderr = scipy.stats.linregress(x, y)

Now you have all the values you need. You can also get the string with the equation of the regression line and the value of the correlation coefficient. f-strings are very convenient for this purpose:

>>>

>>> line = f'Regression line: y={intercept:.2f}+{slope:.2f}x, r={r:.2f}'
>>> line
'Regression line: y=-85.93+7.44x, r=0.76'

Now, create the x-y plot with .plot():

fig, ax = plt.subplots()
ax.plot(x, y, linewidth=0, marker='s', label='Data points')
ax.plot(x, intercept + slope * x, label=line)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend(facecolor='white')
plt.show()

Your output should look like this:

The red squares represent the observations, while the blue line is the regression line. Its equation is listed in the legend, together with the correlation coefficient.

Heatmaps of Correlation Matrices

The correlation matrix can become really big and confusing when you have a lot of features! Fortunately, you can present it visually as a heatmap where each field has the color that corresponds to its value. You’ll need the correlation matrix:

>>>

>>> corr_matrix = np.corrcoef(xyz).round(decimals=2)
>>> corr_matrix
array([[ 1.  ,  0.76, -0.97],
       [ 0.76,  1.  , -0.83],
       [-0.97, -0.83,  1.  ]])

It can be convenient for you to round the numbers in the correlation matrix with .round(), as they’re going to be shown be on the heatmap.

Finally, create your heatmap with .imshow() and the correlation matrix as its argument:

fig, ax = plt.subplots()
im = ax.imshow(corr_matrix)
im.set_clim(-1, 1)
ax.grid(False)
ax.xaxis.set(ticks=(0, 1, 2), ticklabels=('x', 'y', 'z'))
ax.yaxis.set(ticks=(0, 1, 2), ticklabels=('x', 'y', 'z'))
ax.set_ylim(2.5, -0.5)
for i in range(3):
    for j in range(3):
        ax.text(j, i, corr_matrix[i, j], ha='center', va='center',
                color='r')
cbar = ax.figure.colorbar(im, ax=ax, format='% .2f')
plt.show()

Your output should look like this:

The result is a table with the coefficients. It sort of looks like the Pandas output with colored backgrounds. The colors help you interpret the output. In this example, the yellow color represents the number 1, green corresponds to 0.76, and purple is used for the negative numbers.

Conclusion

You now know that correlation coefficients are statistics that measure the association between variables or features of datasets. They’re very important in data science and machine learning.

You can now use Python to calculate:

Pearson’s product-moment correlation coefficient
Spearman’s rank correlation coefficient
Kendall’s rank correlation coefficient

Now you can use NumPy, SciPy, and Pandas correlation functions and methods to effectively calculate these (and other) statistics, even when you work with large datasets. You also know how to visualize data, regression lines, and correlation matrices with Matplotlib plots and heatmaps.

If you have any questions or comments, please put them in the comments section below!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Run Python Versions in Docker: How to Try the Latest Python Release

2019-12-18T14:00:00+00:00

There’s always a new version of Python under development. However, it can be cumbersome to compile Python yourself to try out a new version! As you work through this tutorial, you’ll see how to run different Python versions using Docker, including how you can have the latest alpha running on your computer within minutes.

In this tutorial, you’ll learn:

Which versions of Python are available
How to get started with Docker
How to run different Python versions in Docker containers
How to use Docker containers as Python environments

Let’s get started!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Understanding Python Versions and Docker

The long journey of moving from Python 2 to Python 3 is coming to a close. Still, it’s important that going forward, you know about the different versions of Python and how to try them out. In general, there are three different kinds of versions you should be aware of:

Released versions: Typically, you’ll be running something like Python 3.6, 3.7, or 3.8. Each of these versions adds new features, so it’s good to be conscious of which version you’re running. For instance, f-strings were introduced in Python 3.6 and won’t work in older versions of Python. Similarly, assignment expressions only became available in Python 3.8.
Development versions: The Python community is continuously working on new versions of Python. At the time of this writing, Python 3.9 is under development. To preview and test new features, users have access to development versions labeled alpha, beta, and release candidate.
Implementations: Python is a language that has several implementations. An implementation of Python contains an interpreter and corresponding libraries. CPython is the reference implementation of Python and the one that is most commonly used. However, there are other implementations like PyPy, IronPython, Jython, MicroPython, and CircuitPython that cover specific use cases.

You’ll usually see which version of Python you’re using when you start a REPL. You can also inspect sys.implementation for more information:

>>>

>>> import sys
>>> sys.implementation.name
'cpython'

>>> sys.implementation.version
sys.version_info(major=3, minor=9, micro=0, releaselevel='alpha', serial=1)

You can see that this code is running the first alpha version of CPython 3.9.

Traditionally, you’d use tools like pyenv and conda to manage different Python versions. Docker can replace these in most cases, and it’s often simpler to use. In the rest of this tutorial, you’ll see how to get started.

Using Docker

Docker is a platform for running containers with prepackaged applications. It’s a very powerful system that’s particularly popular for packaging and deploying applications and microservices. In this section, you’ll see the fundamental concepts you’ll need to know to use Docker.

Installing Docker

Docker is available on all major operating systems: Windows, macOS, and Linux. See the official guide for instructions on how to install Docker on your system. Unless you have special needs, you can use the Docker Engine - Community version.

Running Containers

Docker uses the concepts of images and containers. An image is a self-contained package that can be run by Docker. A container is a running image with a certain state. There are several repositories containing pre-built Docker images. Docker Hub is the default repository that you’ll use in this tutorial. For a first example, run the hello-world image:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:451ce787d12369c5df2a32c85e5a03d52cbcef6eb3586dd03075f3...
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.
[ ... Full output clipped ... ]

The first lines show that Docker downloaded hello-world from Docker Hub. When it runs this image, the resulting container produces a "Hello from Docker!" message that prints to your terminal.

Building Your Own Images Using Dockerfiles

You can create your own images using Dockerfiles, which is a plain text file that describes how a Docker image should be set up. The following is an example of a Dockerfile:

 1 FROM ubuntu
 2 RUN apt update && apt install -y cowsay
 3 CMD ["/usr/games/cowsay", "Dockerfiles are cool!"]

A Dockerfile consists of a list of Docker commands. In the example above, there are three steps:

Line 1 bases the image on an existing image called ubuntu. You can do this independently of which system you’re running Docker on.
Line 2 installs a program named cowsay.
Line 3 prepares a command that runs cowsay when the image is executed.

To use this Dockerfile, save it in a text file named Dockerfile, without any file extension.

Note: You can build and run Linux images on any platform, so images like ubuntu are great for building applications that should be available cross-platform.

In contrast, a Windows image will only run on Windows, and a macOS image will only run on macOS.

Next, build an image from your Dockerfile:

$ docker build -t cowsay .

The command will give a lot of output as it’s building the image. -t cowsay will tag your image with the name cowsay. You can use tags to keep track of your images. The final dot in the command specifies the current directory as the build context for your image. This directory should be the one containing Dockerfile.

You can now run your very own Docker image:

$ docker run --rm cowsay
 _______________________
< Dockerfiles are cool! >
 -----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

The --rm option will clean up your container after use. It’s a good habit to use --rm to avoid filling up your system with stale Docker containers.

Note: Docker has several commands for managing your images and containers. You can list your images and containers using docker images and docker ps -a, respectively.

Both images and containers are assigned a 12-character ID that you can find in these listings. To delete an image or container, use either docker rmi <image_id> or docker rm <container_id> with the correct ID.

The docker command line is very powerful. Use docker --help and the official documentation for more information.

Running Python in a Docker Container

The Docker community releases and maintains Dockerfiles for all new versions of Python, which you can use to try out new Python features. Additionally, the Python core developers maintain a Docker image with all currently available versions of Python. In this section, you’ll learn how to run different Python versions in Docker.

Playing With the REPL

When you run a Python image from Docker Hub, the interpreter is set up so you can play with the REPL directly. To start the REPL in a Python container, run the following command:

$ docker run -it --rm python:rc
Python 3.8.0rc1 (default, Oct  2 2019, 23:30:03)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

This command will download the python:rc image from Docker Hub, start a container, and run python inside that container. The -it options are necessary for running the container interactively. The rc tag is shorthand for release candidate and points to the latest development version of Python. In this case, it’s the last release candidate of Python 3.8:

>>>

>>> import sys
>>> f"{sys.version_info[:] = }"
"sys.version_info[:] = (3, 8, 0, 'candidate', 1)"

The first time you run a container, it may take some time to download. Later invocations will essentially be immediate. You can exit the REPL like usual, for example, by typing exit(). This also exits the container.

Note: The Docker Hub Python images are kept reasonably well up-to-date. As new releases mature, their alpha and beta versions are made available at the rc tag.

However, if you want to test out the absolute latest versions of Python, then the core developers’ image might be a better bet:

$ docker run -it --rm quay.io/python-devs/ci-image:master

You’ll see a few more examples of using this image later.

You can find a list of all available Python images at Docker Hub. python:latest will always give you the latest stable version of Python, while python:rc will provide you with the most recent development version. You can also request specific versions like python:3.6.3 or python:3.8.0b4, the fourth beta version of Python 3.8. You can even run PyPy using a tag like pypy:latest.

Setting Up Your Python Environment

A Docker container is an isolated environment. Therefore, you usually don’t need to add a virtual environment inside the container. Instead, you can run pip directly to install the necessary packages. To modify the container to include the extra packages, you use a Dockerfile. The following example adds parse and realpython-reader to a Python 3.7.5 container:

 1 FROM python:3.7.5-slim
 2 RUN python -m pip install \
 3         parse \
 4         realpython-reader

Save this file with the name Dockerfile. The -slim tag in line 1 points to a Dockerfile based on a minimal Debian installation. This tag gives a significantly slimmer Docker image, but the downside is that you may need to install more additional tools yourself.

Other designations include -alpine and -windowsservercore. You can find more information about these image variants on Docker Hub.

Note: If you’d like to use a virtual environment inside a Docker container, then there’s one caveat you should be aware of. Each RUN command runs in a separate process, which means that the typical activation of a virtual environment won’t work inside of a Dockerfile.

Instead, you should manually activate the virtual environment by setting the VIRTUAL_ENV and PATH environment variables:

FROM python:3.7.5-slim

# Set up and activate virtual environment
ENV VIRTUAL_ENV "/venv"
RUN python -m venv $VIRTUAL_ENV
ENV PATH "$VIRTUAL_ENV/bin:$PATH"

# Python commands run inside the virtual environment
RUN python -m pip install \
        parse \
        realpython-reader

See Elegantly activating a virtualenv in a Dockerfile for more information.

To build and run your Dockerfile, use the following commands:

$ docker build -t rp .
[ ... Output clipped ... ]

$ docker run -it --rm rp

As you build the image, you tag it with the name rp. This name is then used when you run the image, starting a new REPL session. You can confirm that parse has been installed in the container:

>>>

>>> import parse
>>> parse.__version__
'1.12.1'

You can also start containers that run custom commands:

$ docker run --rm rp realpython
The latest tutorials from Real Python (https://realpython.com/)
  0 Run Python Versions in Docker: How to Try the Latest Python Release
[ ... Full output clipped ... ]

Instead of starting a REPL, this runs the realpython command inside the rp container, which lists the latest tutorials published on Real Python. For more information about the realpython-reader package, check out How to Publish an Open-Source Python Package to PyPI.

Running Python Scripts Using Docker

In this section, you’ll see how to run scripts inside Docker. First, save the following example script to a file named headlines.py on your computer:

# headlines.py

import parse
from reader import feed

tutorial = feed.get_article(0)
headlines = [
    r.named["header"]
    for r in parse.findall("\n## {header}\n", tutorial)
]
print("\n".join(headlines))

The script first downloads the latest tutorial from Real Python. Then it uses parse to find all headlines in the tutorial and prints them to the console.

There are two general ways to run scripts like this in your Docker container:

Mount a local directory as a volume in the Docker container.
Copy the script into the Docker container.

The first option is especially useful during testing, as you don’t need to rebuild your Docker image when you make changes to your script. To mount your directory as a volume, use the -v option:

$ docker run --rm -v /home/realpython/code:/app rp python /app/headlines.py
Understanding Python Versions and Docker
Using Docker
Running Python in a Docker Container
Conclusion
Further Reading

The option -v /home/realpython/code:/app says that the local directory /home/realpython/code should be mounted as /app inside the container. You can then run the script with the command python /app/headlines.py.

You’ll want to copy your script into your container if you’re going to deploy your script to another machine. You do this by adding a few steps to your Dockerfile:

FROM python:3.7.5-slim
WORKDIR /usr/src/app
RUN python -m pip install \
        parse \
        realpython-reader
COPY headlines.py .
CMD ["python", "headlines.py"]

You set a working directory inside your container to control where commands are run. You can then copy headlines.py to that working directory inside the container, and change the default command to run headlines.py with python. Rebuild your image as usual, and run the container:

$ docker build -t rp .
[ ... Output clipped ... ]

$ docker run --rm rp
Understanding Python Versions and Docker
Using Docker
Running Python in a Docker Container
Conclusion
Further Reading

Note that your script is run when you run the container because you specified the CMD command in the Dockerfile.

See the Python image description on Docker Hub for more information about building your own Dockerfiles.

Running the Latest Alpha

So far, you’ve been pulling images from Docker Hub, but there are many image repositories available. For instance, many cloud providers like AWS, GCP, and DigitalOcean offer dedicated container registries.

The core developers’ Python image is available at Quay.io. To use images from non-default repositories, you use the fully qualified named. For instance, you can run the core developers’ image as follows:

$ docker run -it --rm quay.io/python-devs/ci-image:master

By default, this starts a shell session inside the container. From the shell session, you can explicitly run Python:

$ python3.9 -c "import sys; print(sys.version_info)"
sys.version_info(major=3, minor=9, micro=0, releaselevel='alpha', serial=1)

You can see all available versions of Python by looking inside /usr/local/bin:

$ ls /usr/local/bin/
2to3              get-pythons.sh  pydoc3.5           python3.7m
2to3-3.4          idle            pydoc3.6           python3.7m-config
2to3-3.5          idle3.4         pydoc3.7           python3.8
2to3-3.6          idle3.5         pydoc3.8           python3.8-config
2to3-3.7          idle3.6         pydoc3.9           python3.9
2to3-3.8          idle3.7         python2.7          python3.9-config
2to3-3.9          idle3.8         python2.7-config   pyvenv-3.4
codecov           idle3.9         python3.4          pyvenv-3.5
coverage          mypy            python3.4m         pyvenv-3.6
coverage-3.6      mypyc           python3.4m-config  pyvenv-3.7
coverage3         pip3.5          python3.5          smtpd.py
dmypy             pip3.6          python3.5m         stubgen
easy_install-3.5  pip3.7          python3.5m-config  tox
easy_install-3.6  pip3.8          python3.6          tox-quickstart
easy_install-3.7  pip3.9          python3.6m         virtualenv
easy_install-3.8  pydoc           python3.6m-config
easy_install-3.9  pydoc3.4        python3.7

This image is especially useful if you want to test your code on several Python versions. The Docker image is updated often and includes the latest development versions of Python. If you’re interested in checking out the latest features of Python, even before they’re officially released, then this image is a great choice.

Conclusion

In this tutorial, you’ve seen a quick introduction to working with different Python versions using Docker. This is a great way to test and see that your code is compatible with newer versions of Python. It only takes a few minutes to wrap your Python script in a Docker container, so you can try out the latest alpha as soon as it’s released!

Now you can:

Start a Python REPL through Docker
Set up your Python environment inside a Docker image
Run scripts inside Docker containers

As you test new Python versions in Docker, you’re providing invaluable help to the Python community. If you have any questions or comments, then please leave them in the comments section below.

Documenting Python Code: A Complete Guide

2019-12-17T14:00:00+00:00

In this course, you’ll learn how to document your Python code! Documenting your code is important because it can help developers and users fully understand its usage and purpose.

You’ll learn about:

The reasons that documenting your code is so important
The differences between commenting and documenting
Best practices for docstrings

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Statistics Fundamentals: How to Describe Your Data

2019-12-16T14:00:00+00:00

In the era of big data and artificial intelligence, data science and machine learning have become essential in many fields of science and technology. A necessary aspect of working with data is the ability to describe, summarize, and represent data visually. Python statistics libraries are comprehensive, popular, and widely used tools that will assist you in working with data.

In this tutorial, you’ll learn:

What numerical quantities you can use to describe and summarize your datasets
How to calculate descriptive statistics in pure Python
How to get descriptive statistics with available Python libraries
How to visualize your datasets

Free Bonus: Click here to download 5 Python + Matplotlib examples with full source code that you can use as a basis for making your own plots and graphics.

Understanding Descriptive Statistics

Descriptive statistics is about describing and summarizing data. It uses two main approaches:

The quantitative approach describes and summarizes data numerically.
The visual approach illustrates data with charts, plots, histograms, and other graphs.

You can apply descriptive statistics to one or many datasets or variables. When you describe and summarize a single variable, you’re performing univariate analysis. When you search for statistical relationships among a pair of variables, you’re doing a bivariate analysis. Similarly, a multivariate analysis is concerned with multiple variables at once.

Types of Measures

In this tutorial, you’ll learn about the following types of measures in descriptive statistics:

Central tendency tells you about the centers of the data. Useful measures include the mean, median, and mode.
Variability tells you about the spread of the data. Useful measures include variance and standard deviation.
Correlation or joint variability tells you about the relation between a pair of variables in a dataset. Useful measures include covariance and the correlation coefficient.

You’ll learn how to understand and calculate these measures with Python.

Population and Samples

In statistics, the population is a set of all elements or items that you’re interested in. Populations are often vast, which makes them inappropriate for collecting and analyzing data. That’s why statisticians usually try to make some conclusions about a population by choosing and examining a representative subset of that population.

This subset of a population is called a sample. Ideally, the sample should preserve the essential statistical features of the population to a satisfactory extent. That way, you’ll be able to use the sample to glean conclusions about the population.

Outliers

An outlier is a data point that differs significantly from the majority of the data taken from a sample or population. There are many possible causes of outliers, but here are a few to start you off:

Natural variation in data
Change in the behavior of the observed system
Errors in data collection

Data collection errors are a particularly prominent cause of outliers. For example, the limitations of measurement instruments or procedures can mean that the correct data is simply not obtainable. Other errors can be caused by miscalculations, data contamination, human error, and more.

There isn’t a precise mathematical definition of outliers. You have to rely on experience, knowledge about the subject of interest, and common sense to determine if a data point is an outlier and how to handle it.

Choosing Python Statistics Libraries

There are many Python statistics libraries out there for you to work with, but in this tutorial, you’ll be learning about some of the most popular and widely used ones:

Python’s statistics is a built-in Python library for descriptive statistics. You can use it if your datasets are not too large or if you can’t rely on importing other libraries.
NumPy is a third-party library for numerical computing, optimized for working with single- and multi-dimensional arrays. Its primary type is the array type called ndarray. This library contains many routines for statistical analysis.
SciPy is a third-party library for scientific computing based on NumPy. It offers additional functionality compared to NumPy, including scipy.stats for statistical analysis.
Pandas is a third-party library for numerical computing based on NumPy. It excels in handling labeled one-dimensional (1D) data with Series objects and two-dimensional (2D) data with DataFrame objects.
Matplotlib is a third-party library for data visualization. It works well in combination with NumPy, SciPy, and Pandas.

Note that, in many cases, Series and DataFrame objects can be used in place of NumPy arrays. Often, you might just pass them to a NumPy or SciPy statistical function. In addition, you can get the unlabeled data from a Series or DataFrame as a np.ndarray object by calling .values or .to_numpy().

Getting Started With Python Statistics Libraries

The built-in Python statistics library has a relatively small number of the most important statistics functions. The official documentation is a valuable resource to find the details. If you’re limited to pure Python, then the Python statistics library might be the right choice.

A good place to start learning about NumPy is the official User Guide, especially the quickstart and basics sections. The official reference can help you refresh your memory on specific NumPy concepts. While you read this tutorial, you might want to check out the statistics section and the official scipy.stats reference as well.

Note:

To learn more about NumPy, check out these resources:

If you want to learn Pandas, then the official Getting Started page is an excellent place to begin. The introduction to data structures can help you learn about the fundamental data types, Series and DataFrame. Likewise, the excellent official introductory tutorial aims to give you enough information to start effectively using Pandas in practice.

Note:

To learn more about Pandas, check out these resources:

matplotlib has a comprehensive official User’s Guide that you can use to dive into the details of using the library. Anatomy of Matplotlib is an excellent resource for beginners who want to start working with matplotlib and its related libraries.

Note:

To learn more about data visualization, check out these resources:

Let’s start using these Python statistics libraries!

Calculating Descriptive Statistics

Start by importing all the packages you’ll need:

>>>

>>> import math
>>> import statistics
>>> import numpy as np
>>> import scipy.stats
>>> import pandas as pd

These are all the packages you’ll need for Python statistics calculations. Usually, you won’t use Python’s built-in math package, but it’ll be useful in this tutorial. Later, you’ll import matplotlib.pyplot for data visualization.

Let’s create some data to work with. You’ll start with Python lists that contain some arbitrary numeric data:

>>>

>>> x = [8.0, 1, 2.5, 4, 28.0]
>>> x_with_nan = [8.0, 1, 2.5, math.nan, 4, 28.0]
>>> x
[8.0, 1, 2.5, 4, 28.0]
>>> x_with_nan
[8.0, 1, 2.5, nan, 4, 28.0]

Now you have the lists x and x_with_nan. They’re almost the same, with the difference that x_with_nan contains a nan value. It’s important to understand the behavior of the Python statistics routines when they come across a not-a-number value (nan). In data science, missing values are common, and you’ll often replace them with nan.

Note: How do you get a nan value?

In Python, you can use any of the following:

You can use all of these functions interchangeably:

>>>

>>> math.isnan(np.nan), np.isnan(math.nan)
(True, True)
>>> math.isnan(y_with_nan[3]), np.isnan(y_with_nan[3])
(True, True)

You can see that the functions are all equivalent. However, please keep in mind that comparing two nan values for equality returns False. In other words, math.nan == math.nan is False!

Now, create np.ndarray and pd.Series objects that correspond to x and x_with_nan:

>>>

>>> y, y_with_nan = np.array(x), np.array(x_with_nan)
>>> z, z_with_nan = pd.Series(x), pd.Series(x_with_nan)
>>> y
array([ 8. ,  1. ,  2.5, 4. , 28. ])
>>> y_with_nan
array([ 8. ,  1. ,  2.5,  nan,  4. , 28. ])
>>> z
0     8.0
1     1.0
2     2.5
3     4.0
4    28.0
dtype: float64
>>> z_with_nan
0     8.0
1     1.0
2     2.5
3     NaN
4     4.0
5    28.0
dtype: float64

You now have two NumPy arrays (y and y_with_nan) and two Pandas Series (z and z_with_nan). All of these are 1D sequences of values.

Note: Although you’ll use lists throughout this tutorial, please keep in mind that, in most cases, you can use tuples in the same way.

You can optionally specify a label for each value in z and z_with_nan.

Measures of Central Tendency

The measures of central tendency show the central or middle values of datasets. There are several definitions of what’s considered to be the center of a dataset. In this tutorial, you’ll learn how to identify and calculate these measures of central tendency:

Mean
Weighted mean
Geometric mean
Harmonic mean
Median
Mode

Mean

The sample mean, also called the sample arithmetic mean or simply the average, is the arithmetic average of all the items in a dataset. The mean of a dataset 𝑥 is mathematically expressed as Σᵢ𝑥ᵢ/𝑛, where 𝑖 = 1, 2, …, 𝑛. In other words, it’s the sum of all the elements 𝑥ᵢ divided by the number of items in the dataset 𝑥.

This figure illustrates the mean of a sample with five data points:

The green dots represent the data points 1, 2.5, 4, 8, and 28. The red dashed line is their mean, or (1 + 2.5 + 4 + 8 + 28) / 5 = 8.7.

You can calculate the mean with pure Python using sum() and len(), without importing libraries:

>>>

>>> mean_ = sum(x) / len(x)
>>> mean_
8.7

Although this is clean and elegant, you can also apply built-in Python statistics functions:

>>>

>>> mean_ = statistics.mean(x)
>>> mean_
8.7
>>> mean_ = statistics.fmean(x)
>>> mean_
8.7

You’ve called the functions mean() and fmean() from the built-in Python statistics library and got the same result as you did with pure Python. fmean() is introduced in Python 3.8 as a faster alternative to mean(). It always returns a floating-point number.

However, if there are nan values among your data, then statistics.mean() and statistics.fmean() will return nan as the output:

>>>

>>> mean_ = statistics.mean(x_with_nan)
>>> mean_
nan
>>> mean_ = statistics.fmean(x_with_nan)
>>> mean_
nan

This result is consistent with the behavior of sum(), because sum(x_with_nan) also returns nan.

If you use NumPy, then you can get the mean with np.mean():

>>>

>>> mean_ = np.mean(y)
>>> mean_
8.7

In the example above, mean() is a function, but you can use the corresponding method .mean() as well:

>>>

>>> mean_ = y.mean()
>>> mean_
8.7

The function mean() and method .mean() from NumPy return the same result as statistics.mean(). This is also the case when there are nan values among your data:

>>>

>>> np.mean(y_with_nan)
nan
>>> y_with_nan.mean()
nan

You often don’t need to get a nan value as a result. If you prefer to ignore nan values, then you can use np.nanmean():

>>>

>>> np.nanmean(y_with_nan)
8.7

nanmean() simply ignores all nan values. It returns the same value as mean() if you were to apply it to the dataset without the nan values.

pd.Series objects also have the method .mean():

>>>

>>> mean_ = z.mean()
>>> mean_
8.7

As you can see, it’s used similarly as in the case of NumPy. However, .mean() from Pandas ignores nan values by default:

>>>

>>> z_with_nan.mean()
8.7

This behavior is the result of the default value of the optional parameter skipna. You can change this parameter to modify the behavior.

Weighted Mean

The weighted mean, also called the weighted arithmetic mean or weighted average, is a generalization of the arithmetic mean that enables you to define the relative contribution of each data point to the result.

You define one weight 𝑤ᵢ for each data point 𝑥ᵢ of the dataset 𝑥, where 𝑖 = 1, 2, …, 𝑛 and 𝑛 is the number of items in 𝑥. Then, you multiply each data point with the corresponding weight, sum all the products, and divide the obtained sum with the sum of weights: Σᵢ(𝑤ᵢ𝑥ᵢ) / Σᵢ𝑤ᵢ.

Note: It’s convenient (and usually the case) that all weights are nonnegative, 𝑤ᵢ ≥ 0, and that their sum is equal to one, or Σᵢ𝑤ᵢ = 1.

The weighted mean is very handy when you need the mean of a dataset containing items that occur with given relative frequencies. For example, say that you have a set in which 20% of all items are equal to 2, 50% of the items are equal to 4, and the remaining 30% of the items are equal to 8. You can calculate the mean of such a set like this:

>>>

>>> 0.2 * 2 + 0.5 * 4 + 0.3 * 8
4.8

Here, you take the frequencies into account with the weights. With this method, you don’t need to know the total number of items.

You can implement the weighted mean in pure Python by combining sum() with either range() or zip():

>>>

>>> x = [8.0, 1, 2.5, 4, 28.0]
>>> w = [0.1, 0.2, 0.3, 0.25, 0.15]
>>> wmean = sum(w[i] * x[i] for i in range(len(x))) / sum(w)
>>> wmean
6.95
>>> wmean = sum(x_ * w_ for (x_, w_) in zip(x, w)) / sum(w)
>>> wmean
6.95

Again, this is a clean and elegant implementation where you don’t need to import any libraries.

However, if you have large datasets, then NumPy is likely to provide a better solution. You can use np.average() to get the weighted mean of NumPy arrays or Pandas Series:

>>>

>>> y, z, w = np.array(x), pd.Series(x), np.array(w)
>>> wmean = np.average(y, weights=w)
>>> wmean
6.95
>>> wmean = np.average(z, weights=w)
>>> wmean
6.95

The result is the same as in the case of the pure Python implementation. You can also use this method on ordinary lists and tuples.

Another solution is to use the element-wise product w * y with np.sum() or .sum():

>>>

>>> (w * y).sum() / w.sum()
6.95

That’s it! You’ve calculated the weighted mean.

However, be careful if your dataset contains nan values:

>>>

>>> w = np.array([0.1, 0.2, 0.3, 0.0, 0.2, 0.1])
>>> (w * y_with_nan).sum() / w.sum()
nan
>>> np.average(y_with_nan, weights=w)
nan
>>> np.average(z_with_nan, weights=w)
nan

In this case, average() returns nan, which is consistent with np.mean().

Harmonic Mean

The harmonic mean is the reciprocal of the mean of the reciprocals of all items in the dataset: 𝑛 / Σᵢ(1/𝑥ᵢ), where 𝑖 = 1, 2, …, 𝑛 and 𝑛 is the number of items in the dataset 𝑥. One variant of the pure Python implementation of the harmonic mean is this:

>>>

>>> hmean = len(x) / sum(1 / item for item in x)
>>> hmean
2.7613412228796843

It’s quite different from the value of the arithmetic mean for the same data x, which you calculated to be 8.7.

You can also calculate this measure with statistics.harmonic_mean():

>>>

>>> hmean = statistics.harmonic_mean(x)
>>> hmean
2.7613412228796843

The example above shows one implementation of statistics.harmonic_mean(). If you have a nan value in a dataset, then it’ll return nan. If there’s at least one 0, then it’ll return 0. If you provide at least one negative number, then you’ll get statistics.StatisticsError:

>>>

>>> statistics.harmonic_mean(x_with_nan)
nan
>>> statistics.harmonic_mean([1, 0, 2])
0
>>> statistics.harmonic_mean([1, 2, -2])  # Raises StatisticsError

Keep these three scenarios in mind when you’re using this method!

A third way to calculate the harmonic mean is to use scipy.stats.hmean():

>>>

>>> scipy.stats.hmean(y)
2.7613412228796843
>>> scipy.stats.hmean(z)
2.7613412228796843

Again, this is a pretty straightforward implementation. However, if your dataset contains nan, 0, a negative number, or anything but positive numbers, then you’ll get a ValueError!

Geometric Mean

The geometric mean is the 𝑛-th root of the product of all 𝑛 elements 𝑥ᵢ in a dataset 𝑥: ⁿ√(Πᵢ𝑥ᵢ), where 𝑖 = 1, 2, …, 𝑛. The following figure illustrates the arithmetic, harmonic, and geometric means of a dataset:

Again, the green dots represent the data points 1, 2.5, 4, 8, and 28. The red dashed line is the mean. The blue dashed line is the harmonic mean, and the yellow dashed line is the geometric mean.

You can implement the geometric mean in pure Python like this:

>>>

>>> gmean = 1
>>> for item in x:
...     gmean *= item
...
>>> gmean **= 1 / len(x)
>>> gmean
4.677885674856041

As you can see, the value of the geometric mean, in this case, differs significantly from the values of the arithmetic (8.7) and harmonic (2.76) means for the same dataset x.

Python 3.8 introduced statistics.geometric_mean(), which converts all values to floating-point numbers and returns their geometric mean:

>>>

>>> gmean = statistics.geometric_mean(x)
>>> gmean
4.67788567485604

You’ve got the same result as in the previous example, but with a minimal rounding error.

If you pass data with nan values, then statistics.geometric_mean() will behave like most similar functions and return nan:

>>>

>>> gmean = statistics.geometric_mean(x_with_nan)
>>> gmean
nan

Indeed, this is consistent with the behavior of statistics.mean(), statistics.fmean(), and statistics.harmonic_mean(). If there’s a zero or negative number among your data, then statistics.geometric_mean() will raise the statistics.StatisticsError.

You can also get the geometric mean with scipy.stats.gmean():

>>>

>>> scipy.stats.gmean(y)
4.67788567485604
>>> scipy.stats.gmean(z)
4.67788567485604

You obtained the same result as with the pure Python implementation.

If you have nan values in a dataset, then gmean() will return nan. If there’s at least one 0, then it’ll return 0.0 and give a warning. If you provide at least one negative number, then you’ll get nan and the warning.

Median

The sample median is the middle element of a sorted dataset. The dataset can be sorted in increasing or decreasing order. If the number of elements 𝑛 of the dataset is odd, then the median is the value at the middle position: 0.5(𝑛 + 1). If 𝑛 is even, then the median is the arithmetic mean of the two values in the middle, that is, the items at the positions 0.5𝑛 and 0.5𝑛 + 1.

For example, if you have the data points 2, 4, 1, 8, and 9, then the median value is 4, which is in the middle of the sorted dataset (1, 2, 4, 8, 9). If the data points are 2, 4, 1, and 8, then the median is 3, which is the average of the two middle elements of the sorted sequence (2 and 4). The following figure illustrates this:

The data points are the green dots, and the purple lines show the median for each dataset. The median value for the upper dataset (1, 2.5, 4, 8, and 28) is 4. If you remove the outlier 28 from the lower dataset, then the median becomes the arithmetic average between 2.5 and 4, which is 3.25.

The figure below shows both the mean and median of the data points 1, 2.5, 4, 8, and 28:

Again, the mean is the red dashed line, while the median is the purple line.

The main difference between the behavior of the mean and median is related to dataset outliers or extremes. The mean is heavily affected by outliers, but the median only depends on outliers either slightly or not at all. Consider the following figure:

The upper dataset again has the items 1, 2.5, 4, 8, and 28. Its mean is 8.7, and the median is 5, as you saw earlier. The lower dataset shows what’s going on when you move the rightmost point with the value 28:

If you increase its value (move it to the right), then the mean will rise, but the median value won’t ever change.
If you decrease its value (move it to the left), then the mean will drop, but the median will remain the same until the value of the moving point is greater than or equal to 4.

You can compare the mean and median as one way to detect outliers and asymmetry in your data. Whether the mean value or the median value is more useful to you depends on the context of your particular problem.

Here is one of many possible pure Python implementations of the median:

>>>

>>> n = len(x)
>>> if n % 2:
...     median_ = sorted(x)[round(0.5*(n-1))]
... else:
...     x_ord, index = sorted(x), round(0.5 * n)
...     median_ = 0.5 * (x_ord[index-1] + x_ord[index])
...
>>> median_
4

Two most important steps of this implementation are as follows:

Sorting the elements of the dataset
Finding the middle element(s) in the sorted dataset

You can get the median with statistics.median():

>>>

>>> median_ = statistics.median(x)
>>> median_
4
>>> median_ = statistics.median(x[:-1])
>>> median_
3.25

The sorted version of x is [1, 2.5, 4, 8.0, 28.0], so the element in the middle is 4. The sorted version of x[:-1], which is x without the last item 28.0, is [1, 2.5, 4, 8.0]. Now, there are two middle elements, 2.5 and 4. Their average is 3.25.

median_low() and median_high() are two more functions related to the median in the Python statistics library. They always return an element from the dataset:

If the number of elements is odd, then there’s a single middle value, so these functions behave just like median().
If the number of elements is even, then there are two middle values. In this case, median_low() returns the lower and median_high() the higher middle value.

You can use these functions just as you’d use median():

>>>

>>> statistics.median_low(x[:-1])
2.5
>>> statistics.median_high(x[:-1])
4

Again, the sorted version of x[:-1] is [1, 2.5, 4, 8.0]. The two elements in the middle are 2.5 (low) and 4 (high).

Unlike most other functions from the Python statistics library, median(), median_low(), and median_high() don’t return nan when there are nan values among the data points:

>>>

>>> statistics.median(x_with_nan)
6.0
>>> statistics.median_low(x_with_nan)
4
>>> statistics.median_high(x_with_nan)
8.0

Beware of this behavior because it might not be what you want!

You can also get the median with np.median():

>>>

>>> median_ = np.median(y)
>>> median_
4.0
>>> median_ = np.median(y[:-1])
>>> median_
3.25

You’ve obtained the same values with statistics.median() and np.median().

However, if there’s a nan value in your dataset, then np.median() issues the RuntimeWarning and returns nan. If this behavior is not what you want, then you can use nanmedian() to ignore all nan values:

>>>

>>> np.nanmedian(y_with_nan)
4.0
>>> np.nanmedian(y_with_nan[:-1])
3.25

The obtained results are the same as with statistics.median() and np.median() applied to the datasets x and y.

Pandas Series objects have the method .median() that ignores nan values by default:

>>>

>>> z.median()
4.0
>>> z_with_nan.median()
4.0

The behavior of .median() is consistent with .mean() in Pandas. You can change this behavior with the optional parameter skipna.

Mode

The sample mode is the value in the dataset that occurs most frequently. If there isn’t a single such value, then the set is multimodal since it has multiple modal values. For example, in the set that contains the points 2, 3, 2, 8, and 12, the number 2 is the mode because it occurs twice, unlike the other items that occur only once.

This is how you can get the mode with pure Python:

>>>

>>> u = [2, 3, 2, 8, 12]
>>> mode_ = max((u.count(item), item) for item in set(u))[1]
>>> mode_
2

You use u.count() to get the number of occurrences of each item in u. The item with the maximal number of occurrences is the mode. Note that you don’t have to use set(u). Instead, you might replace it with just u and iterate over the entire list.

Note: set(u) returns a Python set with all unique items in u. You can use this trick to optimize working with larger data, especially when you expect to see a lot of duplicates.

You can obtain the mode with statistics.mode() and statistics.multimode():

>>>

>>> mode_ = statistics.mode(u)
>>> mode_
>>> mode_ = statistics.multimode(u)
>>> mode_
[2]

As you can see, mode() returned a single value, while multimode() returned the list that contains the result. This isn’t the only difference between the two functions, though. If there’s more than one modal value, then mode() raises StatisticsError, while multimode() returns the list with all modes:

>>>

>>> v = [12, 15, 12, 15, 21, 15, 12]
>>> statistics.mode(v)  # Raises StatisticsError
>>> statistics.multimode(v)
[12, 15]

You should pay special attention to this scenario and be careful when you’re choosing between these two functions.

statistics.mode() and statistics.multimode() handle nan values as regular values and can return nan as the modal value:

>>>

>>> statistics.mode([2, math.nan, 2])
2
>>> statistics.multimode([2, math.nan, 2])
[2]
>>> statistics.mode([2, math.nan, 0, math.nan, 5])
nan
>>> statistics.multimode([2, math.nan, 0, math.nan, 5])
[nan]

In the first example above, the number 2 occurs twice and is the modal value. In the second example, nan is the modal value since it occurs twice, while the other values occur only once.

Note: statistics.multimode() is introduced in Python 3.8.

You can also get the mode with scipy.stats.mode():

>>>

>>> u, v = np.array(u), np.array(v)
>>> mode_ = scipy.stats.mode(u)
>>> mode_
ModeResult(mode=array([2]), count=array([2]))
>>> mode_ = scipy.stats.mode(v)
>>> mode_
ModeResult(mode=array([12]), count=array([3]))

This function returns the object with the modal value and the number of times it occurs. If there are multiple modal values in the dataset, then only the smallest value is returned.

You can get the mode and its number of occurrences as NumPy arrays with dot notation:

>>>

>>> mode_.mode
array([12])
>>> mode_.count
array([3])

This code uses .mode to return the smallest mode (12) in the array v and .count to return the number of times it occurs (3). scipy.stats.mode() is also flexible with nan values. It allows you to define desired behavior with the optional parameter nan_policy. This parameter can take on the values 'propagate', 'raise' (an error), or 'omit'.

Pandas Series objects have the method .mode() that handles multimodal values well and ignores nan values by default:

>>>

>>> u, v, w = pd.Series(u), pd.Series(v), pd.Series([2, 2, math.nan])
>>> u.mode()
0    2
dtype: int64
>>> v.mode()
0    12
1    15
dtype: int64
>>> w.mode()
0    2.0
dtype: float64

As you can see, .mode() returns a new pd.Series that holds all modal values. If you want .mode() to take nan values into account, then just pass the optional argument dropna=False.

Measures of Variability

The measures of central tendency aren’t sufficient to describe data. You’ll also need the measures of variability that quantify the spread of data points. In this section, you’ll learn how to identify and calculate the following variability measures:

Variance
Standard deviation
Skewness
Percentiles
Ranges

Variance

The sample variance quantifies the spread of the data. It shows numerically how far the data points are from the mean. You can express the sample variance of the dataset 𝑥 with 𝑛 elements mathematically as 𝑠² = Σᵢ(𝑥ᵢ − mean(𝑥))² / (𝑛 − 1), where 𝑖 = 1, 2, …, 𝑛 and mean(𝑥) is the sample mean of 𝑥. If you want to understand deeper why you divide the sum with 𝑛 − 1 instead of 𝑛, then you can dive deeper into Bessel’s correction.

The following figure shows you why it’s important to consider the variance when describing datasets:

There are two datasets in this figure:

Green dots: This dataset has a smaller variance or a smaller average difference from the mean. It also has a smaller range or a smaller difference between the largest and smallest item.
White dots: This dataset has a larger variance or a larger average difference from the mean. It also has a bigger range or a bigger difference between the largest and smallest item.

Note that these two datasets have the same mean and median, even though they appear to differ significantly. Neither the mean nor the median can describe this difference. That’s why you need the measures of variability.

Here’s how you can calculate the sample variance with pure Python:

>>>

>>> n = len(x)
>>> mean_ = sum(x) / n
>>> var_ = sum((item - mean_)**2 for item in x) / (n - 1)
>>> var_
123.19999999999999

This approach is sufficient and calculates the sample variance well. However, the shorter and more elegant solution is to call the existing function statistics.variance():

>>>

>>> var_ = statistics.variance(x)
>>> var_
123.2

You’ve obtained the same result for the variance as above. variance() can avoid calculating the mean if you provide the mean explicitly as the second argument: statistics.variance(x, mean_).

If you have nan values among your data, then statistics.variance() will return nan:

>>>

>>> statistics.variance(x_with_nan)
nan

This behavior is consistent with mean() and most other functions from the Python statistics library.

You can also calculate the sample variance with NumPy. You should use the function np.var() or the corresponding method .var():

>>>

>>> var_ = np.var(y, ddof=1)
>>> var_
123.19999999999999
>>> var_ = y.var(ddof=1)
>>> var_
123.19999999999999

It’s very important to specify the parameter ddof=1. That’s how you set the delta degrees of freedom to 1. This parameter allows the proper calculation of 𝑠², with (𝑛 − 1) in the denominator instead of 𝑛.

If you have nan values in the dataset, then np.var() and .var() will return nan:

>>>

>>> np.var(y_with_nan, ddof=1)
nan
>>> y_with_nan.var(ddof=1)
nan

This is consistent with np.mean() and np.average(). If you want to skip nan values, then you should use np.nanvar():

>>>

>>> np.nanvar(y_with_nan, ddof=1)
123.19999999999999

np.nanvar() ignores nan values. It also needs you to specify ddof=1.

pd.Series objects have the method .var() that skips nan values by default:

>>>

>>> z.var(ddof=1)
123.19999999999999
>>> z_with_nan.var(ddof=1)
123.19999999999999

It also has the parameter ddof, but its default value is 1, so you can omit it. If you want a different behavior related to nan values, then use the optional parameter skipna.

You calculate the population variance similarly to the sample variance. However, you have to use 𝑛 in the denominator instead of 𝑛 − 1: Σᵢ(𝑥ᵢ − mean(𝑥))² / 𝑛. In this case, 𝑛 is the number of items in the entire population. You can get the population variance similar to the sample variance, with the following differences:

Replace (n - 1) with n in the pure Python implementation.
Use statistics.pvariance() instead of statistics.variance().
Specify the parameter ddof=0 if you use NumPy or Pandas. In NumPy, you can omit ddof because its default value is 0.

Note that you should always be aware of whether you’re working with a sample or the entire population whenever you’re calculating the variance!

Standard Deviation

The sample standard deviation is another measure of data spread. It’s connected to the sample variance, as standard deviation, 𝑠, is the positive square root of the sample variance. The standard deviation is often more convenient than the variance because it has the same unit as the data points. Once you get the variance, you can calculate the standard deviation with pure Python:

>>>

>>> std_ = var_ ** 0.5
>>> std_
11.099549540409285

Although this solution works, you can also use statistics.stdev():

>>>

>>> std_ = statistics.stdev(x)
>>> std_
11.099549540409287

Of course, the result is the same as before. Like variance(), stdev() doesn’t calculate the mean if you provide it explicitly as the second argument: statistics.stdev(x, mean_).

You can get the standard deviation with NumPy in almost the same way. You can use the function std() and the corresponding method .std() to calculate the standard deviation. If there are nan values in the dataset, then they’ll return nan. To ignore nan values, you should use np.nanstd(). You use std(), .std(), and nanstd() from NumPy as you would use var(), .var(), and nanvar():

>>>

>>> np.std(y, ddof=1)
11.099549540409285
>>> y.std(ddof=1)
11.099549540409285
>>> np.std(y_with_nan, ddof=1)
nan
>>> y_with_nan.std(ddof=1)
nan
>>> np.nanstd(y_with_nan, ddof=1)
11.099549540409285

Don’t forget to set the delta degrees of freedom to 1!

pd.Series objects also have the method .std() that skips nan by default:

>>>

>>> z.std(ddof=1)
11.099549540409285
>>> z_with_nan.std(ddof=1)
11.099549540409285

The parameter ddof defaults to 1, so you can omit it. Again, if you want to treat nan values differently, then apply the parameter skipna.

The population standard deviation refers to the entire population. It’s the positive square root of the population variance. You can calculate it just like the sample standard deviation, with the following differences:

Find the square root of the population variance in the pure Python implementation.
Use statistics.pstdev() instead of statistics.stdev().
Specify the parameter ddof=0 if you use NumPy or Pandas. In NumPy, you can omit ddof because its default value is 0.

As you can see, you can determine the standard deviation in Python, NumPy, and Pandas in almost the same way as you determine the variance. You use different but analogous functions and methods with the same arguments.

Skewness

The sample skewness measures the asymmetry of a data sample.

There are several mathematical definitions of skewness. One common expression to calculate the skewness of the dataset 𝑥 with 𝑛 elements is (𝑛² / ((𝑛 − 1)(𝑛 − 2))) (Σᵢ(𝑥ᵢ − mean(𝑥))³ / (𝑛𝑠³)). A simpler expression is Σᵢ(𝑥ᵢ − mean(𝑥))³ 𝑛 / ((𝑛 − 1)(𝑛 − 2)𝑠³), where 𝑖 = 1, 2, …, 𝑛 and mean(𝑥) is the sample mean of 𝑥. The skewness defined like this is called the adjusted Fisher-Pearson standardized moment coefficient.

The previous figure showed two datasets that were quite symmetrical. In other words, their points had similar distances from the mean. In contrast, the following image illustrates two asymmetrical sets:

The first set is represented by the green dots and the second with the white ones. Usually, negative skewness values indicate that there’s a dominant tail on the left side, which you can see with the first set. Positive skewness values correspond to a longer or fatter tail on the right side, which you can see in the second set. If the skewness is close to 0 (for example, between −0.5 and 0.5), then the dataset is considered quite symmetrical.

Once you’ve calculated the size of your dataset n, the sample mean mean_, and the standard deviation std_, you can get the sample skewness with pure Python:

>>>

>>> x = [8.0, 1, 2.5, 4, 28.0]
>>> n = len(x)
>>> mean_ = sum(x) / n
>>> var_ = sum((item - mean_)**2 for item in x) / (n - 1)
>>> std_ = var_ ** 0.5
>>> skew_ = (sum((item - mean_)**3 for item in x)
...          * n / ((n - 1) * (n - 2) * std_**3))
>>> skew_
1.9470432273905929

The skewness is positive, so x has a right-side tail.

You can also calculate the sample skewness with scipy.stats.skew():

>>>

>>> y, y_with_nan = np.array(x), np.array(x_with_nan)
>>> scipy.stats.skew(y, bias=False)
1.9470432273905927
>>> scipy.stats.skew(y_with_nan, bias=False)
nan

The obtained result is the same as the pure Python implementation. The parameter bias is set to False to enable the corrections for statistical bias. The optional parameter nan_policy can take the values 'propagate', 'raise', or 'omit'. It allows you to control how you’ll handle nan values.

Pandas Series objects have the method .skew() that also returns the skewness of a dataset:

>>>

>>> z, z_with_nan = pd.Series(x), pd.Series(x_with_nan)
>>> z.skew()
1.9470432273905924
>>> z_with_nan.skew()
1.9470432273905924

Like other methods, .skew() ignores nan values by default, because of the default value of the optional parameter skipna.

Percentiles

The sample 𝑝 percentile is the element in the dataset such that 𝑝% of the elements in the dataset are less than or equal to that value. Also, (100 − 𝑝)% of the elements are greater than or equal to that value. If there are two such elements in the dataset, then the sample 𝑝 percentile is their arithmetic mean. Each dataset has three quartiles, which are the percentiles that divide the dataset into four parts:

The first quartile is the sample 25th percentile. It divides roughly 25% of the smallest items from the rest of the dataset.
The second quartile is the sample 50th percentile or the median. Approximately 25% of the items lie between the first and second quartiles and another 25% between the second and third quartiles.
The third quartile is the sample 75th percentile. It divides roughly 25% of the largest items from the rest of the dataset.

Each part has approximately the same number of items. If you want to divide your data into several intervals, then you can use statistics.quantiles():

>>>

>>> x = [-5.0, -1.1, 0.1, 2.0, 8.0, 12.8, 21.0, 25.8, 41.0]
>>> statistics.quantiles(x, n=2)
[8.0]
>>> statistics.quantiles(x, n=4, method='inclusive')
[0.1, 8.0, 21.0]

In this example, 8.0 is the median of x, while 0.1 and 21.0 are the sample 25th and 75th percentiles, respectively. The parameter n defines the number of resulting equal-probability percentiles, and method determines how to calculate them.

Note: statistics.quantiles() is introduced in Python 3.8.

You can also use np.percentile() to determine any sample percentile in your dataset. For example, this is how you can find the 5th and 95th percentiles:

>>>

>>> y = np.array(x)
>>> np.percentile(y, 5)
-3.44
>>> np.percentile(y, 95)
34.919999999999995

percentile() takes several arguments. You have to provide the dataset as the first argument and the percentile value as the second. The dataset can be in the form of a NumPy array, list, tuple, or similar data structure. The percentile can be a number between 0 and 100 like in the example above, but it can also be a sequence of numbers:

>>>

>>> np.percentile(y, [25, 50, 75])
array([ 0.1,  8. , 21. ])
>>> np.median(y)
8.0

This code calculates the 25th, 50th, and 75th percentiles all at once. If the percentile value is a sequence, then percentile() returns a NumPy array with the results. The first statement returns the array of quartiles. The second statement returns the median, so you can confirm it’s equal to the 50th percentile, which is 8.0.

If you want to ignore nan values, then use np.nanpercentile() instead:

>>>

>>> y_with_nan = np.insert(y, 2, np.nan)
>>> y_with_nan
array([-5. , -1.1,  nan,  0.1,  2. ,  8. , 12.8, 21. , 25.8, 41. ])
>>> np.nanpercentile(y_with_nan, [25, 50, 75])
array([ 0.1,  8. , 21. ])

That’s how you can avoid nan values.

NumPy also offers you very similar functionality in quantile() and nanquantile(). If you use them, then you’ll need to provide the quantile values as the numbers between 0 and 1 instead of percentiles:

>>>

>>> np.quantile(y, 0.05)
-3.44
>>> np.quantile(y, 0.95)
34.919999999999995
>>> np.quantile(y, [0.25, 0.5, 0.75])
array([ 0.1,  8. , 21. ])
>>> np.nanquantile(y_with_nan, [0.25, 0.5, 0.75])
array([ 0.1,  8. , 21. ])

The results are the same as in the previous examples, but here your arguments are between 0 and 1. In other words, you passed 0.05 instead of 5 and 0.95 instead of 95.

pd.Series objects have the method .quantile():

>>>

>>> z, z_with_nan = pd.Series(y), pd.Series(y_with_nan)
>>> z.quantile(0.05)
-3.44
>>> z.quantile(0.95)
34.919999999999995
>>> z.quantile([0.25, 0.5, 0.75])
0.25     0.1
0.50     8.0
0.75    21.0
dtype: float64
>>> z_with_nan.quantile([0.25, 0.5, 0.75])
0.25     0.1
0.50     8.0
0.75    21.0
dtype: float64

.quantile() also needs you to provide the quantile value as the argument. This value can be a number between 0 and 1 or a sequence of numbers. In the first case, .quantile() returns a scalar. In the second case, it returns a new Series holding the results.

Ranges

The range of data is the difference between the maximum and minimum element in the dataset. You can get it with the function np.ptp():

>>>

>>> np.ptp(y)
46.0
>>> np.ptp(z)
46.0
>>> np.ptp(y_with_nan)
nan
>>> np.ptp(z_with_nan)
46.0

This function returns nan if there are nan values in your NumPy array. If you use a Pandas Series object, then it will return a number.

Alternatively, you can use built-in Python, NumPy, or Pandas functions and methods to calculate the maxima and minima of sequences:

max() and min() from the Python standard library
amax() and amin() from NumPy
nanmax() and nanmin() from NumPy to ignore nan values
.max() and .min() from NumPy
.max() and .min() from Pandas to ignore nan values by default

Here are some examples of how you would use these routines:

>>>

>>> np.amax(y) - np.amin(y)
46.0
>>> np.nanmax(y_with_nan) - np.nanmin(y_with_nan)
46.0
>>> y.max() - y.min()
46.0
>>> z.max() - z.min()
46.0
>>> z_with_nan.max() - z_with_nan.min()
46.0

That’s how you get the range of data.

The interquartile range is the difference between the first and third quartile. Once you calculate the quartiles, you can take their difference:

>>>

>>> quartiles = np.quantile(y, [0.25, 0.75])
>>> quartiles[1] - quartiles[0]
20.9
>>> quartiles = z.quantile([0.25, 0.75])
>>> quartiles[0.75] - quartiles[0.25]
20.9

Note that you access the values in a Pandas Series object with the labels 0.75 and 0.25.

Summary of Descriptive Statistics

SciPy and Pandas offer useful routines to quickly get descriptive statistics with a single function or method call. You can use scipy.stats.describe() like this:

>>>

>>> result = scipy.stats.describe(y, ddof=1, bias=False)
>>> result
DescribeResult(nobs=9, minmax=(-5.0, 41.0), mean=11.622222222222222, variance=228.75194444444446, skewness=0.9249043136685094, kurtosis=0.14770623629658886)

You have to provide the dataset as the first argument. The argument can be a NumPy array, list, tuple, or similar data structure. You can omit ddof=1 since it’s the default and only matters when you’re calculating the variance. You can pass bias=False to force correcting the skewness and kurtosis for statistical bias.

Note: The optional parameter nan_policy can take the values 'propagate' (default), 'raise' (an error), or 'omit'. This parameter allows you to control what’s happening when there are nan values.

describe() returns an object that holds the following descriptive statistics:

nobs: the number of observations or elements in your dataset
minmax: the tuple with the minimum and maximum values of your dataset
mean: the mean of your dataset
variance: the variance of your dataset
skewness: the skewness of your dataset
kurtosis: the kurtosis of your dataset

You can access particular values with dot notation:

>>>

>>> result.nobs
9
>>> result.minmax[0]  # Min
-5.0
>>> result.minmax[1]  # Max
41.0
>>> result.mean
11.622222222222222
>>> result.variance
228.75194444444446
>>> result.skewness
0.9249043136685094
>>> result.kurtosis
0.14770623629658886

With SciPy, you’re just one function call away from a descriptive statistics summary for your dataset.

Pandas has similar, if not better, functionality. Series objects have the method .describe():

>>>

>>> result = z.describe()
>>> result
count     9.000000
mean     11.622222
std      15.124548
min      -5.000000
25%       0.100000
50%       8.000000
75%      21.000000
max      41.000000
dtype: float64

It returns a new Series that holds the following:

count: the number of elements in your dataset
mean: the mean of your dataset
std: the standard deviation of your dataset
min and max: the minimum and maximum values of your dataset
25%, 50%, and 75%: the quartiles of your dataset

If you want the resulting Series object to contain other percentiles, then you should specify the value of the optional parameter percentiles. You can access each item of result with its label:

>>>

>>> result['mean']
11.622222222222222
>>> result['std']
15.12454774346805
>>> result['min']
-5.0
>>> result['max']
41.0
>>> result['25%']
0.1
>>> result['50%']
8.0
>>> result['75%']
21.0

That’s how you can get descriptive statistics of a Series object with a single method call using Pandas.

Measures of Correlation Between Pairs of Data

You’ll often need to examine the relationship between the corresponding elements of two variables in a dataset. Say there are two variables, 𝑥 and 𝑦, with an equal number of elements, 𝑛. Let 𝑥₁ from 𝑥 correspond to 𝑦₁ from 𝑦, 𝑥₂ from 𝑥 to 𝑦₂ from 𝑦, and so on. You can then say that there are 𝑛 pairs of corresponding elements: (𝑥₁, 𝑦₁), (𝑥₂, 𝑦₂), and so on.

You’ll see the following measures of correlation between pairs of data:

Positive correlation exists when larger values of 𝑥 correspond to larger values of 𝑦 and vice versa.
Negative correlation exists when larger values of 𝑥 correspond to smaller values of 𝑦 and vice versa.
Weak or no correlation exists if there is no such apparent relationship.

The following figure shows examples of negative, weak, and positive correlation:

The plot on the left with the red dots shows negative correlation. The plot in the middle with the green dots shows weak correlation. Finally, the plot on the right with the blue dots shows positive correlation.

Note: There’s one important thing you should always have in mind when working with correlation among a pair of variables, and that’s that correlation is not a measure or indicator of causation, but only of association!

The two statistics that measure the correlation between datasets are covariance and the correlation coefficient. Let’s define some data to work with these measures. You’ll create two Python lists and use them to get corresponding NumPy arrays and Pandas Series:

>>>

>>> x = list(range(-10, 11))
>>> y = [0, 2, 2, 2, 2, 3, 3, 6, 7, 4, 7, 6, 6, 9, 4, 5, 5, 10, 11, 12, 14]
>>> x_, y_ = np.array(x), np.array(y)
>>> x__, y__ = pd.Series(x_), pd.Series(y_)

Now that you have the two variables, you can start exploring the relationship between them.

Covariance

The sample covariance is a measure that quantifies the strength and direction of a relationship between a pair of variables:

If the correlation is positive, then the covariance is positive, as well. A stronger relationship corresponds to a higher value of the covariance.
If the correlation is negative, then the covariance is negative, as well. A stronger relationship corresponds to a lower (or higher absolute) value of the covariance.
If the correlation is weak, then the covariance is close to zero.

The covariance of the variables 𝑥 and 𝑦 is mathematically defined as 𝑠ˣʸ = Σᵢ (𝑥ᵢ − mean(𝑥)) (𝑦ᵢ − mean(𝑦)) / (𝑛 − 1), where 𝑖 = 1, 2, …, 𝑛, mean(𝑥) is the sample mean of 𝑥, and mean(𝑦) is the sample mean of 𝑦. It follows that the covariance of two identical variables is actually the variance: 𝑠ˣˣ = Σᵢ(𝑥ᵢ − mean(𝑥))² / (𝑛 − 1) = (𝑠ˣ)² and 𝑠ʸʸ = Σᵢ(𝑦ᵢ − mean(𝑦))² / (𝑛 − 1) = (𝑠ʸ)².

This is how you can calculate the covariance in pure Python:

>>>

>>> n = len(x)
>>> mean_x, mean_y = sum(x) / n, sum(y) / n
>>> cov_xy = (sum((x[k] - mean_x) * (y[k] - mean_y) for k in range(n))
...           / (n - 1))
>>> cov_xy
19.95

First, you have to find the mean of x and y. Then, you apply the mathematical formula for the covariance.

NumPy has the function cov() that returns the covariance matrix:

>>>

>>> cov_matrix = np.cov(x_, y_)
>>> cov_matrix
array([[38.5       , 19.95      ],
       [19.95      , 13.91428571]])

Note that cov() has the optional parameters bias, which defaults to False, and ddof, which defaults to None. Their default values are suitable for getting the sample covariance matrix. The upper-left element of the covariance matrix is the covariance of x and x, or the variance of x. Similarly, the lower-right element is the covariance of y and y, or the variance of y. You can check to see that this is true:

>>>

>>> x_.var(ddof=1)
38.5
>>> y_.var(ddof=1)
13.914285714285711

As you can see, the variances of x and y are equal to cov_matrix[0, 0] and cov_matrix[1, 1], respectively.

The other two elements of the covariance matrix are equal and represent the actual covariance between x and y:

>>>

>>> cov_xy = cov_matrix[0, 1]
>>> cov_xy
19.95
>>> cov_xy = cov_matrix[1, 0]
>>> cov_xy
19.95

You’ve obtained the same value of the covariance with np.cov() as with pure Python.

Pandas Series have the method .cov() that you can use to calculate the covariance:

>>>

>>> cov_xy = x__.cov(y__)
>>> cov_xy
19.95
>>> cov_xy = y__.cov(x__)
>>> cov_xy
19.95

Here, you call .cov() on one Series object and pass the other object as the first argument.

Correlation Coefficient

The correlation coefficient, or Pearson product-moment correlation coefficient, is denoted by the symbol 𝑟. The coefficient is another measure of the correlation between data. You can think of it as a standardized covariance. Here are some important facts about it:

The value 𝑟 > 0 indicates positive correlation.
The value 𝑟 < 0 indicates negative correlation.
The value r = 1 is the maximum possible value of 𝑟. It corresponds to a perfect positive linear relationship between variables.
The value r = −1 is the minimum possible value of 𝑟. It corresponds to a perfect negative linear relationship between variables.
The value r ≈ 0, or when 𝑟 is around zero, means that the correlation between variables is weak.

The mathematical formula for the correlation coefficient is 𝑟 = 𝑠ˣʸ / (𝑠ˣ𝑠ʸ) where 𝑠ˣ and 𝑠ʸ are the standard deviations of 𝑥 and 𝑦 respectively. If you have the means (mean_x and mean_y) and standard deviations (std_x, std_y) for the datasets x and y, as well as their covariance cov_xy, then you can calculate the correlation coefficient with pure Python:

>>>

>>> var_x = sum((item - mean_x)**2 for item in x) / (n - 1)
>>> var_y = sum((item - mean_y)**2 for item in y) / (n - 1)
>>> std_x, std_y = var_x ** 0.5, var_y ** 0.5
>>> r = cov_xy / (std_x * std_y)
>>> r
0.861950005631606

You’ve got the variable r that represents the correlation coefficient.

scipy.stats has the routine pearsonr() that calculates the correlation coefficient and the 𝑝-value:

>>>

>>> r, p = scipy.stats.pearsonr(x_, y_)
>>> r
0.861950005631606
>>> p
5.122760847201171e-07

pearsonr() returns a tuple with two numbers. The first one is 𝑟 and the second is the 𝑝-value.

Similar to the case of the covariance matrix, you can apply np.corrcoef() with x_ and y_ as the arguments and get the correlation coefficient matrix:

>>>

>>> corr_matrix = np.corrcoef(x_, y_)
>>> corr_matrix
array([[1.        , 0.86195001],
       [0.86195001, 1.        ]])

The upper-left element is the correlation coefficient between x_ and x_. The lower-right element is the correlation coefficient between y_ and y_. Their values are equal to 1.0. The other two elements are equal and represent the actual correlation coefficient between x_ and y_:

>>>

>>> r = corr_matrix[0, 1]
>>> r
0.8619500056316061
>>> r = corr_matrix[1, 0]
>>> r
0.861950005631606

Of course, the result is the same as with pure Python and pearsonr().

You can get the correlation coefficient with scipy.stats.linregress():

>>>

>>> scipy.stats.linregress(x_, y_)
LinregressResult(slope=0.5181818181818181, intercept=5.714285714285714, rvalue=0.861950005631606, pvalue=5.122760847201164e-07, stderr=0.06992387660074979)

linregress() takes x_ and y_, performs linear regression, and returns the results. slope and intercept define the equation of the regression line, while rvalue is the correlation coefficient. To access particular values from the result of linregress(), including the correlation coefficient, use dot notation:

>>>

>>> result = scipy.stats.linregress(x_, y_)
>>> r = result.rvalue
>>> r
0.861950005631606

That’s how you can perform linear regression and obtain the correlation coefficient.

Pandas Series have the method .corr() for calculating the correlation coefficient:

>>>

>>> r = x__.corr(y__)
>>> r
0.8619500056316061
>>> r = y__.corr(x__)
>>> r
0.861950005631606

You should call .corr() on one Series object and pass the other object as the first argument.

Working With 2D Data

Statisticians often work with 2D data. Here are some examples of 2D data formats:

Database tables
CSV files
Excel, Calc, and Google spreadsheets

NumPy and SciPy provide a comprehensive means to work with 2D data. Pandas has the class DataFrame specifically to handle 2D labeled data.

Axes

Start by creating a 2D NumPy array:

>>>

>>> a = np.array([[1, 1, 1],
...               [2, 3, 1],
...               [4, 9, 2],
...               [8, 27, 4],
...               [16, 1, 1]])
>>> a
array([[ 1,  1,  1],
       [ 2,  3,  1],
       [ 4,  9,  2],
       [ 8, 27,  4],
       [16,  1,  1]])

Now you have a 2D dataset, which you’ll use in this section. You can apply Python statistics functions and methods to it just as you would to 1D data:

>>>

>>> np.mean(a)
5.4
>>> a.mean()
5.4
>>> np.median(a)
2.0
>>> a.var(ddof=1)
53.40000000000001

As you can see, you get statistics (like the mean, median, or variance) across all data in the array a. Sometimes, this behavior is what you want, but in some cases, you’ll want these quantities calculated for each row or column of your 2D array.

The functions and methods you’ve used so far have one optional parameter called axis, which is essential for handling 2D data. axis can take on any of the following values:

axis=None says to calculate the statistics across all data in the array. The examples above work like this. This behavior is often the default in NumPy.
axis=0 says to calculate the statistics across all rows, that is, for each column of the array. This behavior is often the default for SciPy statistical functions.
axis=1 says to calculate the statistics across all columns, that is, for each row of the array.

Let’s see axis=0 in action with np.mean():

>>>

>>> np.mean(a, axis=0)
array([6.2, 8.2, 1.8])
>>> a.mean(axis=0)
array([6.2, 8.2, 1.8])

The two statements above return new NumPy arrays with the mean for each column of a. In this example, the mean of the first column is 6.2. The second column has the mean 8.2, while the third has 1.8.

If you provide axis=1 to mean(), then you’ll get the results for each row:

>>>

>>> np.mean(a, axis=1)
array([ 1.,  2.,  5., 13.,  6.])
>>> a.mean(axis=1)
array([ 1.,  2.,  5., 13.,  6.])

As you can see, the first row of a has the mean 1.0, the second 2.0, and so on.

Note: You can extend these rules to multi-dimensional arrays, but that’s beyond the scope of this tutorial. Feel free to dive into this topic on your own!

The parameter axis works the same way with other NumPy functions and methods:

>>>

>>> np.median(a, axis=0)
array([4., 3., 1.])
>>> np.median(a, axis=1)
array([1., 2., 4., 8., 1.])
>>> a.var(axis=0, ddof=1)
array([ 37.2, 121.2,   1.7])
>>> a.var(axis=1, ddof=1)
array([  0.,   1.,  13., 151.,  75.])

You’ve got the medians and sample variations for all columns (axis=0) and rows (axis=1) of the array a.

This is very similar when you work with SciPy statistics functions. But remember that in this case, the default value for axis is 0:

>>>

>>> scipy.stats.gmean(a)  # Default: axis=0
array([4.        , 3.73719282, 1.51571657])
>>> scipy.stats.gmean(a, axis=0)
array([4.        , 3.73719282, 1.51571657])

If you omit axis or provide axis=0, then you’ll get the result across all rows, that is, for each column. For example, the first column of a has a geometric mean of 4.0, and so on.

If you specify axis=1, then you’ll get the calculations across all columns, that is for each row:

>>>

>>> scipy.stats.gmean(a, axis=1)
array([1.        , 1.81712059, 4.16016765, 9.52440631, 2.5198421 ])

In this example, the geometric mean of the first row of a is 1.0. For the second row, it’s approximately 1.82, and so on.

If you want statistics for the entire dataset, then you have to provide axis=None:

>>>

>>> scipy.stats.gmean(a, axis=None)
2.829705017016332

The geometric mean of all the items in the array a is approximately 2.83.

You can get a Python statistics summary with a single function call for 2D data with scipy.stats.describe(). It works similar to 1D arrays, but you have to be careful with the parameter axis:

>>>

>>> scipy.stats.describe(a, axis=None, ddof=1, bias=False)
DescribeResult(nobs=15, minmax=(1, 27), mean=5.4, variance=53.40000000000001, skewness=2.264965290423389, kurtosis=5.212690982795767)
>>> scipy.stats.describe(a, ddof=1, bias=False)  # Default: axis=0
DescribeResult(nobs=5, minmax=(array([1, 1, 1]), array([16, 27,  4])), mean=array([6.2, 8.2, 1.8]), variance=array([ 37.2, 121.2,   1.7]), skewness=array([1.32531471, 1.79809454, 1.71439233]), kurtosis=array([1.30376344, 3.14969121, 2.66435986]))
>>> scipy.stats.describe(a, axis=1, ddof=1, bias=False)
DescribeResult(nobs=3, minmax=(array([1, 1, 2, 4, 1]), array([ 1,  3,  9, 27, 16])), mean=array([ 1.,  2.,  5., 13.,  6.]), variance=array([  0.,   1.,  13., 151.,  75.]), skewness=array([0.        , 0.        , 1.15206964, 1.52787436, 1.73205081]), kurtosis=array([-3. , -1.5, -1.5, -1.5, -1.5]))

When you provide axis=None, you get the summary across all data. Most results are scalars. If you set axis=0 or omit it, then the return value is the summary for each column. So, most results are the arrays with the same number of items as the number of columns. If you set axis=1, then describe() returns the summary for all rows.

You can get a particular value from the summary with dot notation:

>>>

>>> result = scipy.stats.describe(a, axis=1, ddof=1, bias=False)
>>> result.mean
array([ 1.,  2.,  5., 13.,  6.])

That’s how you can see a statistics summary for a 2D array with a single function call.

DataFrames

The class DataFrame is one of the fundamental Pandas data types. It’s very comfortable to work with because it has labels for rows and columns. Use the array a and create a DataFrame:

>>>

>>> row_names = ['first', 'second', 'third', 'fourth', 'fifth']
>>> col_names = ['A', 'B', 'C']
>>> df = pd.DataFrame(a, index=row_names, columns=col_names)
>>> df
         A   B  C
first    1   1  1
second   2   3  1
third    4   9  2
fourth   8  27  4
fifth   16   1  1

In practice, the names of the columns matter and should be descriptive. The names of the rows are sometimes specified automatically as 0, 1, and so on. You can specify them explicitly with the parameter index, though you’re free to omit index if you like.

DataFrame methods are very similar to Series methods, though the behavior is different. If you call Python statistics methods without arguments, then the DataFrame will return the results for each column:

>>>

>>> df.mean()
A    6.2
B    8.2
C    1.8
dtype: float64
>>> df.var()
A     37.2
B    121.2
C      1.7
dtype: float64

What you get is a new Series that holds the results. In this case, the Series holds the mean and variance for each column. If you want the results for each row, then just specify the parameter axis=1:

>>>

>>> df.mean(axis=1)
first      1.0
second     2.0
third      5.0
fourth    13.0
fifth      6.0
dtype: float64
>>> df.var(axis=1)
first       0.0
second      1.0
third      13.0
fourth    151.0
fifth      75.0
dtype: float64

The result is a Series with the desired quantity for each row. The labels 'first', 'second', and so on refer to the different rows.

You can isolate each column of a DataFrame like this:

>>>

>>> df['A']
first      1
second     2
third      4
fourth     8
fifth     16
Name: A, dtype: int64

Now, you have the column 'A' in the form of a Series object and you can apply the appropriate methods:

>>>

>>> df['A'].mean()
6.2
>>> df['A'].var()
37.20000000000001

That’s how you can obtain the statistics for a single column.

Sometimes, you might want to use a DataFrame as a NumPy array and apply some function to it. It’s possible to get all data from a DataFrame with .values or .to_numpy():

>>>

>>> df.values
array([[ 1,  1,  1],
       [ 2,  3,  1],
       [ 4,  9,  2],
       [ 8, 27,  4],
       [16,  1,  1]])
>>> df.to_numpy()
array([[ 1,  1,  1],
       [ 2,  3,  1],
       [ 4,  9,  2],
       [ 8, 27,  4],
       [16,  1,  1]])

df.values and df.to_numpy() give you a NumPy array with all items from the DataFrame without row and column labels. Note that df.to_numpy() is more flexible because you can specify the data type of items and whether you want to use the existing data or copy it.

Like Series, DataFrame objects have the method .describe() that returns another DataFrame with the statistics summary for all columns:

>>>

>>> df.describe()
              A          B        C
count   5.00000   5.000000  5.00000
mean    6.20000   8.200000  1.80000
std     6.09918  11.009087  1.30384
min     1.00000   1.000000  1.00000
25%     2.00000   1.000000  1.00000
50%     4.00000   3.000000  1.00000
75%     8.00000   9.000000  2.00000
max    16.00000  27.000000  4.00000

The summary contains the following results:

count: the number of items in each column
mean: the mean of each column
std: the standard deviation
min and max: the minimum and maximum values
25%, 50%, and 75%: the percentiles

If you want the resulting DataFrame object to contain other percentiles, then you should specify the value of the optional parameter percentiles.

You can access each item of the summary like this:

>>>

>>> df.describe().at['mean', 'A']
6.2
>>> df.describe().at['50%', 'B']
3.0

That’s how you can get descriptive Python statistics in one Series object with a single Pandas method call.

Visualizing Data

In addition to calculating the numerical quantities like mean, median, or variance, you can use visual methods to present, describe, and summarize data. In this section, you’ll learn how to present your data visually using the following graphs:

Box plots
Histograms
Pie charts
Bar charts
X-Y plots
Heatmaps

matplotlib.pyplot is a very convenient and widely-used library, though it’s not the only Python library available for this purpose. You can import it like this:

>>>

>>> import matplotlib.pyplot as plt
>>> plt.style.use('ggplot')

Now, you have matplotlib.pyplot imported and ready for use. The second statement sets the style for your plots by choosing colors, line widths, and other stylistic elements. You’re free to omit these if you’re satisfied with the default style settings.

Note: This section focuses on representing data and keeps stylistic settings to a minimum. You’ll see links to the official documentation for used routines from matplotlib.pyplot, so you can explore the options that you won’t see here.

You’ll use pseudo-random numbers to get data to work with. You don’t need knowledge on random numbers to be able to understand this section. You just need some arbitrary numbers, and pseudo-random generators are a convenient tool to get them. The module np.random generates arrays of pseudo-random numbers:

Normally distributed numbers are generated with np.random.randn().
Uniformly distributed integers are generated with np.random.randint().

NumPy 1.17 introduced another module for pseudo-random number generation. To learn more about it, check the official documentation.

Box Plots

The box plot is an excellent tool to visually represent descriptive statistics of a given dataset. It can show the range, interquartile range, median, mode, outliers, and all quartiles. First, create some data to represent with a box plot:

>>>

>>> np.random.seed(seed=0)
>>> x = np.random.randn(1000)
>>> y = np.random.randn(100)
>>> z = np.random.randn(10)

The first statement sets the seed of the NumPy random number generator with seed(), so you can get the same results each time you run the code. You don’t have to set the seed, but if you don’t specify this value, then you’ll get different results each time.

The other statements generate three NumPy arrays with normally distributed pseudo-random numbers. x refers to the array with 1000 items, y has 100, and z contains 10 items. Now that you have the data to work with, you can apply .boxplot() to get the box plot:

fig, ax = plt.subplots()
ax.boxplot((x, y, z), vert=False, showmeans=True, meanline=True,
           labels=('x', 'y', 'z'), patch_artist=True,
           medianprops={'linewidth': 2, 'color': 'purple'},
           meanprops={'linewidth': 2, 'color': 'red'})
plt.show()

The parameters of .boxplot() define the following:

x is your data.
vert sets the plot orientation to horizontal when False. The default orientation is vertical.
showmeans shows the mean of your data when True.
meanline represents the mean as a line when True. The default representation is a point.
labels: the labels of your data.
patch_artist determines how to draw the graph.
medianprops denotes the properties of the line representing the median.
meanprops indicates the properties of the line or dot representing the mean.

There are other parameters, but their analysis is beyond the scope of this tutorial.

The code above produces an image like this:

You can see three box plots. Each of them corresponds to a single dataset (x, y, or z) and show the following:

The mean is the red dashed line.
The median is the purple line.
The first quartile is the left edge of the blue rectangle.
The third quartile is the right edge of the blue rectangle.
The interquartile range is the length of the blue rectangle.
The range contains everything from left to right.
The outliers are the dots to the left and right.

A box plot can show so much information in a single figure!

Histograms

Histograms are particularly useful when there are a large number of unique values in a dataset. The histogram divides the values from a sorted dataset into intervals, also called bins. Often, all bins are of equal width, though this doesn’t have to be the case. The values of the lower and upper bounds of a bin are called the bin edges.

The frequency is a single value that corresponds to each bin. It’s the number of elements of the dataset with the values between the edges of the bin. By convention, all bins but the rightmost one are half-open. They include the values equal to the lower bounds, but exclude the values equal to the upper bounds. The rightmost bin is closed because it includes both bounds. If you divide a dataset with the bin edges 0, 5, 10, and 15, then there are three bins:

The first and leftmost bin contains the values greater than or equal to 0 and less than 5.
The second bin contains the values greater than or equal to 5 and less than 10.
The third and rightmost bin contains the values greater than or equal to 10 and less than or equal to 15.

The function np.histogram() is a convenient way to get data for histograms:

>>>

>>> hist, bin_edges = np.histogram(x, bins=10)
>>> hist
array([  9,  20,  70, 146, 217, 239, 160,  86,  38,  15])
>>> bin_edges
array([-3.04614305, -2.46559324, -1.88504342, -1.3044936 , -0.72394379,
       -0.14339397,  0.43715585,  1.01770566,  1.59825548,  2.1788053 ,
        2.75935511])

It takes the array with your data and the number (or edges) of bins and returns two NumPy arrays:

hist contains the frequency or the number of items corresponding to each bin.
bin_edges contains the edges or bounds of the bin.

What histogram() calculates, .hist() can show graphically:

fig, ax = plt.subplots()
ax.hist(x, bin_edges, cumulative=False)
ax.set_xlabel('x')
ax.set_ylabel('Frequency')
plt.show()

The first argument of .hist() is the sequence with your data. The second argument defines the edges of the bins. The third disables the option to create a histogram with cumulative values. The code above produces a figure like this:

You can see the bin edges on the horizontal axis and the frequencies on the vertical axis.

It’s possible to get the histogram with the cumulative numbers of items if you provide the argument cumulative=True to .hist():

fig, ax = plt.subplots()
ax.hist(x, bin_edges, cumulative=True)
ax.set_xlabel('x')
ax.set_ylabel('Frequency')
plt.show()

This code yields the following figure:

It shows the histogram with the cumulative values. The frequency of the first and leftmost bin is the number of items in this bin. The frequency of the second bin is the sum of the numbers of items in the first and second bins. The other bins follow this same pattern. Finally, the frequency of the last and rightmost bin is the total number of items in the dataset (in this case, 1000). You can also directly draw a histogram with pd.Series.hist() using matplotlib in the background.

Pie Charts

Pie charts represent data with a small number of labels and given relative frequencies. They work well even with the labels that can’t be ordered (like nominal data). A pie chart is a circle divided into multiple slices. Each slice corresponds to a single distinct label from the dataset and has an area proportional to the relative frequency associated with that label.

Let’s define data associated to three labels:

>>>

>>> x, y, z = 128, 256, 1024

Now, create a pie chart with .pie():

fig, ax = plt.subplots()
ax.pie((x, y, z), labels=('x', 'y', 'z'), autopct='%1.1f%%')
plt.show()

The first argument of .pie() is your data, and the second is the sequence of the corresponding labels. autopct defines the format of the relative frequencies shown on the figure. You’ll get a figure that looks like this:

The pie chart shows x as the smallest part of the circle, y as the next largest, and then z as the largest part. The percentages denote the relative size of each value compared to their sum.

Bar Charts

Bar charts also illustrate data that correspond to given labels or discrete numeric values. They can show the pairs of data from two datasets. Items of one set are the labels, while the corresponding items of the other are their frequencies. Optionally, they can show the errors related to the frequencies, as well.

The bar chart shows parallel rectangles called bars. Each bar corresponds to a single label and has a height proportional to the frequency or relative frequency of its label. Let’s generate three datasets, each with 21 items:

>>>

>>> x = np.arange(21)
>>> y = np.random.randint(21, size=21)
>>> err = np.random.randn(21)

You use np.arange() to get x, or the array of consecutive integers from 0 to 20. You’ll use this to represent the labels. y is an array of uniformly distributed random integers, also between 0 and 20. This array will represent the frequencies. err contains normally distributed floating-point numbers, which are the errors. These values are optional.

You can create a bar chart with .bar() if you want vertical bars or .barh() if you’d like horizontal bars:

fig, ax = plt.subplots())
ax.bar(x, y, yerr=err)
ax.set_xlabel('x')
ax.set_ylabel('y')
plt.show()

This code should produce the following figure:

The heights of the red bars correspond to the frequencies y, while the lengths of the black lines show the errors err. If you don’t want to include the errors, then omit the parameter yerr of .bar().

X-Y Plots

The x-y plot or scatter plot represents the pairs of data from two datasets. The horizontal x-axis shows the values from the set x, while the vertical y-axis shows the corresponding values from the set y. You can optionally include the regression line and the correlation coefficient. Let’s generate two datasets and perform linear regression with scipy.stats.linregress():

>>>

>>> x = np.arange(21)
>>> y = 5 + 2 * x + 2 * np.random.randn(21)
>>> slope, intercept, r, *__ = scipy.stats.linregress(x, y)
>>> line = f'Regression line: y={intercept:.2f}+{slope:.2f}x, r={r:.2f}'

The dataset x is again the array with the integers from 0 to 20. y is calculated as a linear function of x distorted with some random noise.

linregress returns several values. You’ll need the slope and intercept of the regression line, as well as the correlation coefficient r. Then you can apply .plot() to get the x-y plot:

fig, ax = plt.subplots()
ax.plot(x, y, linewidth=0, marker='s', label='Data points')
ax.plot(x, intercept + slope * x, label=line)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend(facecolor='white')
plt.show()

The result of the code above is this figure:

You can see the data points (x-y pairs) as red squares, as well as the blue regression line.

Heatmaps

A heatmap can be used to visually show a matrix. The colors represent the numbers or elements of the matrix. Heatmaps are particularly useful for illustrating the covariance and correlation matrices. You can create the heatmap for a covariance matrix with .imshow():

matrix = np.cov(x, y).round(decimals=2)
fig, ax = plt.subplots()
ax.imshow(matrix)
ax.grid(False)
ax.xaxis.set(ticks=(0, 1), ticklabels=('x', 'y'))
ax.yaxis.set(ticks=(0, 1), ticklabels=('x', 'y'))
ax.set_ylim(1.5, -0.5)
for i in range(2):
    for j in range(2):
        ax.text(j, i, matrix[i, j], ha='center', va='center', color='w')
plt.show()

Here, the heatmap contains the labels 'x' and 'y' as well as the numbers from the covariance matrix. You’ll get a figure like this:

The yellow field represents the largest element from the matrix 130.34, while the purple one corresponds to the smallest element 38.5. The blue squares in between are associated with the value 69.9.

You can obtain the heatmap for the correlation coefficient matrix following the same logic:

matrix = np.corrcoef(x, y).round(decimals=2)
fig, ax = plt.subplots()
ax.imshow(matrix)
ax.grid(False)
ax.xaxis.set(ticks=(0, 1), ticklabels=('x', 'y'))
ax.yaxis.set(ticks=(0, 1), ticklabels=('x', 'y'))
ax.set_ylim(1.5, -0.5)
for i in range(2):
    for j in range(2):
        ax.text(j, i, matrix[i, j], ha='center', va='center', color='w')
plt.show()

The result is the figure below:

The yellow color represents the value 1.0, and the purple color shows 0.99.

Conclusion

You now know the quantities that describe and summarize datasets and how to calculate them in Python. It’s possible to get descriptive statistics with pure Python code, but that’s rarely necessary. Usually, you’ll use some of the libraries created especially for this purpose:

Use Python’s statistics for the most important Python statistics functions.
Use NumPy to handle arrays efficiently.
Use SciPy for additional Python statistics routines for NumPy arrays.
Use Pandas to work with labeled datasets.
Use Matplotlib to visualize data with plots, charts, and histograms.

In the era of big data and artificial intelligence, you must know how to calculate descriptive statistics measures. Now you’re ready to dive deeper into the world of data science and machine learning! If you have questions or comments, then please put them in the comments section below.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Data Engineer Interview Questions With Python

2019-12-11T14:00:00+00:00

Going to interviews can be a time-consuming and tiring process, and technical interviews can be even more stressful! This tutorial is aimed to prepare you for some common questions you’ll encounter during your data engineer interview. You’ll learn how to answer questions about databases, Python, and SQL.

By the end of this tutorial, you’ll be able to:

Understand common data engineer interview questions
Distinguish between relational and non-relational databases
Set up databases using Python
Use Python for querying data

Becoming a Data Engineer

The data engineering role can be a vast and varied one. You’ll need to have a working knowledge of multiple technologies and concepts. Data engineers are flexible in their thinking. As a result, they can be proficient in multiple topics, such as databases, software development, DevOps, and big data.

What Does a Data Engineer Do?

Given its varied skill set, a data engineering role can span many different job descriptions. A data engineer can be responsible for database design, schema design, and creating multiple database solutions. This work might also involve a Database Administrator.

As a data engineer, you might act as a bridge between the database and the data science teams. In that case, you’ll be responsible for data cleaning and preparation, as well. If big data is involved, then it’s your job to come up with an efficient solution for that data. This work can overlap with the DevOps role.

You’ll also need to make efficient data queries for reporting and analysis. You might need to interact with multiple databases or write Stored Procedures. For many solutions like high-traffic websites or services, there may be more than one database present. In these cases, the data engineer is responsible for setting up the databases, maintaining them, and transferring data between them.

How Can Python Help Data Engineers?

Python is known for being the swiss army knife of programming languages. It’s especially useful in data science, backend systems, and server-side scripting. That’s because Python has strong typing, simple syntax, and an abundance of third-party libraries to use. Pandas, SciPy, Tensorflow, SQLAlchemy, and NumPy are some of the most widely used libraries in production across different industries.

Most importantly, Python decreases development time, which means fewer expenses for companies. For a data engineer, most code execution is database-bound, not CPU-bound. Because of this, it makes sense to capitalize on Python’s simplicity, even at the cost of slower performance when compared to compiled languages such as C# and Java.

Answering Data Engineer Interview Questions

Now that you know what your role might consist of, it’s time to learn how to answer some data engineer interview questions! While there’s a lot of ground to cover, you’ll see practical Python examples throughout the tutorial to guide you along the way.

Questions on Relational Databases

Databases are one of the most crucial components in a system. Without them, there can be no state and no history. While you may not have considered database design to be a priority, know that it can have a significant impact on how quickly your page loads. In the past few years, several large corporations have introduced several new tools and techniques:

NoSQL
Cache databases
Graph databases
NoSQL support in SQL databases

These and other techniques were invented to try and increase the speed at which databases process requests. You’ll likely need to talk about these concepts in your data engineer interview, so let’s go over some questions!

Q1: Relational vs Non-Relational Databases

A relational database is one where data is stored in the form of a table. Each table has a schema, which is the columns and types a record is required to have. Each schema must have at least one primary key that uniquely identifies that record. In other words, there are no duplicate rows in your database. Moreover, each table can be related to other tables using foreign keys.

One important aspect of relational databases is that a change in a schema must be applied to all records. This can sometimes cause breakages and big headaches during migrations. Non-relational databases tackle things in a different way. They are inherently schema-less, which means that records can be saved with different schemas and with a different, nested structure. Records can still have primary keys, but a change in the schema is done on an entry-by-entry basis.

You would need to perform a speed comparison test based on the type of function being performed. You can choose INSERT, UPDATE, DELETE, or another function. Schema design, indices, the number of aggregations, and the number of records will also affect this analysis, so you’ll need to test thoroughly. You’ll learn more about how to do this later on.

Databases also differ in scalability. A non-relational database may be less of a headache to distribute. That’s because a collection of related records can be easily stored on a particular node. On the other hand, relational databases require more thought and usually make use of a master-slave system.

A SQLite Example

Now that you’ve answered what relational databases are, it’s time to dig into some Python! SQLite is a convenient database that you can use on your local machine. The database is a single file, which makes it ideal for prototyping purposes. First, import the required Python library and create a new database:

import sqlite3

db = sqlite3.connect(':memory:')  # Using an in-memory database
cur = db.cursor()

You’re now connected to an in-memory database and have your cursor object ready to go.

Next, you’ll create the following three tables:

Customer: This table will contain a primary key as well as the customer’s first and last names.
Items: This table will contain a primary key, the item name, and the item price.
Items Bought: This table will contain an order number, date, and price. It will also connect to the primary keys in the Items and Customer tables.

Now that you have an idea of what your tables will look like, you can go ahead and create them:

cur.execute('''CREATE TABLE IF NOT EXISTS Customer (
                id integer PRIMARY KEY,
                firstname varchar(255),
                lastname varchar(255) )''')
cur.execute('''CREATE TABLE IF NOT EXISTS Item (
                id integer PRIMARY KEY,
                title varchar(255),
                price decimal )''')
cur.execute('''CREATE TABLE IF NOT EXISTS BoughtItem (
                ordernumber integer PRIMARY KEY,
                customerid integer,
                itemid integer,
                price decimal,
                CONSTRAINT customerid
                    FOREIGN KEY (customerid) REFERENCES Customer(id),
                CONSTRAINT itemid
                    FOREIGN KEY (itemid) REFERENCES Item(id) )''')

You’ve passed a query to cur.execute() to create your three tables.

The last step is to populate your tables with data:

cur.execute('''INSERT INTO Customer(firstname, lastname)
               VALUES ('Bob', 'Adams'),
                      ('Amy', 'Smith'),
                      ('Rob', 'Bennet');''')
cur.execute('''INSERT INTO Item(title, price)
               VALUES ('USB', 10.2),
                      ('Mouse', 12.23),
                      ('Monitor', 199.99);''')
cur.execute('''INSERT INTO BoughtItem(customerid, itemid, price)
               VALUES (1, 1, 10.2),
                      (1, 2, 12.23),
                      (1, 3, 199.99),
                      (2, 3, 180.00),
                      (3, 2, 11.23);''') # Discounted price

Now that there are a few records in each table, you can use this data to answer a few more data engineer interview questions.

Q2: SQL Aggregation Functions

Aggregation functions are those that perform a mathematical operation on a result set. Some examples include AVG, COUNT, MIN, MAX, and SUM. Often, you’ll need GROUP BY and HAVING clauses to complement these aggregations. One useful aggregation function is AVG, which you can use to compute the mean of a given result set:

>>>

>>> cur.execute('''SELECT itemid, AVG(price) FROM BoughtItem GROUP BY itemid''')
>>> print(cur.fetchall())
[(1, 10.2), (2, 11.73), (3, 189.995)]

Here, you’ve retrieved the average price for each of the items bought in your database. You can see that the item with an itemid of 1 has an average price of $10.20.

To make the above output easier to understand, you can display the item name instead of the itemid:

>>>

>>> cur.execute('''SELECT item.title, AVG(boughtitem.price) FROM BoughtItem as boughtitem
...             INNER JOIN Item as item on (item.id = boughtitem.itemid)
...             GROUP BY boughtitem.itemid''')
...
>>> print(cur.fetchall())
[('USB', 10.2), ('Mouse', 11.73), ('Monitor', 189.995)]

Now, you see more easily that the item with an average price of $10.20 is the USB.

Another useful aggregation is SUM. You can use this function to display the total amount of money that each customer spent:

>>>

>>> cur.execute('''SELECT customer.firstname, SUM(boughtitem.price) FROM BoughtItem as boughtitem
...             INNER JOIN Customer as customer on (customer.id = boughtitem.customerid)
...             GROUP BY customer.firstname''')
...
>>> print(cur.fetchall())
[('Amy', 180), ('Bob', 222.42000000000002), ('Rob', 11.23)]

On average, the customer named Amy spent about $180, while Rob only spent $11.23!

If your interviewer likes databases, then you might want to brush up on nested queries, join types, and the steps a relational database takes to perform your query.

Q3: Speeding Up SQL Queries

Speed depends on various factors, but is mostly affected by how many of each of the following are present:

Joins
Aggregations
Traversals
Records

The greater the number of joins, the higher the complexity and the larger the number of traversals in tables. Multiple joins are quite expensive to perform on several thousands of records involving several tables because the database also needs to cache the intermediate result! At this point, you might start to think about how to increase your memory size.

Speed is also affected by whether or not there are indices present in the database. Indices are extremely important and allow you to quickly search through a table and find a match for some column specified in the query.

Indices sort the records at the cost of higher insert time, as well as some storage. Multiple columns can be combined to create a single index. For example, the columns date and price might be combined because your query depends on both conditions.

Q4: Debugging SQL Queries

Most databases include an EXPLAIN QUERY PLAN that describes the steps the database takes to execute the query. For SQLite, you can enable this functionality by adding EXPLAIN QUERY PLAN in front of a SELECT statement:

>>>

>>> cur.execute('''EXPLAIN QUERY PLAN SELECT customer.firstname, item.title, 
...                item.price, boughtitem.price FROM BoughtItem as boughtitem
...                INNER JOIN Customer as customer on (customer.id = boughtitem.customerid)
...                INNER JOIN Item as item on (item.id = boughtitem.itemid)''')
...
>>> print(cur.fetchall())
[(4, 0, 0, 'SCAN TABLE BoughtItem AS boughtitem'), 
(6, 0, 0, 'SEARCH TABLE Customer AS customer USING INTEGER PRIMARY KEY (rowid=?)'), 
(9, 0, 0, 'SEARCH TABLE Item AS item USING INTEGER PRIMARY KEY (rowid=?)')]

This query tries to list the first name, item title, original price, and bought price for all the bought items.

Here’s what the query plan itself looks like:

SCAN TABLE BoughtItem AS boughtitem
SEARCH TABLE Customer AS customer USING INTEGER PRIMARY KEY (rowid=?)
SEARCH TABLE Item AS item USING INTEGER PRIMARY KEY (rowid=?)

Note that fetch statement in your Python code only returns the explanation, but not the results. That’s because EXPLAIN QUERY PLAN is not intended to be used in production.

Questions on Non-Relational Databases

In the previous section, you laid out the differences between relational and non-relational databases and used SQLite with Python. Now you’re going to focus on NoSQL. Your goal is to highlight its strengths, differences, and use cases.

A MongoDB Example

You’ll use the same data as before, but this time your database will be MongoDB. This NoSQL database is document-based and scales very well. First things first, you’ll need to install the required Python library:

$ pip install pymongo

You also might want to install the MongoDB Compass Community. It includes a local IDE that’s perfect for visualizing the database. With it, you can see the created records, create triggers, and act as visual admin for the database.

Note: To run the code in this section, you’ll need a running database server. To learn more about how to set it up, check out Introduction to MongoDB and Python.

Here’s how you create the database and insert some data:

import pymongo

client = pymongo.MongoClient("mongodb://localhost:27017/")

# Note: This database is not created until it is populated by some data
db = client["example_database"]

customers = db["customers"]
items = db["items"]

customers_data = [{ "firstname": "Bob", "lastname": "Adams" },
                  { "firstname": "Amy", "lastname": "Smith" },
                  { "firstname": "Rob", "lastname": "Bennet" },]
items_data = [{ "title": "USB", "price": 10.2 },
              { "title": "Mouse", "price": 12.23 },
              { "title": "Monitor", "price": 199.99 },]

customers.insert_many(customers_data)
items.insert_many(items_data)

As you might have noticed, MongoDB stores data records in collections, which are the equivalent to a list of dictionaries in Python. In practice, MongoDB stores BSON documents.

Q5: Querying Data With MongoDB

Let’s try to replicate the BoughtItem table first, as you did in SQL. To do this, you must append a new field to a customer. MongoDB’s documentation specifies that the keyword operator set can be used to update a record without having to write all the existing fields:

# Just add "boughtitems" to the customer where the firstname is Bob
bob = customers.update_many(
        {"firstname": "Bob"},
        {
            "$set": {
                "boughtitems": [
                    {
                        "title": "USB",
                        "price": 10.2,
                        "currency": "EUR",
                        "notes": "Customer wants it delivered via FedEx",
                        "original_item_id": 1
                    }
                ]
            },
        }
    )

Notice how you added additional fields to the customer without explicitly defining the schema beforehand. Nifty!

In fact, you can update another customer with a slightly altered schema:

amy = customers.update_many(
        {"firstname": "Amy"},
        {
            "$set": {
                "boughtitems":[
                    {
                        "title": "Monitor",
                        "price": 199.99,
                        "original_item_id": 3,
                        "discounted": False
                    }
                ]
            } ,
        }
    )
print(type(amy))  # pymongo.results.UpdateResult

Similar to SQL, document-based databases also allow queries and aggregations to be executed. However, the functionality can differ both syntactically and in the underlying execution. In fact, you might have noticed that MongoDB reserves the $ character to specify some command or aggregation on the records, such as $group. You can learn more about this behavior in the official docs.

You can perform queries just like you did in SQL. To start, you can create an index:

>>>

>>> customers.create_index([("name", pymongo.DESCENDING)])

This is optional, but it speeds up queries that require name lookups.

Then, you can retrieve the customer names sorted in ascending order:

>>>

>>> items = customers.find().sort("name", pymongo.ASCENDING)

You can also iterate through and print the bought items:

>>>

>>> for item in items:
...     print(item.get('boughtitems'))    
...
None
[{'title': 'Monitor', 'price': 199.99, 'original_item_id': 3, 'discounted': False}]
[{'title': 'USB', 'price': 10.2, 'currency': 'EUR', 'notes': 'Customer wants it delivered via FedEx', 'original_item_id': 1}]

You can even retrieve a list of unique names in the database:

>>>

>>> customers.distinct("firstname")
['Bob', 'Amy', 'Rob']

Now that you know the names of the customers in your database, you can create a query to retrieve information about them:

>>>

>>> for i in customers.find({"$or": [{'firstname':'Bob'}, {'firstname':'Amy'}]}, 
...                                  {'firstname':1, 'boughtitems':1, '_id':0}):
...     print(i)
...
{'firstname': 'Bob', 'boughtitems': [{'title': 'USB', 'price': 10.2, 'currency': 'EUR', 'notes': 'Customer wants it delivered via FedEx', 'original_item_id': 1}]}
{'firstname': 'Amy', 'boughtitems': [{'title': 'Monitor', 'price': 199.99, 'original_item_id': 3, 'discounted': False}]}

Here’s the equivalent SQL query:

SELECT firstname, boughtitems FROM customers WHERE firstname LIKE ('Bob', 'Amy')

Note that even though the syntax may differ only slightly, there’s a drastic difference in the way queries are executed underneath the hood. This is to be expected because of the different query structures and use cases between SQL and NoSQL databases.

Q6: NoSQL vs SQL

If you have a constantly changing schema, such as financial regulatory information, then NoSQL can modify the records and nest related information. Imagine the number of joins you’d have to do in SQL if you had eight orders of nesting! However, this situation is more common than you would think.

Now, what if you want to run reports, extract information on that financial data, and infer conclusions? In this case, you need to run complex queries, and SQL tends to be faster in this respect.

Note: SQL databases, particularly PostgreSQL, have also released a feature that allows queryable JSON data to be inserted as part of a record. While this can combine the best of both worlds, speed may be of concern.

It’s faster to query unstructured data from a NoSQL database than it is to query JSON fields from a JSON-type column in PostgreSQL. You can always do a speed comparison test for a definitive answer.

Nonetheless, this feature might reduce the need for an additional database. Sometimes, pickled or serialized objects are stored in records in the form of binary types, and then de-serialized on read.

Speed isn’t the only metric, though. You’ll also want to take into account things like transactions, atomicity, durability, and scalability. Transactions are important in financial applications, and such features take precedence.

Since there’s a wide range of databases, each with its own features, it’s the data engineer’s job to make an informed decision on which database to use in each application. For more information, you can read up on ACID properties relating to database transactions.

You may also be asked what other databases you know of in your data engineer interview. There are several other relevant databases that are used by many companies:

Elastic Search is highly efficient in text search. It leverages its document-based database to create a powerful search tool.
Newt DB combines ZODB and the PostgreSQL JSONB feature to create a Python-friendly NoSQL database.
InfluxDB is used in time-series applications to store events.

The list goes on, but this illustrates how a wide variety of available databases all cater to their niche industry.

Questions on Cache Databases

Cache databases hold frequently accessed data. They live alongside the main SQL and NoSQL databases. Their aim is to alleviate load and serve requests faster.

A Redis Example

You’ve covered SQL and NoSQL databases for long-term storage solutions, but what about faster, more immediate storage? How can a data engineer change how fast data is retrieved from a database?

Typical web-applications retrieve commonly-used data, like a user’s profile or name, very often. If all of the data is contained in one database, then the number of hits the database server gets is going to be over the top and unnecessary. As such, a faster, more immediate storage solution is needed.

While this reduces server load, it also creates two headaches for the data engineer, backend team, and DevOps team. First, you’ll now need some database that has a faster read time than your main SQL or NoSQL database. However, the contents of both databases must eventually match. (Welcome to the problem of state consistency between databases! Enjoy.)

The second headache is that DevOps now needs to worry about scalability, redundancy, and so on for the new cache database. In the next section, you’ll dive into issues like these with the help of Redis.

Q7: How to Use Cache Databases

You may have gotten enough information from the introduction to answer this question! A cache database is a fast storage solution used to store short-lived, structured, or unstructured data. It can be partitioned and scaled according to your needs, but it’s typically much smaller in size than your main database. Because of this, your cache database can reside in memory, allowing you to bypass the need to read from a disk.

Note: If you’ve ever used dictionaries in Python, then Redis follows the same structure. It’s a key-value store, where you can SET and GET data just like a Python dict.

When a request comes in, you first check the cache database, then the main database. This way, you can prevent any unnecessary and repetitive requests from reaching the main database’s server. Since a cache database has a lower read time, you also benefit from a performance increase!

You can use pip to install the required library:

$ pip install redis

Now, consider a request to get the user’s name from their ID:

import redis
from datetime import timedelta

# In a real web application, configuration is obtained from settings or utils
r = redis.Redis()

# Assume this is a getter handling a request
def get_name(request, *args, **kwargs):
    id = request.get('id')
    if id in r:
        return r.get(id)  # Assume that we have an {id: name} store
    else:
        # Get data from the main DB here, assume we already did it
        name = 'Bob'
        # Set the value in the cache database, with an expiration time
        r.setex(id, timedelta(minutes=60), value=name)
        return name

This code checks if the name is in Redis using the id key. If not, then the name is set with an expiration time, which you use because the cache is short-lived.

Now, what if your interviewer asks you what’s wrong with this code? Your response should be that there’s no exception handling! Databases can have many problems, like dropped connections, so it’s always a good idea to try and catch those exceptions.

Questions on Design Patterns and ETL Concepts

In large applications, you’ll often use more than one type of database. In fact, it’s possible to use PostgreSQL, MongoDB, and Redis all within just one application! One challenging problem is dealing with state changes between databases, which exposes the developer to issues of consistency. Consider the following scenario:

A value in Database #1 is updated.
That same value in Database #2 is kept the same (not updated).
A query is run on Database #2.

Now, you’ve got yourself an inconsistent and outdated result! The results returned from the second database won’t reflect the updated value in the first one. This can happen with any two databases, but it’s especially common when the main database is a NoSQL database, and information is transformed into SQL for query purposes.

Databases may have background workers to tackle such problems. These workers extract data from one database, transform it in some way, and load it into the target database. When you’re converting from a NoSQL database to a SQL one, the Extract, transform, load (ETL) process takes the following steps:

Extract: There is a MongoDB trigger whenever a record is created, updated, and so on. A callback function is called asynchronously on a separate thread.
Transform: Parts of the record are extracted, normalized, and put into the correct data structure (or row) to be inserted into SQL.
Load: The SQL database is updated in batches, or as a single record for high volume writes.

This workflow is quite common in financial, gaming, and reporting applications. In these cases, the constantly-changing schema requires a NoSQL database, but reporting, analysis, and aggregations require a SQL database.

Q8: ETL Challenges

There are several challenging concepts in ETL, including the following:

Big data
Stateful problems
Asynchronous workers
Type-matching

The list goes on! However, since the steps in the ETL process are well-defined and logical, the data and backend engineers will typically worry more about performance and availability rather than implementation.

If your application is writing thousands of records per second to MongoDB, then your ETL worker needs to keep up with transforming, loading, and delivering the data to the user in the requested form. Speed and latency can become an issue, so these workers are typically written in fast languages. You can use compiled code for the transform step to speed things up, as this part is usually CPU-bound.

Note: Multi-processing and separation of workers are other solutions that you might want to consider.

If you’re dealing with a lot of CPU-intensive functions, then you might want to check out Numba. This library compiles functions to make them faster on execution. Best of all, this is easily implemented in Python, though there are some limitations on what functions can be used in these compiled functions.

Q9: Design Patterns in Big Data

Imagine Amazon needs to create a recommender system to suggest suitable products to users. The data science team needs data and lots of it! They go to you, the data engineer, and ask you to create a separate staging database warehouse. That’s where they’ll clean up and transform the data.

You might be shocked to receive such a request. When you have terabytes of data, you’ll need multiple machines to handle all of that information. A database aggregation function can be a very complex operation. How can you query, aggregate, and make use of relatively big data in an efficient way?

Apache had initially introduced MapReduce, which follows the map, shuffle, reduce workflow. The idea is to map different data on separate machines, also called clusters. Then, you can perform work on the data, grouped by a key, and finally, aggregate the data in the final stage.

This workflow is still used today, but it’s been fading recently in favor of Spark. The design pattern, however, forms the basis of most big data workflows and is a highly intriguing concept. You can read more on MapReduce at IBM Analytics.

Q10: Common Aspects of the ETL Process and Big Data Workflows

You might think this a rather odd question, but it’s simply a check of your computer science knowledge, as well as your overall design knowledge and experience.

Both workflows follow the Producer-Consumer pattern. A worker (the Producer) produces data of some kind and outputs it to a pipeline. This pipeline can take many forms, including network messages and triggers. After the Producer outputs the data, the Consumer consumes and makes use of it. These workers typically work in an asynchronous manner and are executed in separate processes.

You can liken the Producer to the extract and transform steps of the ETL process. Similarly, in big data, the mapper can be seen as the Producer, while the reducer is effectively the Consumer. This separation of concerns is extremely important and effective in the development and architecture design of applications.

Conclusion

Congratulations! You’ve covered a lot of ground and answered several data engineer interview questions. You now understand a bit more about the many different hats a data engineer can wear, as well as what your responsibilities are with respect to databases, design, and workflow.

Armed with this knowledge, you can now:

Use Python with SQL, NoSQL, and cache databases
Use Python in ETL and query applications
Plan projects ahead of time, keeping design and workflow in mind

While interview questions can be varied, you’ve been exposed to multiple topics and learned to think outside the box in many different areas of computer science. Now you’re ready to have an awesome interview!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Variables in Python

2019-12-10T14:00:00+00:00

If you want to write code that is more complex, then your program will need data that can change as program execution proceeds.

Here’s what you’ll learn in this course:

How every item of data in a Python program can be described by the abstract term object
How to manipulate objects using symbolic names called variables

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

MicroPython: An Intro to Programming Hardware in Python

2019-12-09T14:00:00+00:00

Are you interested in the Internet of Things, home automation, and connected devices? Have you ever wondered what it would be like to build a blaster, a laser sword, or even your own robot? If so, then you’re in luck! MicroPython can help you do all of those things and more.

In this tutorial, you’ll learn about:

The history of MicroPython
The differences between MicroPython and other programming languages
The hardware you’ll use to build devices
The process to set up, code, and deploy your own MicroPython project

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Everything’s Coming Up Python

Python’s popularity has skyrocketed in recent years. These days, it’s used everywhere from DevOps to statistical analysis, and even in desktop software. But for a long time, there was one field where Python use was conspicuously missing. Developers working with microcontrollers had not yet adopted the language.

All of that changed in 2013 when Damien George launched a Kickstarter campaign. Damien, an undergraduate at Cambridge University, was an avid robot programmer. He wanted to move the Python world from machines that worked with capacities in the gigabytes down to the kilobytes. His Kickstarter campaign was an attempt to back his development while he turned his proof of concept into a finished implementation.

Many developers jumped at the chance, not only to use Python on microcontrollers but also to get an early version of Damien’s own reference hardware, which was built especially for the task! In fact, by the end of the campaign, Damien had blown past his £15,000 goal. Thanks to over 1,900 backers, he reached just shy of £100,000.

Python Beyond the Desktop

Finally, Python had moved off of desktops and servers and into the world of sensors, actuators, motors, LCD displays, buttons, and circuits. While this presented many challenges, there were also copious opportunities. Desktop and server hardware requires gigahertz processors, gigabytes of RAM, and terabytes of storage. They also need fully-fledged operating systems, device drivers, and true multitasking.

In the microcontroller world, however, MicroPython is the operating system. Essentially, it sits on top of the processors with clock speeds harking back to the 1980s and 90s. MicroPython has no help to deal with all the intricacies of code execution, IO, storage, booting, and so on. If you want to interface with hardware, then you need all code to enable that.

Yet somehow, Damien managed to squeeze a powerful, high-performing, compact Python implementation down into these miniature computers. That opened up a whole new world of potential. If you’re interested to learn more about the differences between CPython and MicroPython, then you can check out a full breakdown on the official GitHub repo.

Electronics as a Hobby and a Career Path

MicroPython is supported by a diverse community of Pythonistas who have a deep interest in seeing the project succeed. Along with testing and supporting the codebase itself, developers contribute tutorials, code libraries, and hardware ports, taking the project far beyond what Damien could have achieved alone.

Over the years, MicroPython has attracted professionals and hobbyists from other platforms who see the potential and convenience of the language. These developers may have come from more mature platforms, like Arduino or PIC. Many came to realize the benefits of using MicroPython, as well as specifying Python and MicroPython together, not least for rapid prototyping and faster time-to-market.

So, is MicroPython ready for serious, mission-critical work? ESA, the European Space Agency, seems to think so! They helped fund Damien’s work on MicroPython because they wanted to use Python in space. You’ll increasingly see a critical mass of hobbyists and learners move into academia, and from there into the professional world of electronics. At that point, MicroPython will truly take off.

Python and STEM

STEM subjects have been under the education microscope for several years, which has led to an explosion of projects and products targeting the classroom. Python and MicroPython are both well-placed for teachers and students alike. The basic syntax and bundled interpreter lend these languages to an effective learning environment. The fact that no development environment is required is only an added bonus!

A few years ago, the BBC launched its Micro:Bit project to get more kids into computing beyond office-style applications. Their goal was to get a microcontroller board into the hands of every school student in the UK. Damien, through his neighbor, acquired one of these boards and quickly got his code running. Thousands of classrooms suddenly had the option to run Python in their lessons!

Most recently, Edublocks has taken the Micro:Bit and enabled a Scratch-like drag-and-drop Python experience. This allowed even more kids to experience microcontrollers and robotics programming for the first time. Patreon donations support the project’s continued growth.

Physical Computing

When it comes to MicroPython, what does physical computing mean? Usually, your project will consist of three elements:

Input: A button, signal, internet event, or sensor reading takes in data.
Processing: A microcontroller processes the input and updates the output.
Output: This can be sent in the form of a motor spinning, an LED lighting up, a counter changing, a message sending, or some similar event.

These elements will usually be connected by wires and powered by some power source.

What Is a Microcontroller?

You may know that a microcontroller is small and not as powerful as the computer on your desktop or in your server rack. What you may not know is that microcontrollers are all around you, adding intelligence and control to your devices. They’re embedded in everything from appliances and home security setups to pacemakers, to HVAC systems and more.

Microcontrollers do relatively simple things day after day, reliably, and in a compact package. They compress a CPU, memory, and IO into one general-purpose chip, rather than requiring a whole board of chips that team up to perform a task. The code that they run is called firmware, which is flashed or burned onto writeable memory before executing.

While those who grew up with early microcomputers (like the ZX81 and Commodore Vic20) may find these chips unbelievably powerful, technically, you have way more power available in your smartwatch. Still, microcontrollers are extremely useful, even if their processing power and memory capabilities are limited. A great deal can be done with these little guys!

Why MicroPython?

If these microcontrollers were proliferating long before MicroPython was dreamed of, then what benefits does MicroPython afford you over traditional practices?

First, the language is more accessible to beginners than competing languages, while still being powerful enough for industrial use cases. You can go from learning the basics to doing real work, and quickly.

Second, Python allows for rapid feedback. This is because you can interactively enter commands and get a response using the REPL. You could even tweak your code and run it right away, rather than iterating through code-compile-upload-execute cycles.

Last, the wealth of Python code and experience out there means that you can do some things more quickly and easily as a Python programmer. I find the Python requests library, string handling, and tools for dealing with JSON much easier to use in the MicroPython world than C++, for example.

What About C++?

C++ is fast, compact, and available everywhere. There are tons of C++ programmers out there, and a rich community of Arduino and PIC developers ready to help you. Isn’t C++, therefore, a better choice?

In my opinion, MicroPython beats C++ in ease of use and convenience. C++ syntax is not as easy to understand right away. What’s more, the code needs to be compiled and then transferred to your board before you can get a result, so you need a compiler handy.

Now, obviously, the tools are getting better, but MicroPython still has the advantage. While C++ may have a speed advantage, MicroPython is fast enough for most purposes. Besides, if you really need C++, then you can even call your C++ code from MicroPython!

What About Assembler?

For raw performance, there is no beating Assembler. However, that doesn’t rule out MicroPython either.

Again, MicroPython tends to be plenty fast enough. If you do need that lowest-level power, then you can add inline assemblers to your MicroPython projects.

What About BASIC?

If you were to switch on one of those old microcomputers from back in the day, then they would almost certainly have booted into BASIC. At the very least, they would have had a dialect of BASIC on hand. This language served as an introduction to programming for a whole generation. This includes Elon Musk, who apparently learned to code on a Vic 20.

Today, the shine of BASIC has dulled somewhat. Python and other block-based languages have replaced it in educational spaces, and it hardly makes a dent in the tech industry. Python has all the advantages of BASIC, with none of the limitations.

What About Raspberry Pi?

Raspberry Pi, which runs Python, is also heavily present in education. It’s found a niche in electronics and physical computing thanks to its general-purpose IO pins. The Raspberry Pi is also a fully-fledged general-purpose Linux desktop computer. It has lots of processor power, memory, and storage capacity, and it even has a GPU.

However, that last aspect can actually be a reason not to choose the Pi and go for a microcontroller board instead! The ability to run desktop applications and multimedia is fantastic, especially when your projects might need that raw power. This might be the case for AI, video streams, and database projects, for example.

But this can cause problems when your use case requires real-time processing. If you need very precise timing, then you don’t want your code to wait while it catches up with dozens of different processes that all want to execute at the same time.

If you want analog input, then the Pi would require additional hardware. In contrast, most microcontrollers capable of running MicroPython have at least one analog input, and maybe even more. In addition, the Pi is not quite as robust, and it can be more expensive. Because of this, it might make more financial sense to leave a microcontroller in your project, rather than a whole Pi.

You don’t have to choose one or the other. Perhaps pairing a Raspberry Pi with a microcontroller is the best solution for your project. For instance, you could use the Pi for processing power and the microcontroller to interface with hardware.

MicroPython Hardware

If you’re interested in giving MicroPython a try, then that’s great! You’ll need some compatible hardware to install MicroPython onto. Fortunately, there are many options, ranging from affordable to premium products. There’s something for every wallet and use case, so take some time to choose the solution that’s right for you.

Built for Python

The Kickstarter that kicked off MicroPython also launched its associated hardware. The MicroPython Pyboard is now up to v1.1.

The Pyboard is one of the most well-specified boards. It’s based on the STM32 and has lots of GPIO. There’s also an SD slot, an accelerometer, and RTC, with a capability of 168 MHzf. If you can find it in stock, then it’ll cost you around $40 USD.

ESP8266 or ESP32

At the other end of the price scale are boards based on the ESP8266. These boards only have one analog input, and there aren’t as many pins as the Pyboard. However, they do have WiFi capability. You can find them in breadboard-friendly boards (such as the NodeMCU) for $10 or less.

The ESP32 is the big brother to the ESP8266. It increases the power and capabilities while adding Bluetooth to the feature set, for a small additional cost. One of the best versions of these boards is the M5 Stack. This unit comes with a peizo speaker, a battery, a card reader, and a color screen.

BBC Micro:Bit

The Micro:Bit is a compact board based around the Nordic nRF51822 microcontroller. It has built-in Bluetooth LE and temperature sensing, plus an accelerometer, a couple of action buttons, and a 5x5 LED grid.

If you’re in the UK, then you might already have one of these boards. They were distributed to schoolchildren in the hopes of inspiring a new generation of coders. Lots of boards inspired by the Micro:Bit are beginning to appear, so it’s bound to increase in popularity!

Adafruit and CircuitPython-Powered Boards

Soon after MicroPython started picking up the pace, Adafruit produced a fork they call CircuitPython. However, there are a few main differences between the two.

One is that CircuitPython offers support for the Adafruit range of hardware. Another difference is the fact that most Adafruit implementations feature the board appearing as a USB-connected drive. In these cases, adding your code is as simple as dragging it to the disk.

The most feature-rich board of the Adafruit premium line-up is the CircuitPlayground Express, with an optional Crickit add-on. When you combine these two boards, you’ll have pins, sensors, motor drivers, RGB LEDs, and more. If you’re looking for an all-in-one solution, then this is the one to check out.

Unfortunately, Adafruit has dropped compatibility with the ESP8266, even for their own ESP8266-based Feather board. Instead, they’ve chosen to go with the ESP32 purely as a WiFi co-processor in future releases.

MicroPython Workflow

If you’re familiar with Arduino or PIC programming, then you might expect to talk first about compilers, development environments, and the toolchain. However, MicroPython is a little different at first glance.

REPL

With MicroPython, as with Python, the language may have come with your hardware, and you have the option of working with it interactively. There are two ways to get an interactive session:

Use a serial terminal connection: This is often done through the command line, or perhaps an IDE.
Use the WebREPL: This is an option for boards with WiFi.

To get a serial REPL on a Mac, for example, you can run the included terminal program Screen and specify your device and the baud rate:

$ screen /dev/tty.wchusbserial1430 115200

You can find your device by listing the attached serial connections:

$ ls /dev/tty.*

The process is similar to what you would do in the Linux terminal.

On Windows, PuTTY is a popular terminal app. There’s also one called Tera Term. Either way, simply connect to the COM port that appears when you attach your device and select 115,200 as your baud rate.

Once you have a REPL connection, you can enter commands just like you would from your Python interactive session. In MicroPython, this interface is also where you might want to do simple OS-style work, such as deleting files or creating folders.

Command-Line Tools

Dave Hyland’s RShell is a well-featured toolset for working with your MicroPython projects.

There’s another great command-line tool for interacting with MicroPython boards called Ampy. This was initially developed by Adafruit but has now been taken on by a community member, since Adafruit is focusing solely on their own hardware.

MicroPython IDEs

There’s a PyCharm plugin that adds MicroPython support to your IDE. It’s available right in the plugin marketplace:

For the Micro:Bit and CircuitPython boards, the best editor right now is the Mu Editor from Nicholas Tollervey. Otherwise, keep an eye on uPyCraft, which looks to have a lot of promise and is already pretty useful.

Once you have connected the correct serial port and selected your device type, you’ll be able to browse the device:

Your device type will either be ESP8266, ESP32, or Micro:Bit. Now you can interact in the REPL!

Setting Up MicroPython on Your Board

There’s one benefit to choosing uPyCraft, and that’s the option to easily burn the MicroPython firmware to your board without having to use command-line tools. You can choose the uPyCraft default firmware, or a firmware you have downloaded:

The BBC Micro:Bit shows up as a disk drive. When you use Mu or the web-based editor, they’ll produce files that you can drop straight onto that disk. If you’re using an ESP8266 or ESP32 board with uPyCraft, then you can also install the Python ESPTool with pip and set up your board’s firmware using that.

Other boards might come with MicroPython or CircuitPython installed, or even use a drag-and-drop installer. However, those kinds of boards can get corrupted or go into infinite loops. Be sure you take the time to look into your board’s documentation for how to replace the firmware.

Creating and Deploying Your Code

To execute your code, in most cases you’re going to create .py text files and execute them on your MicroPython device. This process is similar to what you would do with CPython. You’ll transfer the files to your board in one of two ways:

Use your IDE. This can be Mu, uPyCraft, or something similar.
Use a command-line tool. This can be RShell, Ampy, or something similar.

When devices that are running MicroPython boot up, they’ll look for a file called boot.py. If that file is found, then the device will execute it automatically. You can also execute .py files from a REPL prompt:

exec(open('my-program.py').read())

In fact, this is similar to what uPyCraft does when you run your edited Python. It transfers the file to your board and then executes it through the REPL.

Writing Your MicroPython Code

You’ll start with a traditional “Hello, World!” project. You can do this in the REPL by connecting to the correct serial port, and entering the following:

  print("Hello World!")

You should see the following output:

You’ve confirmed that your setup is working.

Creating Digital Output

Now, let’s change the output type. Using the following code, you can make the onboard LED blink:

import time
import machine
blueled = machine.Pin(2, machine.Pin.OUT)

# Blink 10 times
for i in range(1,11):
    blueled.value(0)
    time.sleep(0.5)
    blueled.value(1)
    time.sleep(0.5)

print("DONE!")

You import time, which lets you insert time delays. You also import machine, a module that gives you easy access to the IO pins on your board. You use this module to set up an object called blueled, which you define as being Pin 2. (On other boards, it may be possibly Pin 16). Then, you set it as an output pin. You should see the following output:

You may notice that on the ESP8266, the LED lights up when it’s off and turns off when it’s given a positive value. Odd, but true!

Note: Most boards will have one or more built-in LEDs. Check out the documentation for your specific board for more information.

What if you don’t have a built-in LED, or you want to light up an LED that’s not on your board? You’ll need a suitably-sized resistor, such as a 220ohm resistor. You’ll need to connect that to the ground and to the short leg of your LED. The positive, long leg of your LED will connect to the GPIO pin 2. On the Wemos D1 Uno board, GPIO 2 is called Digital Pin 9 on the front face of the board. It has the GPIO number underneath.

In the case of an external LED, it will light up when the value is 1 or on:

You can see the pin has been lit up.

Fading LEDs

In the previous example, you used a for loop, but your hardware can actually blink the LED itself. To do this, set the pin output as PWM, which stands for Pulse Width Modulation (PWM).

This allows you to create a signal of on and off pulses. When you turn an LED on and off very quickly many times a second, it appears that the LED is set to a certain brightness due to how our eyes work. Here’s how your code would look:

from machine import Pin
from machine import PWM
import time

# Set our pin 2 to PWM
pwm = PWM(Pin(2))

# Brightness between 0 and 1023
pwm.duty(700)

# Frequency in Hertz
pwm.freq(1)

In this new code, you set the brightness and the blink frequency in Hertz, and then let the hardware take over.

You can also use PWM to fade the LED in and out:

from machine import Pin
from machine import PWM
import time

# Set our pin 2 to PWM
pwm = PWM(Pin(2))

# Frequency = 100hz
pwm.freq(100)

while 1:
  # Brightness between 0 and 1023
  for brightness in range (0, 1023, 100):
    pwm.duty(brightness)
    print(brightness)
    time.sleep(0.1)

  # Brightness between 1023 and 0
  for brightness in range (1023, 0, -100):
    pwm.duty(brightness)
    print(brightness)
    time.sleep(0.1)

This code gives a pleasing, soft effect:

If you’re using the onboard LED, then the light might instead fade out and then in.

Conclusion

It’s never been more accessible for you to program robots, microcontrollers, electronics, and other hardware. Traditionally, to program these devices, you had to use low-level languages like assembler or C++, and sacrifice a lot of functionality. That all changed with the introduction of MicroPython, a version of Python 3 crammed into the tiny capacity of smaller physical computing devices!

In this tutorial, you dove into MicroPython and the world of electronics hardware. You’ve learned about the history of MicroPython and how it compares to other platforms. You also went through the MicroPython workflow, deployed code to your own board, and brought about a real-world effect.

MicroPython is continuing to grow. Developers in the community are always adding new code, tools, projects, and tutorials. There’s never been a more exciting time to be a MicroPython developer!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Beautiful Soup: Build a Web Scraper With Python

2019-12-04T14:00:00+00:00

The incredible amount of data on the Internet is a rich resource for any field of research or personal interest. To effectively harvest that data, you’ll need to become skilled at web scraping. The Python libraries requests and Beautiful Soup are powerful tools for the job. If you like to learn with hands-on examples and you have a basic understanding of Python and HTML, then this tutorial is for you.

In this tutorial, you’ll learn how to:

Use requests and Beautiful Soup for scraping and parsing data from the Web
Walk through a web scraping pipeline from start to finish
Build a script that fetches job offers from the Web and displays relevant information in your console

This is a powerful project because you’ll be able to apply the same process and the same tools to any static website out there on the World Wide Web. You can download the source code for the project and all examples in this tutorial by clicking on the link below:

Get Sample Code: Click here to get the sample code you'll use for the project and examples in this tutorial.

Let’s get started!

What Is Web Scraping?

Web scraping is the process of gathering information from the Internet. Even copy-pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation. Some websites don’t like it when automatic scrapers gather their data, while others don’t mind.

If you’re scraping a page respectfully for educational purposes, then you’re unlikely to have any problems. Still, it’s a good idea to do some research on your own and make sure that you’re not violating any Terms of Service before you start a large-scale project. To learn more about the legal aspects of web scraping, check out Legal Perspectives on Scraping Data From The Modern Web.

Why Scrape the Web?

Say you’re a surfer (both online and in real life) and you’re looking for employment. However, you’re not looking for just any job. With a surfer’s mindset, you’re waiting for the perfect opportunity to roll your way!

There’s a job site that you like that offers exactly the kinds of jobs you’re looking for. Unfortunately, a new position only pops up once in a blue moon. You think about checking up on it every day, but that doesn’t sound like the most fun and productive way to spend your time.

Thankfully, the world offers other ways to apply that surfer’s mindset! Instead of looking at the job site every day, you can use Python to help automate the repetitive parts of your job search. Automated web scraping can be a solution to speed up the data collection process. You write your code once and it will get the information you want many times and from many pages.

In contrast, when you try to get the information you want manually, you might spend a lot of time clicking, scrolling, and searching. This is especially true if you need large amounts of data from websites that are regularly updated with new content. Manual web scraping can take a lot of time and repetition.

There’s so much information on the Web, and new information is constantly added. Something among all that data is likely of interest to you, and much of it is just out there for the taking. Whether you’re actually on the job hunt, gathering data to support your grassroots organization, or are finally looking to get all the lyrics from your favorite artist downloaded to your computer, automated web scraping can help you accomplish your goals.

Challenges of Web Scraping

The Web has grown organically out of many sources. It combines a ton of different technologies, styles, and personalities, and it continues to grow to this day. In other words, the Web is kind of a hot mess! This can lead to a few challenges you’ll see when you try web scraping.

One challenge is variety. Every website is different. While you’ll encounter general structures that tend to repeat themselves, each website is unique and will need its own personal treatment if you want to extract the information that’s relevant to you.

Another challenge is durability. Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks precisely what you want from your resource of interest. The first time you run your script, it works flawlessly. But when you run the same script only a short while later, you run into a discouraging and lengthy stack of tracebacks!

This is a realistic scenario, as many websites are in active development. Once the site’s structure has changed, your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that many changes to websites are small and incremental, so you’ll likely be able to update your scraper with only minimal adjustments.

However, keep in mind that because the internet is dynamic, the scrapers you’ll build will probably require constant maintenance. You can set up continuous integration to run scraping tests periodically to ensure that your main script doesn’t break without your knowledge.

APIs: An Alternative to Web Scraping

Some website providers offer Application Programming Interfaces (APIs) that allow you to access their data in a predefined manner. With APIs, you can avoid parsing HTML and instead access the data directly using formats like JSON and XML. HTML is primarily a way to visually present content to users.

When you use an API, the process is generally more stable than gathering the data through web scraping. That’s because APIs are made to be consumed by programs, rather than by human eyes. If the design of a website changes, then it doesn’t mean that the structure of the API has changed.

However, APIs can change as well. Both the challenges of variety and durability apply to APIs just as they do to websites. Additionally, it’s much harder to inspect the structure of an API by yourself if the provided documentation is lacking in quality.

The approach and tools you need to gather information using APIs are outside the scope of this tutorial. To learn more about it, check out API Integration in Python.

Scraping the Monster Job Site

In this tutorial, you’ll build a web scraper that fetches Software Developer job listings from the Monster job aggregator site. Your web scraper will parse the HTML to pick out the relevant pieces of information and filter that content for specific words.

You can scrape any site on the Internet that you can look at, but the difficulty of doing so depends on the site. This tutorial offers you an introduction to web scraping to help you understand the overall process. Then, you can apply this same process for every website you’ll want to scrape.

Part 1: Inspect Your Data Source

The first step is to head over to the site you want to scrape using your favorite browser. You’ll need to understand the site structure to extract the information you’re interested in.

Explore the Website

Click through the site and interact with it just like any normal user would. For example, you could search for Software Developer jobs in Australia using the site’s native search interface:

You can see that there’s a list of jobs returned on the left side, and there are more detailed descriptions about the selected job on the right side. When you click on any of the jobs on the left, the content on the right changes. You can also see that when you interact with the website, the URL in your browser’s address bar also changes.

Decipher the Information in URLs

A lot of information can be encoded in a URL. Your web scraping journey will be much easier if you first become familiar with how URLs work and what they’re made of. Try to pick apart the URL of the site you’re currently on:

https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia

You can deconstruct the above URL into two main parts:

The base URL represents the path to the search functionality of the website. In the example above, the base URL is https://www.monster.com/jobs/search/.
The query parameters represent additional values that can be declared on the page. In the example above, the query parameters are ?q=Software-Developer&where=Australia.

Any job you’ll search for on this website will use the same base URL. However, the query parameters will change depending on what you’re looking for. You can think of them as query strings that get sent to the database to retrieve specific records.

Query parameters generally consist of three things:

Start: The beginning of the query parameters is denoted by a question mark (?).
Information: The pieces of information constituting one query parameter are encoded in key-value pairs, where related keys and values are joined together by an equals sign (key=value).
Separator: Every URL can have multiple query parameters, which are separated from each other by an ampersand (&).

Equipped with this information, you can pick apart the URL’s query parameters into two key-value pairs:

q=Software-Developer selects the type of job you’re looking for.
where=Australia selects the location you’re looking for.

Try to change the search parameters and observe how that affects your URL. Go ahead and enter new values in the search bar up top:

Change these values to observe the changes in the URL.

Next, try to change the values directly in your URL. See what happens when you paste the following URL into your browser’s address bar:

https://www.monster.com/jobs/search/?q=Programmer&where=New-York

You’ll notice that changes in the search box of the site are directly reflected in the URL’s query parameters and vice versa. If you change either of them, then you’ll see different results on the website. When you explore URLs, you can get information on how to retrieve data from the website’s server.

Inspect the Site Using Developer Tools

Next, you’ll want to learn more about how the data is structured for display. You’ll need to understand the page structure to pick what you want from the HTML response that you’ll collect in one of the upcoming steps.

Developer tools can help you understand the structure of a website. All modern browsers come with developer tools installed. In this tutorial, you’ll see how to work with the developer tools in Chrome. The process will be very similar to other modern browsers.

In Chrome, you can open up the developer tools through the menu View → Developer → Developer Tools. You can also access them by right-clicking on the page and selecting the Inspect option, or by using a keyboard shortcut.

Developer tools allow you to interactively explore the site’s DOM to better understand the source that you’re working with. To dig into your page’s DOM, select the Elements tab in developer tools. You’ll see a structure with clickable HTML elements. You can expand, collapse, and even edit elements right in your browser:

The HTML on the right represents the structure of the page you can see on the left.

You can think of the text displayed in your browser as the HTML structure of that page. If you’re interested, then you can read more about the difference between the DOM and HTML on CSS-TRICKS.

When you right-click elements on the page, you can select Inspect to zoom to their location in the DOM. You can also hover over the HTML text on your right and see the corresponding elements light up on the page.

Task: Find a single job posting. What HTML element is it wrapped in, and what other HTML elements does it contain?

Play around and explore! The more you get to know the page you’re working with, the easier it will be to scrape it. However, don’t get too overwhelmed with all that HTML text. You’ll use the power of programming to step through this maze and cherry-pick only the interesting parts with Beautiful Soup.

Part 2: Scrape HTML Content From a Page

Now that you have an idea of what you’re working with, it’s time to get started using Python. First, you’ll want to get the site’s HTML code into your Python script so that you can interact with it. For this task, you’ll use Python’s requests library. Type the following in your terminal to install it:

$ pip3 install requests

Then open up a new file in your favorite text editor. All you need to retrieve the HTML are a few lines of code:

import requests

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

This code performs an HTTP request to the given URL. It retrieves the HTML data that the server sends back and stores that data in a Python object.

If you take a look at the downloaded content, then you’ll notice that it looks very similar to the HTML you were inspecting earlier with developer tools. To improve the structure of how the HTML is displayed in your console output, you can print the object’s .content attribute with pprint().

Static Websites

The website you’re scraping in this tutorial serves static HTML content. In this scenario, the server that hosts the site sends back HTML documents that already contain all the data you’ll get to see as a user.

When you inspected the page with developer tools earlier on, you discovered that a job posting consists of the following long and messy-looking HTML:

<section class="card-content" data-jobid="4755ec59-d0db-4ce9-8385-b4df7c1e9f7c" onclick="MKImpressionTrackingMouseDownHijack(this, event)">
<div class="flex-row">
<div class="mux-company-logo thumbnail"></div>
<div class="summary">
<header class="card-header">
<h2 class="title"><a data-bypass="true" data-m_impr_a_placement_id="JSR2CW" data-m_impr_j_cid="4" data-m_impr_j_coc="" data-m_impr_j_jawsid="371676273" data-m_impr_j_jobid="0" data-m_impr_j_jpm="2" data-m_impr_j_jpt="3" data-m_impr_j_lat="30.1882" data-m_impr_j_lid="619" data-m_impr_j_long="-95.6732" data-m_impr_j_occid="11838" data-m_impr_j_p="3" data-m_impr_j_postingid="4755ec59-d0db-4ce9-8385-b4df7c1e9f7c" data-m_impr_j_pvc="4496dab8-a60c-4f02-a2d1-6213320e7213" data-m_impr_s_t="t" data-m_impr_uuid="0b620778-73c7-4550-9db5-df4efad23538" href="https://job-openings.monster.com/python-developer-woodlands-wa-us-lancesoft-inc/4755ec59-d0db-4ce9-8385-b4df7c1e9f7c" onclick="clickJobTitle('plid=619&amp;pcid=4&amp;poccid=11838','Software Developer',''); clickJobTitleSiteCat('{&quot;events.event48&quot;:&quot;true&quot;,&quot;eVar25&quot;:&quot;Python Developer&quot;,&quot;eVar66&quot;:&quot;Monster&quot;,&quot;eVar67&quot;:&quot;JSR2CW&quot;,&quot;eVar26&quot;:&quot;_LanceSoft Inc&quot;,&quot;eVar31&quot;:&quot;Woodlands_WA_&quot;,&quot;prop24&quot;:&quot;2019-07-02T12:00&quot;,&quot;eVar53&quot;:&quot;1500127001001&quot;,&quot;eVar50&quot;:&quot;Aggregated&quot;,&quot;eVar74&quot;:&quot;regular&quot;}')">Python Developer
</a></h2>
</header>
<div class="company">
<span class="name">LanceSoft Inc</span>
<ul class="list-inline">
</ul>
</div>
<div class="location">
<span class="name">
Woodlands, WA
</span>
</div>
</div>
<div class="meta flex-col">
<time datetime="2017-05-26T12:00">2 days ago</time>
<span class="mux-tooltip applied-only" data-mux="tooltip" title="Applied">
<i aria-hidden="true" class="icon icon-applied"></i>
<span class="sr-only">Applied</span>
</span>
<span class="mux-tooltip saved-only" data-mux="tooltip" title="Saved">
<i aria-hidden="true" class="icon icon-saved"></i>
<span class="sr-only">Saved</span>
</span>
</div>
</div>
</section>

It can be difficult to wrap your head around such a long block of HTML code. To make it easier to read, you can use an HTML formatter to automatically clean it up a little more. Good readability helps you better understand the structure of any code block. While it may or may not help to improve the formatting of the HTML, it’s always worth a try.

Note: Keep in mind that every website will look different. That’s why it’s necessary to inspect and understand the structure of the site you’re currently working with before moving forward.

The HTML above definitely has a few confusing parts in it. For example, you can scroll to the right to see the large number of attributes that the <a> element has. Luckily, the class names on the elements that you’re interested in are relatively straightforward:

class="title": the title of the job posting
class="company": the company that offers the position
class="location": the location where you’d be working

In case you ever get lost in a large pile of HTML, remember that you can always go back to your browser and use developer tools to further explore the HTML structure interactively.

By now, you’ve successfully harnessed the power and user-friendly design of Python’s requests library. With only a few lines of code, you managed to scrape the static HTML content from the web and make it available for further processing.

However, there are a few more challenging situations you might encounter when you’re scraping websites. Before you begin using Beautiful Soup to pick the relevant information from the HTML that you just scraped, take a quick look at two of these situations.

Hidden Websites

Some pages contain information that’s hidden behind a login. That means you’ll need an account to be able to see (and scrape) anything from the page. The process to make an HTTP request from your Python script is different than how you access a page from your browser. That means that just because you can log in to the page through your browser, that doesn’t mean you’ll be able to scrape it with your Python script.

However, there are some advanced techniques that you can use with the requests to access the content behind logins. These techniques will allow you to log in to websites while making the HTTP request from within your script.

Dynamic Websites

Static sites are easier to work with because the server sends you an HTML page that already contains all the information as a response. You can parse an HTML response with Beautiful Soup and begin to pick out the relevant data.

On the other hand, with a dynamic website the server might not send back any HTML at all. Instead, you’ll receive JavaScript code as a response. This will look completely different from what you saw when you inspected the page with your browser’s developer tools.

Note: To offload work from the server to the clients’ machines, many modern websites avoid crunching numbers on their servers whenever possible. Instead, they’ll send JavaScript code that your browser will execute locally to produce the desired HTML.

As mentioned before, what happens in the browser is not related to what happens in your script. Your browser will diligently execute the JavaScript code it receives back from a server and create the DOM and HTML for you locally. However, doing a request to a dynamic website in your Python script will not provide you with the HTML page content.

When you use requests, you’ll only receive what the server sends back. In the case of a dynamic website, you’ll end up with some JavaScript code, which you won’t be able to parse using Beautiful Soup. The only way to go from the JavaScript code to the content you’re interested in is to execute the code, just like your browser does. The requests library can’t do that for you, but there are other solutions that can.

For example, requests-html is a project created by the author of the requests library that allows you to easily render JavaScript using syntax that’s similar to the syntax in requests. It also includes capabilities for parsing the data by using Beautiful Soup under the hood.

Note: Another popular choice for scraping dynamic content is Selenium. You can think of Selenium as a slimmed-down browser that executes the JavaScript code for you before passing on the rendered HTML response to your script.

You won’t go deeper into scraping dynamically-generated content in this tutorial. For now, it’s enough for you to remember that you’ll need to look into the above-mentioned options if the page you’re interested in is generated in your browser dynamically.

Part 3: Parse HTML Code With Beautiful Soup

You’ve successfully scraped some HTML from the Internet, but when you look at it now, it just seems like a huge mess. There are tons of HTML elements here and there, thousands of attributes scattered around—and wasn’t there some JavaScript mixed in as well? It’s time to parse this lengthy code response with Beautiful Soup to make it more accessible and pick out the data that you’re interested in.

Beautiful Soup is a Python library for parsing structured data. It allows you to interact with HTML in a similar way to how you would interact with a web page using developer tools. Beautiful Soup exposes a couple of intuitive functions you can use to explore the HTML you received. To get started, use your terminal to install the Beautiful Soup library:

$ pip3 install beautifulsoup4

Then, import the library and create a Beautiful Soup object:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

When you add the two highlighted lines of code, you’re creating a Beautiful Soup object that takes the HTML content you scraped earlier as its input. When you instantiate the object, you also instruct Beautiful Soup to use the appropriate parser.

Find Elements by ID

In an HTML web page, every element can have an id attribute assigned. As the name already suggests, that id attribute makes the element uniquely identifiable on the page. You can begin to parse your page by selecting a specific element by its ID.

Switch back to developer tools and identify the HTML object that contains all of the job postings. Explore by hovering over parts of the page and using right-click to Inspect.

Note: Keep in mind that it’s helpful to periodically switch back to your browser and interactively explore the page using developer tools. This helps you learn how to find the exact elements you’re looking for.

At the time of this writing, the element you’re looking for is a <div> with an id attribute that has the value "ResultsContainer". It has a couple of other attributes as well, but below is the gist of what you’re looking for:

<div id="ResultsContainer">
    <!-- all the job listings -->
</div>

Beautiful Soup allows you to find that specific element easily by its ID:

results = soup.find(id='ResultsContainer')

For easier viewing, you can .prettify() any Beautiful Soup object when you print it out. If you call this method on the results variable that you just assigned above, then you should see all the HTML contained within the <div>:

print(results.prettify())

When you use the element’s ID, you’re able to pick one element out from among the rest of the HTML. This allows you to work with only this specific part of the page’s HTML. It looks like the soup just got a little thinner! However, it’s still quite dense.

Find Elements by HTML Class Name

You’ve seen that every job posting is wrapped in a <section> element with the class card-content. Now you can work with your new Beautiful Soup object called results and select only the job postings. These are, after all, the parts of the HTML that you’re interested in! You can do this in one line of code:

job_elems = results.find_all('section', class_='card-content')

Here, you call .find_all() on a Beautiful Soup object, which returns an iterable containing all the HTML for all the job listings displayed on that page.

Take a look at all of them:

for job_elem in job_elems:
    print(job_elem, end='\n'*2)

That’s already pretty neat, but there’s still a lot of HTML! You’ve seen earlier that your page has descriptive class names on some elements. Let’s pick out only those:

for job_elem in job_elems:
    # Each job_elem is a new BeautifulSoup object.
    # You can use the same methods on it as you did before.
    title_elem = job_elem.find('h2', class_='title')
    company_elem = job_elem.find('div', class_='company')
    location_elem = job_elem.find('div', class_='location')
    print(title_elem)
    print(company_elem)
    print(location_elem)
    print()

Great! You’re getting closer and closer to the data you’re actually interested in. Still, there’s a lot going on with all those HTML tags and attributes floating around:

<h2 class="title"><a data-bypass="true" data-m_impr_a_placement_id="JSR2CW" data-m_impr_j_cid="4" data-m_impr_j_coc="" data-m_impr_j_jawsid="371676273" data-m_impr_j_jobid="0" data-m_impr_j_jpm="2" data-m_impr_j_jpt="3" data-m_impr_j_lat="30.1882" data-m_impr_j_lid="619" data-m_impr_j_long="-95.6732" data-m_impr_j_occid="11838" data-m_impr_j_p="3" data-m_impr_j_postingid="4755ec59-d0db-4ce9-8385-b4df7c1e9f7c" data-m_impr_j_pvc="4496dab8-a60c-4f02-a2d1-6213320e7213" data-m_impr_s_t="t" data-m_impr_uuid="0b620778-73c7-4550-9db5-df4efad23538" href="https://job-openings.monster.com/python-developer-woodlands-wa-us-lancesoft-inc/4755ec59-d0db-4ce9-8385-b4df7c1e9f7c" onclick="clickJobTitle('plid=619&amp;pcid=4&amp;poccid=11838','Software Developer',''); clickJobTitleSiteCat('{&quot;events.event48&quot;:&quot;true&quot;,&quot;eVar25&quot;:&quot;Python Developer&quot;,&quot;eVar66&quot;:&quot;Monster&quot;,&quot;eVar67&quot;:&quot;JSR2CW&quot;,&quot;eVar26&quot;:&quot;_LanceSoft Inc&quot;,&quot;eVar31&quot;:&quot;Woodlands_WA_&quot;,&quot;prop24&quot;:&quot;2019-07-02T12:00&quot;,&quot;eVar53&quot;:&quot;1500127001001&quot;,&quot;eVar50&quot;:&quot;Aggregated&quot;,&quot;eVar74&quot;:&quot;regular&quot;}')">Python Developer
</a></h2>
<div class="company">
<span class="name">LanceSoft Inc</span>
<ul class="list-inline">
</ul>
</div>
<div class="location">
<span class="name">
Woodlands, WA
</span>
</div>

You’ll see how to narrow down this output in the next section.

Extract Text From HTML Elements

For now, you only want to see the title, company, and location of each job posting. And behold! Beautiful Soup has got you covered. You can add .text to a Beautiful Soup object to return only the text content of the HTML elements that the object contains:

for job_elem in job_elems:
    title_elem = job_elem.find('h2', class_='title')
    company_elem = job_elem.find('div', class_='company')
    location_elem = job_elem.find('div', class_='location')
    print(title_elem.text)
    print(company_elem.text)
    print(location_elem.text)
    print()

Run the above code snippet and you’ll see the text content displayed. However, you’ll also get a lot of whitespace. Since you’re now working with Python strings, you can .strip() the superfluous whitespace. You can also apply any other familiar Python string methods to further clean up your text.

Note: The web is messy and you can’t rely on a page structure to be consistent throughout. Therefore, you’ll more often than not run into errors while parsing HTML.

When you run the above code, you might encounter an AttributeError:

AttributeError: 'NoneType' object has no attribute 'text'

If that’s the case, then take a step back and inspect your previous results. Were there any items with a value of None? You might have noticed that the structure of the page is not entirely uniform. There could be an advertisement in there that displays in a different way than the normal job postings, which may return different results. For this tutorial, you can safely disregard the problematic element and skip over it while parsing the HTML:

for job_elem in job_elems:
    title_elem = job_elem.find('h2', class_='title')
    company_elem = job_elem.find('div', class_='company')
    location_elem = job_elem.find('div', class_='location')
    if None in (title_elem, company_elem, location_elem):
        continue
    print(title_elem.text.strip())
    print(company_elem.text.strip())
    print(location_elem.text.strip())
    print()

Feel free to explore why one of the elements is returned as None. You can use the conditional statement you wrote above to print() out and inspect the relevant element in more detail. What do you think is going on there?

After you complete the above steps try running your script again. The results finally look much better:

Python Developer
LanceSoft Inc
Woodlands, WA

Senior Engagement Manager
Zuora
Sydney, NSW

Find Elements by Class Name and Text Content

By now, you’ve cleaned up the list of jobs that you saw on the website. While that’s pretty neat already, you can make your script more useful. However, not all of the job listings seem to be developer jobs that you’d be interested in as a Python developer. So instead of printing out all of the jobs from the page, you’ll first filter them for some keywords.

You know that job titles in the page are kept within <h2> elements. To filter only for specific ones, you can use the string argument:

python_jobs = results.find_all('h2', string='Python Developer')

This code finds all <h2> elements where the contained string matches 'Python Developer' exactly. Note that you’re directly calling the method on your first results variable. If you go ahead and print() the output of the above code snippet to your console, then you might be disappointed because it will probably be empty:

[]

There was definitely a job with that title in the search results, so why is it not showing up? When you use string= like you did above, your program looks for exactly that string. Any differences in capitalization or whitespace will prevent the element from matching. In the next section, you’ll find a way to make the string more general.

Pass a Function to a Beautiful Soup Method

In addition to strings, you can often pass functions as arguments to Beautiful Soup methods. You can change the previous line of code to use a function instead:

python_jobs = results.find_all('h2',
                               string=lambda text: 'python' in text.lower())

Now you’re passing an anonymous function to the string= argument. The lambda function looks at the text of each <h2> element, converts it to lowercase, and checks whether the substring 'python' is found anywhere in there. Now you’ve got a match:

>>>

>>> print(len(python_jobs))
1

Your program has found a match!

Note: In case you still don’t get a match, try adapting your search string. The job offers on this page are constantly changing and there might not be a job listed that includes the substring 'python' in its title at the time that you’re working through this tutorial.

The process of finding specific elements depending on their text content is a powerful way to filter your HTML response for the information that you’re looking for. Beautiful Soup allows you to use either exact strings or functions as arguments for filtering text in Beautiful Soup objects.

Extract Attributes From HTML Elements

At this point, your Python script already scrapes the site and filters its HTML for relevant job postings. Well done! However, one thing that’s still missing is the link to apply for a job.

While you were inspecting the page, you found that the link is part of the element that has the title HTML class. The current code strips away the entire link when accessing the .text attribute of its parent element. As you’ve seen before, .text only contains the visible text content of an HTML element. Tags and attributes are not part of that. To get the actual URL, you want to extract one of those attributes instead of discarding it.

Look at the list of filtered results python_jobs that you created above. The URL is contained in the href attribute of the nested <a> tag. Start by fetching the <a> element. Then, extract the value of its href attribute using square-bracket notation:

python_jobs = results.find_all('h2',
                               string=lambda text: "python" in text.lower())

for p_job in python_jobs:
    link = p_job.find('a')['href']
    print(p_job.text.strip())
    print(f"Apply here: {link}\n")

The filtered results will only show links to job opportunities that include python in their title. You can use the same square-bracket notation to extract other HTML attributes as well. A common use case is to fetch the URL of a link, as you did above.

Building the Job Search Tool

If you’ve written the code alongside this tutorial, then you can already run your script as-is. To wrap up your journey into web scraping, you could give the code a final makeover and create a command line interface app that looks for Software Developer jobs in any location you define.

You can check out a command line app version of the code you built in this tutorial at the link below:

Get Sample Code: Click here to get the sample code you'll use for the project and examples in this tutorial.

If you’re interested in learning how to adapt your script as a command line interface, then check out How to Build Command Line Interfaces in Python With argparse.

Additional Practice

Below is a list of other job boards. These linked pages also return their search results as static HTML responses. To keep practicing your new skills, you can revisit the web scraping process using any or all of the following sites:

Go through this tutorial again from the top using one of these other sites. You’ll see that the structure of each website is different and that you’ll need to re-build the code in a slightly different way to fetch the data you want. This is a great way to practice the concepts that you just learned. While it might make you sweat every so often, your coding skills will be stronger for it!

During your second attempt, you can also explore additional features of Beautiful Soup. Use the documentation as your guidebook and inspiration. Additional practice will help you become more proficient at web scraping using Python, requests, and Beautiful Soup.

Conclusion

Beautiful Soup is packed with useful functionality to parse HTML data. It’s a trusted and helpful companion for your web scraping adventures. Its documentation is comprehensive and relatively user-friendly to get started with. You’ll find that Beautiful Soup will cater to most of your parsing needs, from navigating to advanced searching through the results.

In this tutorial, you’ve learned how to scrape data from the Web using Python, requests, and Beautiful Soup. You built a script that fetches job postings from the Internet and went through the full web scraping process from start to finish.

You learned how to:

Inspect the HTML structure of your target site with your browser’s developer tools
Gain insight into how to decipher the data encoded in URLs
Download the page’s HTML content using Python’s requests library
Parse the downloaded HTML with Beautiful Soup to extract relevant information

With this general pipeline in mind and powerful libraries in your toolkit, you can go out and see what other websites you can scrape! Have fun, and remember to always be respectful and use your programming skills responsibly.

You can download the source code for the sample script that you built in this tutorial by clicking on the link below:

Get Sample Code: Click here to get the sample code you'll use for the project and examples in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python, Boto3, and AWS S3: Demystified

2019-12-03T14:00:00+00:00

Amazon Web Services (AWS) has become a leader in cloud computing. One of its core components is S3, the object storage service offered by AWS. With its impressive availability and durability, it has become the standard way to store videos, images, and data. You can combine S3 with other services to build infinitely scalable applications.

Boto3 is the name of the Python SDK for AWS. It allows you to directly create, update, and delete AWS resources from your Python scripts.

If you’ve had some AWS exposure before, have your own AWS account, and want to take your skills to the next level by starting to use AWS services from within your Python code, then keep watching.

By the end of this course, you’ll:

Be confident working with buckets and objects directly from your Python scripts
Know how to avoid common pitfalls when using Boto3 and S3
Understand how to set up your data from the start to avoid performance issues later
Learn how to configure your objects to take advantage of S3’s best features

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Pandas: How to Read and Write Files

2019-12-02T14:00:00+00:00

Pandas is a powerful and flexible Python package that allows you to work with labeled and time series data. It also provides statistics methods, enables plotting, and more. One crucial feature of Pandas is its ability to write and read Excel, CSV, and many other types of files. Functions like the Pandas read_csv() method enable you to work with files effectively. You can use them to save the data and labels from Pandas objects to a file and load them later as Pandas Series or DataFrame instances.

In this tutorial, you’ll learn:

What the Pandas IO tools API is
How to read and write data to and from files
How to work with various file formats
How to work with big data efficiently

Let’s start reading and writing files!

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Installing Pandas

The code in this tutorial is executed with CPython 3.7.4 and Pandas 0.25.1. It would be beneficial to make sure you have the latest versions of Python and Pandas on your machine. You might want to create a new virtual environment and install the dependencies for this tutorial.

First, you’ll need the Pandas library. You may already have it installed. If you don’t, then you can install it with pip:

$ pip install pandas

Once the installation process completes, you should have Pandas installed and ready.

Anaconda is an excellent Python distribution that comes with Python, many useful packages like Pandas, and a package and environment manager called Conda. To learn more about Anaconda, check out Setting Up Python for Machine Learning on Windows.

If you don’t have Pandas in your virtual environment, then you can install it with Conda:

$ conda install pandas

Conda is powerful as it manages the dependencies and their versions. To learn more about working with Conda, you can check out the official documentation.

Preparing Data

In this tutorial, you’ll use the data related to 20 countries. Here’s an overview of the data and sources you’ll be working with:

Country is denoted by the country name. Each country is in the top 10 list for either population, area, or gross domestic product (GDP). The row labels for the dataset are the three-letter country codes defined in ISO 3166-1. The column label for the dataset is COUNTRY.
Population is expressed in millions. The data comes from a list of countries and dependencies by population on Wikipedia. The column label for the dataset is POP.
Area is expressed in thousands of kilometers squared. The data comes from a list of countries and dependencies by area on Wikipedia. The column label for the dataset is AREA.
Gross domestic product is expressed in millions of U.S. dollars, according to the United Nations data for 2017. You can find this data in the list of countries by nominal GDP on Wikipedia. The column label for the dataset is GDP.
Continent is either Africa, Asia, Oceania, Europe, North America, or South America. You can find this information on Wikipedia as well. The column label for the dataset is CONT.
Independence day is a date that commemorates a nation’s independence. The data comes from the list of national independence days on Wikipedia. The dates are shown in ISO 8601 format. The first four digits represent the year, the next two numbers are the month, and the last two are for the day of the month. The column label for the dataset is IND_DAY.

This is how the data looks as a table:

	COUNTRY	POP	AREA	GDP	CONT	IND_DAY
CHN	China	1398.72	9596.96	12234.78	Asia
IND	India	1351.16	3287.26	2575.67	Asia	1947-08-15
USA	US	329.74	9833.52	19485.39	N.America	1776-07-04
IDN	Indonesia	268.07	1910.93	1015.54	Asia	1945-08-17
BRA	Brazil	210.32	8515.77	2055.51	S.America	1822-09-07
PAK	Pakistan	205.71	881.91	302.14	Asia	1947-08-14
NGA	Nigeria	200.96	923.77	375.77	Africa	1960-10-01
BGD	Bangladesh	167.09	147.57	245.63	Asia	1971-03-26
RUS	Russia	146.79	17098.25	1530.75		1992-06-12
MEX	Mexico	126.58	1964.38	1158.23	N.America	1810-09-16
JPN	Japan	126.22	377.97	4872.42	Asia
DEU	Germany	83.02	357.11	3693.20	Europe
FRA	France	67.02	640.68	2582.49	Europe	1789-07-14
GBR	UK	66.44	242.50	2631.23	Europe
ITA	Italy	60.36	301.34	1943.84	Europe
ARG	Argentina	44.94	2780.40	637.49	S.America	1816-07-09
DZA	Algeria	43.38	2381.74	167.56	Africa	1962-07-05
CAN	Canada	37.59	9984.67	1647.12	N.America	1867-07-01
AUS	Australia	25.47	7692.02	1408.68	Oceania
KAZ	Kazakhstan	18.53	2724.90	159.41	Asia	1991-12-16

You may notice that some of the data is missing. For example, the continent for Russia is not specified because it spreads across both Europe and Asia. There are also several missing independence days because the data source omits them.

You can organize this data in Python using a nested dictionary:

data = {
    'CHN': {'COUNTRY': 'China', 'POP': 1_398.72, 'AREA': 9_596.96,
            'GDP': 12_234.78, 'CONT': 'Asia'},
    'IND': {'COUNTRY': 'India', 'POP': 1_351.16, 'AREA': 3_287.26,
            'GDP': 2_575.67, 'CONT': 'Asia', 'IND_DAY': '1947-08-15'},
    'USA': {'COUNTRY': 'US', 'POP': 329.74, 'AREA': 9_833.52,
            'GDP': 19_485.39, 'CONT': 'N.America',
            'IND_DAY': '1776-07-04'},
    'IDN': {'COUNTRY': 'Indonesia', 'POP': 268.07, 'AREA': 1_910.93,
            'GDP': 1_015.54, 'CONT': 'Asia', 'IND_DAY': '1945-08-17'},
    'BRA': {'COUNTRY': 'Brazil', 'POP': 210.32, 'AREA': 8_515.77,
            'GDP': 2_055.51, 'CONT': 'S.America', 'IND_DAY': '1822-09-07'},
    'PAK': {'COUNTRY': 'Pakistan', 'POP': 205.71, 'AREA': 881.91,
            'GDP': 302.14, 'CONT': 'Asia', 'IND_DAY': '1947-08-14'},
    'NGA': {'COUNTRY': 'Nigeria', 'POP': 200.96, 'AREA': 923.77,
            'GDP': 375.77, 'CONT': 'Africa', 'IND_DAY': '1960-10-01'},
    'BGD': {'COUNTRY': 'Bangladesh', 'POP': 167.09, 'AREA': 147.57,
            'GDP': 245.63, 'CONT': 'Asia', 'IND_DAY': '1971-03-26'},
    'RUS': {'COUNTRY': 'Russia', 'POP': 146.79, 'AREA': 17_098.25,
            'GDP': 1_530.75, 'IND_DAY': '1992-06-12'},
    'MEX': {'COUNTRY': 'Mexico', 'POP': 126.58, 'AREA': 1_964.38,
            'GDP': 1_158.23, 'CONT': 'N.America', 'IND_DAY': '1810-09-16'},
    'JPN': {'COUNTRY': 'Japan', 'POP': 126.22, 'AREA': 377.97,
            'GDP': 4_872.42, 'CONT': 'Asia'},
    'DEU': {'COUNTRY': 'Germany', 'POP': 83.02, 'AREA': 357.11,
            'GDP': 3_693.20, 'CONT': 'Europe'},
    'FRA': {'COUNTRY': 'France', 'POP': 67.02, 'AREA': 640.68,
            'GDP': 2_582.49, 'CONT': 'Europe', 'IND_DAY': '1789-07-14'},
    'GBR': {'COUNTRY': 'UK', 'POP': 66.44, 'AREA': 242.50,
            'GDP': 2_631.23, 'CONT': 'Europe'},
    'ITA': {'COUNTRY': 'Italy', 'POP': 60.36, 'AREA': 301.34,
            'GDP': 1_943.84, 'CONT': 'Europe'},
    'ARG': {'COUNTRY': 'Argentina', 'POP': 44.94, 'AREA': 2_780.40,
            'GDP': 637.49, 'CONT': 'S.America', 'IND_DAY': '1816-07-09'},
    'DZA': {'COUNTRY': 'Algeria', 'POP': 43.38, 'AREA': 2_381.74,
            'GDP': 167.56, 'CONT': 'Africa', 'IND_DAY': '1962-07-05'},
    'CAN': {'COUNTRY': 'Canada', 'POP': 37.59, 'AREA': 9_984.67,
            'GDP': 1_647.12, 'CONT': 'N.America', 'IND_DAY': '1867-07-01'},
    'AUS': {'COUNTRY': 'Australia', 'POP': 25.47, 'AREA': 7_692.02,
            'GDP': 1_408.68, 'CONT': 'Oceania'},
    'KAZ': {'COUNTRY': 'Kazakhstan', 'POP': 18.53, 'AREA': 2_724.90,
            'GDP': 159.41, 'CONT': 'Asia', 'IND_DAY': '1991-12-16'}
}

columns = ('COUNTRY', 'POP', 'AREA', 'GDP', 'CONT', 'IND_DAY')

Each row of the table is written as an inner dictionary whose keys are the column names and values are the corresponding data. These dictionaries are then collected as the values in the outer data dictionary. The corresponding keys for data are the three-letter country codes.

You can use this data to create an instance of a Pandas DataFrame. First, you need to import Pandas:

>>>

>>> import pandas as pd

Now that you have Pandas imported, you can use the DataFrame constructor and data to create a DataFrame object.

data is organized in such a way that the country codes correspond to columns. You can reverse the rows and columns of a DataFrame with the property .T:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df
        COUNTRY      POP     AREA      GDP       CONT     IND_DAY
CHN       China  1398.72  9596.96  12234.8       Asia         NaN
IND       India  1351.16  3287.26  2575.67       Asia  1947-08-15
USA          US   329.74  9833.52  19485.4  N.America  1776-07-04
IDN   Indonesia   268.07  1910.93  1015.54       Asia  1945-08-17
BRA      Brazil   210.32  8515.77  2055.51  S.America  1822-09-07
PAK    Pakistan   205.71   881.91   302.14       Asia  1947-08-14
NGA     Nigeria   200.96   923.77   375.77     Africa  1960-10-01
BGD  Bangladesh   167.09   147.57   245.63       Asia  1971-03-26
RUS      Russia   146.79  17098.2  1530.75        NaN  1992-06-12
MEX      Mexico   126.58  1964.38  1158.23  N.America  1810-09-16
JPN       Japan   126.22   377.97  4872.42       Asia         NaN
DEU     Germany    83.02   357.11   3693.2     Europe         NaN
FRA      France    67.02   640.68  2582.49     Europe  1789-07-14
GBR          UK    66.44    242.5  2631.23     Europe         NaN
ITA       Italy    60.36   301.34  1943.84     Europe         NaN
ARG   Argentina    44.94   2780.4   637.49  S.America  1816-07-09
DZA     Algeria    43.38  2381.74   167.56     Africa  1962-07-05
CAN      Canada    37.59  9984.67  1647.12  N.America  1867-07-01
AUS   Australia    25.47  7692.02  1408.68    Oceania         NaN
KAZ  Kazakhstan    18.53   2724.9   159.41       Asia  1991-12-16

Now you have your DataFrame object populated with the data about each country.

Note: You can use .transpose() instead of .T to reverse the rows and columns of your dataset. If you use .transpose(), then you can set the optional parameter copy to specify if you want to copy the underlying data. The default behavior is False.

Versions of Python older than 3.6 did not guarantee the order of keys in dictionaries. To ensure the order of columns is maintained for older versions of Python and Pandas, you can specify index=columns:

>>>

>>> df = pd.DataFrame(data=data, index=columns).T

Now that you’ve prepared your data, you’re ready to start working with files!

Using the Pandas `read_csv()` and `.to_csv()` Functions

A comma-separated values (CSV) file is a plaintext file with a .csv extension that holds tabular data. This is one of the most popular file formats for storing large amounts of data. Each row of the CSV file represents a single table row. The values in the same row are by default separated with commas, but you could change the separator to a semicolon, tab, space, or some other character.

Write a CSV File

You can save your Pandas DataFrame as a CSV file with .to_csv():

>>>

>>> df.to_csv('data.csv')

That’s it! You’ve created the file data.csv in your current working directory. You can expand the code block below to see how your CSV file should look:

,COUNTRY,POP,AREA,GDP,CONT,IND_DAY
CHN,China,1398.72,9596.96,12234.78,Asia,
IND,India,1351.16,3287.26,2575.67,Asia,1947-08-15
USA,US,329.74,9833.52,19485.39,N.America,1776-07-04
IDN,Indonesia,268.07,1910.93,1015.54,Asia,1945-08-17
BRA,Brazil,210.32,8515.77,2055.51,S.America,1822-09-07
PAK,Pakistan,205.71,881.91,302.14,Asia,1947-08-14
NGA,Nigeria,200.96,923.77,375.77,Africa,1960-10-01
BGD,Bangladesh,167.09,147.57,245.63,Asia,1971-03-26
RUS,Russia,146.79,17098.25,1530.75,,1992-06-12
MEX,Mexico,126.58,1964.38,1158.23,N.America,1810-09-16
JPN,Japan,126.22,377.97,4872.42,Asia,
DEU,Germany,83.02,357.11,3693.2,Europe,
FRA,France,67.02,640.68,2582.49,Europe,1789-07-14
GBR,UK,66.44,242.5,2631.23,Europe,
ITA,Italy,60.36,301.34,1943.84,Europe,
ARG,Argentina,44.94,2780.4,637.49,S.America,1816-07-09
DZA,Algeria,43.38,2381.74,167.56,Africa,1962-07-05
CAN,Canada,37.59,9984.67,1647.12,N.America,1867-07-01
AUS,Australia,25.47,7692.02,1408.68,Oceania,
KAZ,Kazakhstan,18.53,2724.9,159.41,Asia,1991-12-16

This text file contains the data separated with commas. The first column contains the row labels. In some cases, you’ll find them irrelevant. If you don’t want to keep them, then you can pass the argument index=False to .to_csv().

Read a CSV File

Once your data is saved in a CSV file, you’ll likely want to load and use it from time to time. You can do that with the Pandas read_csv() function:

>>>

>>> df = pd.read_csv('data.csv', index_col=0)
>>> df
        COUNTRY      POP      AREA       GDP       CONT     IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia         NaN
IND       India  1351.16   3287.26   2575.67       Asia  1947-08-15
USA          US   329.74   9833.52  19485.39  N.America  1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia  1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America  1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia  1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa  1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia  1971-03-26
RUS      Russia   146.79  17098.25   1530.75        NaN  1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America  1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia         NaN
DEU     Germany    83.02    357.11   3693.20     Europe         NaN
FRA      France    67.02    640.68   2582.49     Europe  1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe         NaN
ITA       Italy    60.36    301.34   1943.84     Europe         NaN
ARG   Argentina    44.94   2780.40    637.49  S.America  1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa  1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America  1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania         NaN
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia  1991-12-16

In this case, the Pandas read_csv() function returns a new DataFrame with the data and labels from the file data.csv, which you specified with the first argument. This string can be any valid path, including URLs.

The parameter index_col specifies the column from the CSV file that contains the row labels. You assign a zero-based column index to this parameter. You should determine the value of index_col when the CSV file contains the row labels to avoid loading them as data.

You’ll learn more about using Pandas with CSV files later on in this tutorial. You can also check out Reading and Writing CSV Files in Python to see how to handle CSV files with the built-in Python library csv as well.

Using Pandas to Write and Read Excel Files

Microsoft Excel is probably the most widely-used spreadsheet software. While older versions used binary .xls files, Excel 2007 introduced the new XML-based .xlsx file. You can read and write Excel files in Pandas, similar to CSV files. However, you’ll need to install the following Python packages first:

xlwt to write to .xls files
openpyxl or XlsxWriter to write to .xlsx files
xlrd to read Excel files

You can install them using pip with a single command:

$ pip install xlwt openpyxl xlsxwriter xlrd

You can also use Conda:

$ conda install xlwt openpyxl xlsxwriter xlrd

Please note that you don’t have to install all these packages. For example, you don’t need both openpyxl and XlsxWriter. If you’re going to work just with .xls files, then you don’t need any of them! However, if you intend to work only with .xlsx files, then you’re going to need at least one of them, but not xlwt. Take some time to decide which packages are right for your project.

Write an Excel File

Once you have those packages installed, you can save your DataFrame in an Excel file with .to_excel():

>>>

>>> df.to_excel('data.xlsx')

The argument 'data.xlsx' represents the target file and, optionally, its path. The above statement should create the file data.xlsx in your current working directory. That file should look like this:

The first column of the file contains the labels of the rows, while the other columns store data.

Read an Excel File

You can load data from Excel files with read_excel():

>>>

>>> df = pd.read_excel('data.xlsx', index_col=0)
>>> df
        COUNTRY      POP      AREA       GDP       CONT     IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia         NaN
IND       India  1351.16   3287.26   2575.67       Asia  1947-08-15
USA          US   329.74   9833.52  19485.39  N.America  1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia  1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America  1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia  1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa  1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia  1971-03-26
RUS      Russia   146.79  17098.25   1530.75        NaN  1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America  1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia         NaN
DEU     Germany    83.02    357.11   3693.20     Europe         NaN
FRA      France    67.02    640.68   2582.49     Europe  1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe         NaN
ITA       Italy    60.36    301.34   1943.84     Europe         NaN
ARG   Argentina    44.94   2780.40    637.49  S.America  1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa  1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America  1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania         NaN
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia  1991-12-16

read_excel() returns a new DataFrame that contains the values from data.xlsx. You can also use read_excel() with OpenDocument spreadsheets, or .ods files.

You’ll learn more about working with Excel files later on in this tutorial. You can also check out Using Pandas to Read Large Excel Files in Python.

Understanding the Pandas IO API

Pandas IO Tools is the API that allows you to save the contents of Series and DataFrame objects to the clipboard, objects, or files of various types. It also enables loading data from the clipboard, objects, or files.

Write Files

Series and DataFrame objects have methods that enable writing data and labels to the clipboard or files. They’re named with the pattern .to_<file-type>(), where <file-type> is the type of the target file.

You’ve learned about .to_csv() and .to_excel(), but there are others, including:

.to_json()
.to_html()
.to_sql()
.to_pickle()

There are still more file types that you can write to, so this list is not exhaustive.

Note: To find similar methods, check the official documentation about serialization, IO, and conversion related to Series and DataFrame objects.

These methods have parameters specifying the target file path where you saved the data and labels. This is mandatory in some cases and optional in others. If this option is available and you choose to omit it, then the methods return the objects (like strings or iterables) with the contents of DataFrame instances.

The optional parameter compression decides how to compress the file with the data and labels. You’ll learn more about it later on. There are a few other parameters, but they’re mostly specific to one or several methods. You won’t go into them in detail here.

Read Files

Pandas functions for reading the contents of files are named using the pattern .read_<file-type>(), where <file-type> indicates the type of the file to read. You’ve already seen the Pandas read_csv() and read_excel() functions. Here are a few others:

read_json()
read_html()
read_sql()
read_pickle()

These functions have a parameter that specifies the target file path. It can be any valid string that represents the path, either on a local machine or in a URL. Other objects are also acceptable depending on the file type.

The optional parameter compression determines the type of decompression to use for the compressed files. You’ll learn about it later on in this tutorial. There are other parameters, but they’re specific to one or several functions. You won’t go into them in detail here.

Working With Different File Types

The Pandas library offers a wide range of possibilities for saving your data to files and loading data from files. In this section, you’ll learn more about working with CSV and Excel files. You’ll also see how to use other types of files, like JSON, web pages, databases, and Python pickle files.

CSV Files

You’ve already learned how to read and write CSV files. Now let’s dig a little deeper into the details. When you use .to_csv() to save your DataFrame, you can provide an argument for the parameter path_or_buff to specify the path, name, and extension of the target file.

path_or_buff is the first argument .to_csv() will get. It can be any string that represents a valid file path that includes the file name and its extension. You’ve seen this in a previous example. However, if you omit path_or_buff, then .to_csv() won’t create any files. Instead, it’ll return the corresponding string:

>>>

>>> df = pd.DataFrame(data=data).T
>>> s = df.to_csv()
>>> print(s)
,COUNTRY,POP,AREA,GDP,CONT,IND_DAY
CHN,China,1398.72,9596.96,12234.78,Asia,
IND,India,1351.16,3287.26,2575.67,Asia,1947-08-15
USA,US,329.74,9833.52,19485.39,N.America,1776-07-04
IDN,Indonesia,268.07,1910.93,1015.54,Asia,1945-08-17
BRA,Brazil,210.32,8515.77,2055.51,S.America,1822-09-07
PAK,Pakistan,205.71,881.91,302.14,Asia,1947-08-14
NGA,Nigeria,200.96,923.77,375.77,Africa,1960-10-01
BGD,Bangladesh,167.09,147.57,245.63,Asia,1971-03-26
RUS,Russia,146.79,17098.25,1530.75,,1992-06-12
MEX,Mexico,126.58,1964.38,1158.23,N.America,1810-09-16
JPN,Japan,126.22,377.97,4872.42,Asia,
DEU,Germany,83.02,357.11,3693.2,Europe,
FRA,France,67.02,640.68,2582.49,Europe,1789-07-14
GBR,UK,66.44,242.5,2631.23,Europe,
ITA,Italy,60.36,301.34,1943.84,Europe,
ARG,Argentina,44.94,2780.4,637.49,S.America,1816-07-09
DZA,Algeria,43.38,2381.74,167.56,Africa,1962-07-05
CAN,Canada,37.59,9984.67,1647.12,N.America,1867-07-01
AUS,Australia,25.47,7692.02,1408.68,Oceania,
KAZ,Kazakhstan,18.53,2724.9,159.41,Asia,1991-12-16

Now you have the string s instead of a CSV file. You also have some missing values in your DataFrame object. For example, the continent for Russia and the independence days for several countries (China, Japan, and so on) are not available. In data science and machine learning, you must handle missing values carefully. Pandas excels here! By default, Pandas uses the NaN value to replace the missing values.

Note: nan, which stands for “not a number,” is a particular floating-point value in Python.

You can get a nan value with any of the following functions:

The continent that corresponds to Russia in df is nan:

>>>

>>> df.loc['RUS', 'CONT']
nan

This example uses .loc[] to get data with the specified row and column names.

When you save your DataFrame to a CSV file, empty strings ('') will represent the missing data. You can see this both in your file data.csv and in the string s. If you want to change this behavior, then use the optional parameter na_rep:

>>>

>>> df.to_csv('new-data.csv', na_rep='(missing)')

This code produces the file new-data.csv where the missing values are no longer empty strings. You can expand the code block below to see how this file should look:

,COUNTRY,POP,AREA,GDP,CONT,IND_DAY
CHN,China,1398.72,9596.96,12234.78,Asia,(missing)
IND,India,1351.16,3287.26,2575.67,Asia,1947-08-15
USA,US,329.74,9833.52,19485.39,N.America,1776-07-04
IDN,Indonesia,268.07,1910.93,1015.54,Asia,1945-08-17
BRA,Brazil,210.32,8515.77,2055.51,S.America,1822-09-07
PAK,Pakistan,205.71,881.91,302.14,Asia,1947-08-14
NGA,Nigeria,200.96,923.77,375.77,Africa,1960-10-01
BGD,Bangladesh,167.09,147.57,245.63,Asia,1971-03-26
RUS,Russia,146.79,17098.25,1530.75,(missing),1992-06-12
MEX,Mexico,126.58,1964.38,1158.23,N.America,1810-09-16
JPN,Japan,126.22,377.97,4872.42,Asia,(missing)
DEU,Germany,83.02,357.11,3693.2,Europe,(missing)
FRA,France,67.02,640.68,2582.49,Europe,1789-07-14
GBR,UK,66.44,242.5,2631.23,Europe,(missing)
ITA,Italy,60.36,301.34,1943.84,Europe,(missing)
ARG,Argentina,44.94,2780.4,637.49,S.America,1816-07-09
DZA,Algeria,43.38,2381.74,167.56,Africa,1962-07-05
CAN,Canada,37.59,9984.67,1647.12,N.America,1867-07-01
AUS,Australia,25.47,7692.02,1408.68,Oceania,(missing)
KAZ,Kazakhstan,18.53,2724.9,159.41,Asia,1991-12-16

Now, the string '(missing)' in the file corresponds to the nan values from df.

When Pandas reads files, it considers the empty string ('') and a few others as missing values by default:

'nan'
'-nan'
'NA'
'N/A'
'NaN'
'null'

If you don’t want this behavior, then you can pass keep_default_na=False to the Pandas read_csv() function. To specify other labels for missing values, use the parameter na_values:

>>>

>>> pd.read_csv('new-data.csv', index_col=0, na_values='(missing)')
        COUNTRY      POP      AREA       GDP       CONT     IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia         NaN
IND       India  1351.16   3287.26   2575.67       Asia  1947-08-15
USA          US   329.74   9833.52  19485.39  N.America  1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia  1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America  1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia  1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa  1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia  1971-03-26
RUS      Russia   146.79  17098.25   1530.75        NaN  1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America  1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia         NaN
DEU     Germany    83.02    357.11   3693.20     Europe         NaN
FRA      France    67.02    640.68   2582.49     Europe  1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe         NaN
ITA       Italy    60.36    301.34   1943.84     Europe         NaN
ARG   Argentina    44.94   2780.40    637.49  S.America  1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa  1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America  1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania         NaN
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia  1991-12-16

Here, you’ve marked the string '(missing)' as a new missing data label, and Pandas replaced it with nan when it read the file.

When you load data from a file, Pandas assigns the data types to the values of each column by default. You can check these types with .dtypes:

>>>

>>> df = pd.read_csv('data.csv', index_col=0)
>>> df.dtypes
COUNTRY     object
POP        float64
AREA       float64
GDP        float64
CONT        object
IND_DAY     object
dtype: object

The columns with strings and dates ('COUNTRY', 'CONT', and 'IND_DAY') have the data type object. Meanwhile, the numeric columns contain 64-bit floating-point numbers (float64).

You can use the parameter dtype to specify the desired data types and parse_dates to force use of datetimes:

>>>

>>> dtypes = {'POP': 'float32', 'AREA': 'float32', 'GDP': 'float32'}
>>> df = pd.read_csv('data.csv', index_col=0, dtype=dtypes,
...                  parse_dates=['IND_DAY'])
>>> df.dtypes
COUNTRY            object
POP               float32
AREA              float32
GDP               float32
CONT               object
IND_DAY    datetime64[ns]
dtype: object
>>> df['IND_DAY']
CHN          NaT
IND   1947-08-15
USA   1776-07-04
IDN   1945-08-17
BRA   1822-09-07
PAK   1947-08-14
NGA   1960-10-01
BGD   1971-03-26
RUS   1992-06-12
MEX   1810-09-16
JPN          NaT
DEU          NaT
FRA   1789-07-14
GBR          NaT
ITA          NaT
ARG   1816-07-09
DZA   1962-07-05
CAN   1867-07-01
AUS          NaT
KAZ   1991-12-16
Name: IND_DAY, dtype: datetime64[ns]

Now, you have 32-bit floating-point numbers ()float32) as specified with dtype. These differ slightly from the original 64-bit numbers because of smaller precision. The values in the last column are considered as dates and have the data type datetime64. That’s why the NaN values in this column are replaced with NaT.

Now that you have real dates, you can save them in the format you like:

>>>

>>> df = pd.read_csv('data.csv', index_col=0, parse_dates=['IND_DAY'])
>>> df.to_csv('formatted-data.csv', date_format='%B %d, %Y')

Here, you’ve specified the parameter date_format to be '%B %d, %Y'. You can expand the code block below to see the resulting file:

,COUNTRY,POP,AREA,GDP,CONT,IND_DAY
CHN,China,1398.72,9596.96,12234.78,Asia,
IND,India,1351.16,3287.26,2575.67,Asia,"August 15, 1947"
USA,US,329.74,9833.52,19485.39,N.America,"July 04, 1776"
IDN,Indonesia,268.07,1910.93,1015.54,Asia,"August 17, 1945"
BRA,Brazil,210.32,8515.77,2055.51,S.America,"September 07, 1822"
PAK,Pakistan,205.71,881.91,302.14,Asia,"August 14, 1947"
NGA,Nigeria,200.96,923.77,375.77,Africa,"October 01, 1960"
BGD,Bangladesh,167.09,147.57,245.63,Asia,"March 26, 1971"
RUS,Russia,146.79,17098.25,1530.75,,"June 12, 1992"
MEX,Mexico,126.58,1964.38,1158.23,N.America,"September 16, 1810"
JPN,Japan,126.22,377.97,4872.42,Asia,
DEU,Germany,83.02,357.11,3693.2,Europe,
FRA,France,67.02,640.68,2582.49,Europe,"July 14, 1789"
GBR,UK,66.44,242.5,2631.23,Europe,
ITA,Italy,60.36,301.34,1943.84,Europe,
ARG,Argentina,44.94,2780.4,637.49,S.America,"July 09, 1816"
DZA,Algeria,43.38,2381.74,167.56,Africa,"July 05, 1962"
CAN,Canada,37.59,9984.67,1647.12,N.America,"July 01, 1867"
AUS,Australia,25.47,7692.02,1408.68,Oceania,
KAZ,Kazakhstan,18.53,2724.9,159.41,Asia,"December 16, 1991"

The format of the dates is different now. The format '%B %d, %Y' means the date will first display the full name of the month, then the day followed by a comma, and finally the full year.

There are several other optional parameters that you can use with .to_csv():

sep denotes a values separator.
decimal indicates a decimal separator.
encoding sets the file encoding.
header specifies whether you want to write column labels in the file.

Here’s how you would pass arguments for sep and header:

>>>

>>> s = df.to_csv(sep=';', header=False)
>>> print(s)
CHN;China;1398.72;9596.96;12234.78;Asia;
IND;India;1351.16;3287.26;2575.67;Asia;1947-08-15
USA;US;329.74;9833.52;19485.39;N.America;1776-07-04
IDN;Indonesia;268.07;1910.93;1015.54;Asia;1945-08-17
BRA;Brazil;210.32;8515.77;2055.51;S.America;1822-09-07
PAK;Pakistan;205.71;881.91;302.14;Asia;1947-08-14
NGA;Nigeria;200.96;923.77;375.77;Africa;1960-10-01
BGD;Bangladesh;167.09;147.57;245.63;Asia;1971-03-26
RUS;Russia;146.79;17098.25;1530.75;;1992-06-12
MEX;Mexico;126.58;1964.38;1158.23;N.America;1810-09-16
JPN;Japan;126.22;377.97;4872.42;Asia;
DEU;Germany;83.02;357.11;3693.2;Europe;
FRA;France;67.02;640.68;2582.49;Europe;1789-07-14
GBR;UK;66.44;242.5;2631.23;Europe;
ITA;Italy;60.36;301.34;1943.84;Europe;
ARG;Argentina;44.94;2780.4;637.49;S.America;1816-07-09
DZA;Algeria;43.38;2381.74;167.56;Africa;1962-07-05
CAN;Canada;37.59;9984.67;1647.12;N.America;1867-07-01
AUS;Australia;25.47;7692.02;1408.68;Oceania;
KAZ;Kazakhstan;18.53;2724.9;159.41;Asia;1991-12-16

The data is separated with a semicolon (';') because you’ve specified sep=';'. Also, since you passed header=False, you see your data without the header row of column names.

The Pandas read_csv() function has many additional options for managing missing data, working with dates and times, quoting, encoding, handling errors, and more. For instance, if you have a file with one data column and want to get a Series object instead of a DataFrame, then you can pass squeeze=True to read_csv(). You’ll learn later on about data compression and decompression, as well as how to skip rows and columns.

JSON Files

JSON stands for JavaScript object notation. JSON files are plaintext files used for data interchange, and humans can read them easily. They follow the ISO/IEC 21778:2017 and ECMA-404 standards and use the .json extension. Python and Pandas work well with JSON files, as Python’s json library offers built-in support for them.

You can save the data from your DataFrame to a JSON file with .to_json(). Start by creating a DataFrame object again. Use the dictionary data that holds the data about countries and then apply .to_json():

>>>

>>> df = pd.DataFrame(data=data).T
>>> df.to_json('data-columns.json')

This code produces the file data-columns.json. You can expand the code block below to see how this file should look:

{"COUNTRY":{"CHN":"China","IND":"India","USA":"US","IDN":"Indonesia","BRA":"Brazil","PAK":"Pakistan","NGA":"Nigeria","BGD":"Bangladesh","RUS":"Russia","MEX":"Mexico","JPN":"Japan","DEU":"Germany","FRA":"France","GBR":"UK","ITA":"Italy","ARG":"Argentina","DZA":"Algeria","CAN":"Canada","AUS":"Australia","KAZ":"Kazakhstan"},"POP":{"CHN":1398.72,"IND":1351.16,"USA":329.74,"IDN":268.07,"BRA":210.32,"PAK":205.71,"NGA":200.96,"BGD":167.09,"RUS":146.79,"MEX":126.58,"JPN":126.22,"DEU":83.02,"FRA":67.02,"GBR":66.44,"ITA":60.36,"ARG":44.94,"DZA":43.38,"CAN":37.59,"AUS":25.47,"KAZ":18.53},"AREA":{"CHN":9596.96,"IND":3287.26,"USA":9833.52,"IDN":1910.93,"BRA":8515.77,"PAK":881.91,"NGA":923.77,"BGD":147.57,"RUS":17098.25,"MEX":1964.38,"JPN":377.97,"DEU":357.11,"FRA":640.68,"GBR":242.5,"ITA":301.34,"ARG":2780.4,"DZA":2381.74,"CAN":9984.67,"AUS":7692.02,"KAZ":2724.9},"GDP":{"CHN":12234.78,"IND":2575.67,"USA":19485.39,"IDN":1015.54,"BRA":2055.51,"PAK":302.14,"NGA":375.77,"BGD":245.63,"RUS":1530.75,"MEX":1158.23,"JPN":4872.42,"DEU":3693.2,"FRA":2582.49,"GBR":2631.23,"ITA":1943.84,"ARG":637.49,"DZA":167.56,"CAN":1647.12,"AUS":1408.68,"KAZ":159.41},"CONT":{"CHN":"Asia","IND":"Asia","USA":"N.America","IDN":"Asia","BRA":"S.America","PAK":"Asia","NGA":"Africa","BGD":"Asia","RUS":null,"MEX":"N.America","JPN":"Asia","DEU":"Europe","FRA":"Europe","GBR":"Europe","ITA":"Europe","ARG":"S.America","DZA":"Africa","CAN":"N.America","AUS":"Oceania","KAZ":"Asia"},"IND_DAY":{"CHN":null,"IND":"1947-08-15","USA":"1776-07-04","IDN":"1945-08-17","BRA":"1822-09-07","PAK":"1947-08-14","NGA":"1960-10-01","BGD":"1971-03-26","RUS":"1992-06-12","MEX":"1810-09-16","JPN":null,"DEU":null,"FRA":"1789-07-14","GBR":null,"ITA":null,"ARG":"1816-07-09","DZA":"1962-07-05","CAN":"1867-07-01","AUS":null,"KAZ":"1991-12-16"}}

data-columns.json has one large dictionary with the column labels as keys and the corresponding inner dictionaries as values.

You can get a different file structure if you pass an argument for the optional parameter orient:

>>>

>>> df.to_json('data-index.json', orient='index')

The orient parameter defaults to 'columns'. Here, you’ve set it to index.

You should get a new file data-index.json. You can expand the code block below to see the changes:

{"CHN":{"COUNTRY":"China","POP":1398.72,"AREA":9596.96,"GDP":12234.78,"CONT":"Asia","IND_DAY":null},"IND":{"COUNTRY":"India","POP":1351.16,"AREA":3287.26,"GDP":2575.67,"CONT":"Asia","IND_DAY":"1947-08-15"},"USA":{"COUNTRY":"US","POP":329.74,"AREA":9833.52,"GDP":19485.39,"CONT":"N.America","IND_DAY":"1776-07-04"},"IDN":{"COUNTRY":"Indonesia","POP":268.07,"AREA":1910.93,"GDP":1015.54,"CONT":"Asia","IND_DAY":"1945-08-17"},"BRA":{"COUNTRY":"Brazil","POP":210.32,"AREA":8515.77,"GDP":2055.51,"CONT":"S.America","IND_DAY":"1822-09-07"},"PAK":{"COUNTRY":"Pakistan","POP":205.71,"AREA":881.91,"GDP":302.14,"CONT":"Asia","IND_DAY":"1947-08-14"},"NGA":{"COUNTRY":"Nigeria","POP":200.96,"AREA":923.77,"GDP":375.77,"CONT":"Africa","IND_DAY":"1960-10-01"},"BGD":{"COUNTRY":"Bangladesh","POP":167.09,"AREA":147.57,"GDP":245.63,"CONT":"Asia","IND_DAY":"1971-03-26"},"RUS":{"COUNTRY":"Russia","POP":146.79,"AREA":17098.25,"GDP":1530.75,"CONT":null,"IND_DAY":"1992-06-12"},"MEX":{"COUNTRY":"Mexico","POP":126.58,"AREA":1964.38,"GDP":1158.23,"CONT":"N.America","IND_DAY":"1810-09-16"},"JPN":{"COUNTRY":"Japan","POP":126.22,"AREA":377.97,"GDP":4872.42,"CONT":"Asia","IND_DAY":null},"DEU":{"COUNTRY":"Germany","POP":83.02,"AREA":357.11,"GDP":3693.2,"CONT":"Europe","IND_DAY":null},"FRA":{"COUNTRY":"France","POP":67.02,"AREA":640.68,"GDP":2582.49,"CONT":"Europe","IND_DAY":"1789-07-14"},"GBR":{"COUNTRY":"UK","POP":66.44,"AREA":242.5,"GDP":2631.23,"CONT":"Europe","IND_DAY":null},"ITA":{"COUNTRY":"Italy","POP":60.36,"AREA":301.34,"GDP":1943.84,"CONT":"Europe","IND_DAY":null},"ARG":{"COUNTRY":"Argentina","POP":44.94,"AREA":2780.4,"GDP":637.49,"CONT":"S.America","IND_DAY":"1816-07-09"},"DZA":{"COUNTRY":"Algeria","POP":43.38,"AREA":2381.74,"GDP":167.56,"CONT":"Africa","IND_DAY":"1962-07-05"},"CAN":{"COUNTRY":"Canada","POP":37.59,"AREA":9984.67,"GDP":1647.12,"CONT":"N.America","IND_DAY":"1867-07-01"},"AUS":{"COUNTRY":"Australia","POP":25.47,"AREA":7692.02,"GDP":1408.68,"CONT":"Oceania","IND_DAY":null},"KAZ":{"COUNTRY":"Kazakhstan","POP":18.53,"AREA":2724.9,"GDP":159.41,"CONT":"Asia","IND_DAY":"1991-12-16"}}

data-index.json also has one large dictionary, but this time the row labels are the keys, and the inner dictionaries are the values.

There are few more options for orient. One of them is 'records':

>>>

>>> df.to_json('data-records.json', orient='records')

This code should yield the file data-records.json. You can expand the code block below to see the content:

[{"COUNTRY":"China","POP":1398.72,"AREA":9596.96,"GDP":12234.78,"CONT":"Asia","IND_DAY":null},{"COUNTRY":"India","POP":1351.16,"AREA":3287.26,"GDP":2575.67,"CONT":"Asia","IND_DAY":"1947-08-15"},{"COUNTRY":"US","POP":329.74,"AREA":9833.52,"GDP":19485.39,"CONT":"N.America","IND_DAY":"1776-07-04"},{"COUNTRY":"Indonesia","POP":268.07,"AREA":1910.93,"GDP":1015.54,"CONT":"Asia","IND_DAY":"1945-08-17"},{"COUNTRY":"Brazil","POP":210.32,"AREA":8515.77,"GDP":2055.51,"CONT":"S.America","IND_DAY":"1822-09-07"},{"COUNTRY":"Pakistan","POP":205.71,"AREA":881.91,"GDP":302.14,"CONT":"Asia","IND_DAY":"1947-08-14"},{"COUNTRY":"Nigeria","POP":200.96,"AREA":923.77,"GDP":375.77,"CONT":"Africa","IND_DAY":"1960-10-01"},{"COUNTRY":"Bangladesh","POP":167.09,"AREA":147.57,"GDP":245.63,"CONT":"Asia","IND_DAY":"1971-03-26"},{"COUNTRY":"Russia","POP":146.79,"AREA":17098.25,"GDP":1530.75,"CONT":null,"IND_DAY":"1992-06-12"},{"COUNTRY":"Mexico","POP":126.58,"AREA":1964.38,"GDP":1158.23,"CONT":"N.America","IND_DAY":"1810-09-16"},{"COUNTRY":"Japan","POP":126.22,"AREA":377.97,"GDP":4872.42,"CONT":"Asia","IND_DAY":null},{"COUNTRY":"Germany","POP":83.02,"AREA":357.11,"GDP":3693.2,"CONT":"Europe","IND_DAY":null},{"COUNTRY":"France","POP":67.02,"AREA":640.68,"GDP":2582.49,"CONT":"Europe","IND_DAY":"1789-07-14"},{"COUNTRY":"UK","POP":66.44,"AREA":242.5,"GDP":2631.23,"CONT":"Europe","IND_DAY":null},{"COUNTRY":"Italy","POP":60.36,"AREA":301.34,"GDP":1943.84,"CONT":"Europe","IND_DAY":null},{"COUNTRY":"Argentina","POP":44.94,"AREA":2780.4,"GDP":637.49,"CONT":"S.America","IND_DAY":"1816-07-09"},{"COUNTRY":"Algeria","POP":43.38,"AREA":2381.74,"GDP":167.56,"CONT":"Africa","IND_DAY":"1962-07-05"},{"COUNTRY":"Canada","POP":37.59,"AREA":9984.67,"GDP":1647.12,"CONT":"N.America","IND_DAY":"1867-07-01"},{"COUNTRY":"Australia","POP":25.47,"AREA":7692.02,"GDP":1408.68,"CONT":"Oceania","IND_DAY":null},{"COUNTRY":"Kazakhstan","POP":18.53,"AREA":2724.9,"GDP":159.41,"CONT":"Asia","IND_DAY":"1991-12-16"}]

data-records.json holds a list with one dictionary for each row. The row labels are not written.

You can get another interesting file structure with orient='split':

>>>

>>> df.to_json('data-split.json', orient='split')

The resulting file is data-split.json. You can expand the code block below to see how this file should look:

{"columns":["COUNTRY","POP","AREA","GDP","CONT","IND_DAY"],"index":["CHN","IND","USA","IDN","BRA","PAK","NGA","BGD","RUS","MEX","JPN","DEU","FRA","GBR","ITA","ARG","DZA","CAN","AUS","KAZ"],"data":[["China",1398.72,9596.96,12234.78,"Asia",null],["India",1351.16,3287.26,2575.67,"Asia","1947-08-15"],["US",329.74,9833.52,19485.39,"N.America","1776-07-04"],["Indonesia",268.07,1910.93,1015.54,"Asia","1945-08-17"],["Brazil",210.32,8515.77,2055.51,"S.America","1822-09-07"],["Pakistan",205.71,881.91,302.14,"Asia","1947-08-14"],["Nigeria",200.96,923.77,375.77,"Africa","1960-10-01"],["Bangladesh",167.09,147.57,245.63,"Asia","1971-03-26"],["Russia",146.79,17098.25,1530.75,null,"1992-06-12"],["Mexico",126.58,1964.38,1158.23,"N.America","1810-09-16"],["Japan",126.22,377.97,4872.42,"Asia",null],["Germany",83.02,357.11,3693.2,"Europe",null],["France",67.02,640.68,2582.49,"Europe","1789-07-14"],["UK",66.44,242.5,2631.23,"Europe",null],["Italy",60.36,301.34,1943.84,"Europe",null],["Argentina",44.94,2780.4,637.49,"S.America","1816-07-09"],["Algeria",43.38,2381.74,167.56,"Africa","1962-07-05"],["Canada",37.59,9984.67,1647.12,"N.America","1867-07-01"],["Australia",25.47,7692.02,1408.68,"Oceania",null],["Kazakhstan",18.53,2724.9,159.41,"Asia","1991-12-16"]]}

data-split.json contains one dictionary that holds the following lists:

The names of the columns
The labels of the rows
The inner lists (two-dimensional sequence) that hold data values

If you don’t provide the value for the optional parameter path_or_buf that defines the file path, then .to_json() will return a JSON string instead of writing the results to a file. This behavior is consistent with .to_csv().

There are other optional parameters you can use. For instance, you can set index=False to forego saving row labels. You can manipulate precision with double_precision, and dates with date_format and date_unit. These last two parameters are particularly important when you have time series among your data:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df['IND_DAY'] = pd.to_datetime(df['IND_DAY'])
>>> df.dtypes
COUNTRY            object
POP                object
AREA               object
GDP                object
CONT               object
IND_DAY    datetime64[ns]
dtype: object

>>> df.to_json('data-time.json')

In this example, you’ve created the DataFrame from the dictionary data and used to_datetime() to convert the values in the last column to datetime64. You can expand the code block below to see the resulting file:

{"COUNTRY":{"CHN":"China","IND":"India","USA":"US","IDN":"Indonesia","BRA":"Brazil","PAK":"Pakistan","NGA":"Nigeria","BGD":"Bangladesh","RUS":"Russia","MEX":"Mexico","JPN":"Japan","DEU":"Germany","FRA":"France","GBR":"UK","ITA":"Italy","ARG":"Argentina","DZA":"Algeria","CAN":"Canada","AUS":"Australia","KAZ":"Kazakhstan"},"POP":{"CHN":1398.72,"IND":1351.16,"USA":329.74,"IDN":268.07,"BRA":210.32,"PAK":205.71,"NGA":200.96,"BGD":167.09,"RUS":146.79,"MEX":126.58,"JPN":126.22,"DEU":83.02,"FRA":67.02,"GBR":66.44,"ITA":60.36,"ARG":44.94,"DZA":43.38,"CAN":37.59,"AUS":25.47,"KAZ":18.53},"AREA":{"CHN":9596.96,"IND":3287.26,"USA":9833.52,"IDN":1910.93,"BRA":8515.77,"PAK":881.91,"NGA":923.77,"BGD":147.57,"RUS":17098.25,"MEX":1964.38,"JPN":377.97,"DEU":357.11,"FRA":640.68,"GBR":242.5,"ITA":301.34,"ARG":2780.4,"DZA":2381.74,"CAN":9984.67,"AUS":7692.02,"KAZ":2724.9},"GDP":{"CHN":12234.78,"IND":2575.67,"USA":19485.39,"IDN":1015.54,"BRA":2055.51,"PAK":302.14,"NGA":375.77,"BGD":245.63,"RUS":1530.75,"MEX":1158.23,"JPN":4872.42,"DEU":3693.2,"FRA":2582.49,"GBR":2631.23,"ITA":1943.84,"ARG":637.49,"DZA":167.56,"CAN":1647.12,"AUS":1408.68,"KAZ":159.41},"CONT":{"CHN":"Asia","IND":"Asia","USA":"N.America","IDN":"Asia","BRA":"S.America","PAK":"Asia","NGA":"Africa","BGD":"Asia","RUS":null,"MEX":"N.America","JPN":"Asia","DEU":"Europe","FRA":"Europe","GBR":"Europe","ITA":"Europe","ARG":"S.America","DZA":"Africa","CAN":"N.America","AUS":"Oceania","KAZ":"Asia"},"IND_DAY":{"CHN":null,"IND":-706320000000,"USA":-6106060800000,"IDN":-769219200000,"BRA":-4648924800000,"PAK":-706406400000,"NGA":-291945600000,"BGD":38793600000,"RUS":708307200000,"MEX":-5026838400000,"JPN":null,"DEU":null,"FRA":-5694969600000,"GBR":null,"ITA":null,"ARG":-4843411200000,"DZA":-236476800000,"CAN":-3234729600000,"AUS":null,"KAZ":692841600000}}

In this file, you have large integers instead of dates for the independence days. That’s because the default value of the optional parameter date_format is 'epoch' whenever orient isn’t 'table'. This default behavior expresses dates as an epoch in milliseconds relative to midnight on January 1, 1970.

However, if you pass date_format='iso', then you’ll get the dates in the ISO 8601 format. In addition, date_unit decides the units of time:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df['IND_DAY'] = pd.to_datetime(df['IND_DAY'])
>>> df.to_json('new-data-time.json', date_format='iso', date_unit='s')

This code produces the following JSON file:

{"COUNTRY":{"CHN":"China","IND":"India","USA":"US","IDN":"Indonesia","BRA":"Brazil","PAK":"Pakistan","NGA":"Nigeria","BGD":"Bangladesh","RUS":"Russia","MEX":"Mexico","JPN":"Japan","DEU":"Germany","FRA":"France","GBR":"UK","ITA":"Italy","ARG":"Argentina","DZA":"Algeria","CAN":"Canada","AUS":"Australia","KAZ":"Kazakhstan"},"POP":{"CHN":1398.72,"IND":1351.16,"USA":329.74,"IDN":268.07,"BRA":210.32,"PAK":205.71,"NGA":200.96,"BGD":167.09,"RUS":146.79,"MEX":126.58,"JPN":126.22,"DEU":83.02,"FRA":67.02,"GBR":66.44,"ITA":60.36,"ARG":44.94,"DZA":43.38,"CAN":37.59,"AUS":25.47,"KAZ":18.53},"AREA":{"CHN":9596.96,"IND":3287.26,"USA":9833.52,"IDN":1910.93,"BRA":8515.77,"PAK":881.91,"NGA":923.77,"BGD":147.57,"RUS":17098.25,"MEX":1964.38,"JPN":377.97,"DEU":357.11,"FRA":640.68,"GBR":242.5,"ITA":301.34,"ARG":2780.4,"DZA":2381.74,"CAN":9984.67,"AUS":7692.02,"KAZ":2724.9},"GDP":{"CHN":12234.78,"IND":2575.67,"USA":19485.39,"IDN":1015.54,"BRA":2055.51,"PAK":302.14,"NGA":375.77,"BGD":245.63,"RUS":1530.75,"MEX":1158.23,"JPN":4872.42,"DEU":3693.2,"FRA":2582.49,"GBR":2631.23,"ITA":1943.84,"ARG":637.49,"DZA":167.56,"CAN":1647.12,"AUS":1408.68,"KAZ":159.41},"CONT":{"CHN":"Asia","IND":"Asia","USA":"N.America","IDN":"Asia","BRA":"S.America","PAK":"Asia","NGA":"Africa","BGD":"Asia","RUS":null,"MEX":"N.America","JPN":"Asia","DEU":"Europe","FRA":"Europe","GBR":"Europe","ITA":"Europe","ARG":"S.America","DZA":"Africa","CAN":"N.America","AUS":"Oceania","KAZ":"Asia"},"IND_DAY":{"CHN":null,"IND":"1947-08-15T00:00:00Z","USA":"1776-07-04T00:00:00Z","IDN":"1945-08-17T00:00:00Z","BRA":"1822-09-07T00:00:00Z","PAK":"1947-08-14T00:00:00Z","NGA":"1960-10-01T00:00:00Z","BGD":"1971-03-26T00:00:00Z","RUS":"1992-06-12T00:00:00Z","MEX":"1810-09-16T00:00:00Z","JPN":null,"DEU":null,"FRA":"1789-07-14T00:00:00Z","GBR":null,"ITA":null,"ARG":"1816-07-09T00:00:00Z","DZA":"1962-07-05T00:00:00Z","CAN":"1867-07-01T00:00:00Z","AUS":null,"KAZ":"1991-12-16T00:00:00Z"}}

The dates in the resulting file are in the ISO 8601 format.

You can load the data from a JSON file with read_json():

>>>

>>> df = pd.read_json('data-index.json', orient='index',
...                   convert_dates=['IND_DAY'])

The parameter convert_dates has a similar purpose as parse_dates when you use it to read CSV files. The optional parameter orient is very important because it specifies how Pandas understands the structure of the file.

There are other optional parameters you can use as well:

Set the encoding with encoding.
Manipulate dates with convert_dates and keep_default_dates.
Impact precision with dtype and precise_float.
Decode numeric data directly to NumPy arrays with numpy=True.

Note that you might lose the order of rows and columns when using the JSON format to store your data.

HTML Files

An HTML is a plaintext file that uses hypertext markup language to help browsers render web pages. The extensions for HTML files are .html and .htm. You’ll need to install an HTML parser library like lxml or html5lib to be able to work with HTML files:

$pip install lxml html5lib

You can also use Conda to install the same packages:

$ conda install lxml html5lib

Once you have these libraries, you can save the contents of your DataFrame as an HTML file with .to_html():

>>>

df = pd.DataFrame(data=data).T
df.to_html('data.html')

This code generates a file data.html. You can expand the code block below to see how this file should look:

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>COUNTRY</th>
      <th>POP</th>
      <th>AREA</th>
      <th>GDP</th>
      <th>CONT</th>
      <th>IND_DAY</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>CHN</th>
      <td>China</td>
      <td>1398.72</td>
      <td>9596.96</td>
      <td>12234.8</td>
      <td>Asia</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>IND</th>
      <td>India</td>
      <td>1351.16</td>
      <td>3287.26</td>
      <td>2575.67</td>
      <td>Asia</td>
      <td>1947-08-15</td>
    </tr>
    <tr>
      <th>USA</th>
      <td>US</td>
      <td>329.74</td>
      <td>9833.52</td>
      <td>19485.4</td>
      <td>N.America</td>
      <td>1776-07-04</td>
    </tr>
    <tr>
      <th>IDN</th>
      <td>Indonesia</td>
      <td>268.07</td>
      <td>1910.93</td>
      <td>1015.54</td>
      <td>Asia</td>
      <td>1945-08-17</td>
    </tr>
    <tr>
      <th>BRA</th>
      <td>Brazil</td>
      <td>210.32</td>
      <td>8515.77</td>
      <td>2055.51</td>
      <td>S.America</td>
      <td>1822-09-07</td>
    </tr>
    <tr>
      <th>PAK</th>
      <td>Pakistan</td>
      <td>205.71</td>
      <td>881.91</td>
      <td>302.14</td>
      <td>Asia</td>
      <td>1947-08-14</td>
    </tr>
    <tr>
      <th>NGA</th>
      <td>Nigeria</td>
      <td>200.96</td>
      <td>923.77</td>
      <td>375.77</td>
      <td>Africa</td>
      <td>1960-10-01</td>
    </tr>
    <tr>
      <th>BGD</th>
      <td>Bangladesh</td>
      <td>167.09</td>
      <td>147.57</td>
      <td>245.63</td>
      <td>Asia</td>
      <td>1971-03-26</td>
    </tr>
    <tr>
      <th>RUS</th>
      <td>Russia</td>
      <td>146.79</td>
      <td>17098.2</td>
      <td>1530.75</td>
      <td>NaN</td>
      <td>1992-06-12</td>
    </tr>
    <tr>
      <th>MEX</th>
      <td>Mexico</td>
      <td>126.58</td>
      <td>1964.38</td>
      <td>1158.23</td>
      <td>N.America</td>
      <td>1810-09-16</td>
    </tr>
    <tr>
      <th>JPN</th>
      <td>Japan</td>
      <td>126.22</td>
      <td>377.97</td>
      <td>4872.42</td>
      <td>Asia</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>DEU</th>
      <td>Germany</td>
      <td>83.02</td>
      <td>357.11</td>
      <td>3693.2</td>
      <td>Europe</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>FRA</th>
      <td>France</td>
      <td>67.02</td>
      <td>640.68</td>
      <td>2582.49</td>
      <td>Europe</td>
      <td>1789-07-14</td>
    </tr>
    <tr>
      <th>GBR</th>
      <td>UK</td>
      <td>66.44</td>
      <td>242.5</td>
      <td>2631.23</td>
      <td>Europe</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>ITA</th>
      <td>Italy</td>
      <td>60.36</td>
      <td>301.34</td>
      <td>1943.84</td>
      <td>Europe</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>ARG</th>
      <td>Argentina</td>
      <td>44.94</td>
      <td>2780.4</td>
      <td>637.49</td>
      <td>S.America</td>
      <td>1816-07-09</td>
    </tr>
    <tr>
      <th>DZA</th>
      <td>Algeria</td>
      <td>43.38</td>
      <td>2381.74</td>
      <td>167.56</td>
      <td>Africa</td>
      <td>1962-07-05</td>
    </tr>
    <tr>
      <th>CAN</th>
      <td>Canada</td>
      <td>37.59</td>
      <td>9984.67</td>
      <td>1647.12</td>
      <td>N.America</td>
      <td>1867-07-01</td>
    </tr>
    <tr>
      <th>AUS</th>
      <td>Australia</td>
      <td>25.47</td>
      <td>7692.02</td>
      <td>1408.68</td>
      <td>Oceania</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>KAZ</th>
      <td>Kazakhstan</td>
      <td>18.53</td>
      <td>2724.9</td>
      <td>159.41</td>
      <td>Asia</td>
      <td>1991-12-16</td>
    </tr>
  </tbody>
</table>

This file shows the DataFrame contents nicely. However, notice that you haven’t obtained an entire web page. You’ve just output the data that corresponds to df in the HTML format.

.to_html() won’t create a file if you don’t provide the optional parameter buf, which denotes the buffer to write to. If you leave this parameter out, then your code will return a string as it did with .to_csv() and .to_json().

Here are some other optional parameters:

header determines whether to save the column names.
index determines whether to save the row labels.
classes assigns cascading style sheet (CSS) classes.
render_links specifies whether to convert URLs to HTML links.
table_id assigns the CSS id to the table tag.
escape decides whether to convert the characters <, >, and & to HTML-safe strings.

You use parameters like these to specify different aspects of the resulting files or strings.

You can create a DataFrame object from a suitable HTML file using read_html(), which will return a DataFrame instance or a list of them:

>>>

>>> df = pd.read_html('data.html', index_col=0, parse_dates=['IND_DAY'])

This is very similar to what you did when reading CSV files. You also have parameters that help you work with dates, missing values, precision, encoding, HTML parsers, and more.

Excel Files

You’ve already learned how to read and write Excel files with Pandas. However, there are a few more options worth considering. For one, when you use .to_excel(), you can specify the name of the target worksheet with the optional parameter sheet_name:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df.to_excel('data.xlsx', sheet_name='COUNTRIES')

Here, you create a file data.xlsx with a worksheet called COUNTRIES that stores the data. The string 'data.xlsx' is the argument for the parameter excel_writer that defines the name of the Excel file or its path.

The optional parameters startrow and startcol both default to 0 and indicate the upper left-most cell where the data should start being written:

>>>

>>> df.to_excel('data-shifted.xlsx', sheet_name='COUNTRIES',
...             startrow=2, startcol=4)

Here, you specify that the table should start in the third row and the fifth column. You also used zero-based indexing, so the third row is denoted by 2 and the fifth column by 4.

Now the resulting worksheet looks like this:

As you can see, the table starts in the third row 2 and the fifth column E.

.read_excel() also has the optional parameter sheet_name that specifies which worksheets to read when loading data. It can take on one of the following values:

The zero-based index of the worksheet
The name of the worksheet
The list of indices or names to read multiple sheets
The value None to read all sheets

Here’s how you would use this parameter in your code:

>>>

>>> df = pd.read_excel('data.xlsx', sheet_name=0, index_col=0,
...                    parse_dates=['IND_DAY'])
>>> df = pd.read_excel('data.xlsx', sheet_name='COUNTRIES', index_col=0,
...                    parse_dates=['IND_DAY'])

Both statements above create the same DataFrame because the sheet_name parameters have the same values. In both cases, sheet_name=0 and sheet_name='COUNTRIES' refer to the same worksheet. The argument parse_dates=['IND_DAY'] tells Pandas to try to consider the values in this column as dates or times.

There are other optional parameters you can use with .read_excel() and .to_excel() to determine the Excel engine, the encoding, the way to handle missing values and infinities, the method for writing column names and row labels, and so on.

SQL Files

Pandas IO tools can also read and write databases. In this next example, you’ll write your data to a database called data.db. To get started, you’ll need the SQLAlchemy package. To learn more about it, you can read the official ORM tutorial. You’ll also need the database driver. Python has a built-in driver for SQLite.

You can install SQLAlchemy with pip:

$ pip install sqlalchemy

You can also install it with Conda:

$ conda install sqlalchemy

Once you have SQLAlchemy installed, import create_engine() and create a database engine:

>>>

>>> from sqlalchemy import create_engine
>>> engine = create_engine('sqlite:///data.db', echo=False)

Now that you have everything set up, the next step is to create a DataFrame object. It’s convenient to specify the data types and apply .to_sql().

>>>

>>> dtypes = {'POP': 'float64', 'AREA': 'float64', 'GDP': 'float64',
...           'IND_DAY': 'datetime64'}
>>> df = pd.DataFrame(data=data).T.astype(dtype=dtypes)
>>> df.dtypes
COUNTRY            object
POP               float64
AREA              float64
GDP               float64
CONT               object
IND_DAY    datetime64[ns]
dtype: object

.astype() is a very convenient method you can use to set multiple data types at once.

Once you’ve created your DataFrame, you can save it to the database with .to_sql():

>>>

>>> df.to_sql('data.db', con=engine, index_label='ID')

The parameter con is used to specify the database connection or engine that you want to use. The optional parameter index_label specifies how to call the database column with the row labels. You’ll often see it take on the value ID, Id, or id.

You should get the database data.db with a single table that looks like this:

The first column contains the row labels. To omit writing them into the database, pass index=False to .to_sql(). The other columns correspond to the columns of the DataFrame.

There are a few more optional parameters. For example, you can use schema to specify the database schema and dtype to determine the types of the database columns. You can also use if_exists, which says what to do if a database with the same name and path already exists:

if_exists='fail' raises a ValueError and is the default.
if_exists='replace' drops the table and inserts new values.
if_exists='append' inserts new values into the table.

You can load the data from the database with read_sql():

>>>

>>> df = pd.read_sql('data.db', con=engine, index_col='ID')
>>> df
        COUNTRY      POP      AREA       GDP       CONT    IND_DAY
ID
CHN       China  1398.72   9596.96  12234.78       Asia        NaT
IND       India  1351.16   3287.26   2575.67       Asia 1947-08-15
USA          US   329.74   9833.52  19485.39  N.America 1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia 1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America 1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia 1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa 1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia 1971-03-26
RUS      Russia   146.79  17098.25   1530.75       None 1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America 1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia        NaT
DEU     Germany    83.02    357.11   3693.20     Europe        NaT
FRA      France    67.02    640.68   2582.49     Europe 1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe        NaT
ITA       Italy    60.36    301.34   1943.84     Europe        NaT
ARG   Argentina    44.94   2780.40    637.49  S.America 1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa 1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America 1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania        NaT
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia 1991-12-16

The parameter index_col specifies the name of the column with the row labels. Note that this inserts an extra row after the header that starts with ID. You can fix this behavior with the following line of code:

>>>

>>> df.index.name = None
>>> df
        COUNTRY      POP      AREA       GDP       CONT    IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia        NaT
IND       India  1351.16   3287.26   2575.67       Asia 1947-08-15
USA          US   329.74   9833.52  19485.39  N.America 1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia 1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America 1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia 1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa 1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia 1971-03-26
RUS      Russia   146.79  17098.25   1530.75       None 1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America 1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia        NaT
DEU     Germany    83.02    357.11   3693.20     Europe        NaT
FRA      France    67.02    640.68   2582.49     Europe 1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe        NaT
ITA       Italy    60.36    301.34   1943.84     Europe        NaT
ARG   Argentina    44.94   2780.40    637.49  S.America 1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa 1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America 1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania        NaT
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia 1991-12-16

Now you have the same DataFrame object as before.

Note that the continent for Russia is now None instead of nan. If you want to fill the missing values with nan, then you can use .fillna():

>>>

>>> df.fillna(value=float('nan'), inplace=True)

.fillna() replaces all missing values with whatever you pass to value. Here, you passed float('nan'), which says to fill all missing values with nan.

Also note that you didn’t have to pass parse_dates=['IND_DAY'] to read_sql(). That’s because your database was able to detect that the last column contains dates. However, you can pass parse_dates if you’d like. You’ll get the same results.

There are other functions that you can use to read databases, like read_sql_table() and read_sql_query(). Feel free to try them out!

Pickle Files

Pickling is the act of converting Python objects into byte streams. Unpickling is the inverse process. Python pickle files are the binary files that keep the data and hierarchy of Python objects. They usually have the extension .pickle or .pkl.

You can save your DataFrame in a pickle file with .to_pickle():

>>>

>>> dtypes = {'POP': 'float64', 'AREA': 'float64', 'GDP': 'float64',
...           'IND_DAY': 'datetime64'}
>>> df = pd.DataFrame(data=data).T.astype(dtype=dtypes)
>>> df.to_pickle('data.pickle')

Like you did with databases, it can be convenient first to specify the data types. Then, you create a file data.pickle to contain your data. You could also pass an integer value to the optional parameter protocol, which specifies the protocol of the pickler.

You can get the data from a pickle file with read_pickle():

>>>

>>> df = pd.read_pickle('data.pickle')
>>> df
        COUNTRY      POP      AREA       GDP       CONT    IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia        NaT
IND       India  1351.16   3287.26   2575.67       Asia 1947-08-15
USA          US   329.74   9833.52  19485.39  N.America 1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia 1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America 1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia 1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa 1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia 1971-03-26
RUS      Russia   146.79  17098.25   1530.75        NaN 1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America 1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia        NaT
DEU     Germany    83.02    357.11   3693.20     Europe        NaT
FRA      France    67.02    640.68   2582.49     Europe 1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe        NaT
ITA       Italy    60.36    301.34   1943.84     Europe        NaT
ARG   Argentina    44.94   2780.40    637.49  S.America 1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa 1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America 1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania        NaT
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia 1991-12-16

read_pickle() returns the DataFrame with the stored data. You can also check the data types:

>>>

>>> df.dtypes
COUNTRY            object
POP               float64
AREA              float64
GDP               float64
CONT               object
IND_DAY    datetime64[ns]
dtype: object

These are the same ones that you specified before using .to_pickle().

As a word of caution, you should always beware of loading pickles from untrusted sources. This can be dangerous! When you unpickle an untrustworthy file, it could execute arbitrary code on your machine, gain remote access to your computer, or otherwise exploit your device in other ways.

Working With Big Data

If your files are too large for saving or processing, then there are several approaches you can take to reduce the required disk space:

Compress your files
Choose only the columns you want
Omit the rows you don’t need
Force the use of less precise data types
Split the data into chunks

You’ll take a look at each of these techniques in turn.

Compress and Decompress Files

You can create an archive file like you would a regular one, with the addition of a suffix that corresponds to the desired compression type:

'.gz'
'.bz2'
'.zip'
'.xz'

Pandas can deduce the compression type by itself:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df.to_csv('data.csv.zip')

Here, you create a compressed .csv file as an archive. The size of the regular .csv file is 1048 bytes, while the compressed file only has 766 bytes.

You can open this compressed file as usual with the Pandas read_csv() function:

>>>

>>> df = pd.read_csv('data.csv.zip', index_col=0,
...                  parse_dates=['IND_DAY'])
>>> df
        COUNTRY      POP      AREA       GDP       CONT    IND_DAY
CHN       China  1398.72   9596.96  12234.78       Asia        NaT
IND       India  1351.16   3287.26   2575.67       Asia 1947-08-15
USA          US   329.74   9833.52  19485.39  N.America 1776-07-04
IDN   Indonesia   268.07   1910.93   1015.54       Asia 1945-08-17
BRA      Brazil   210.32   8515.77   2055.51  S.America 1822-09-07
PAK    Pakistan   205.71    881.91    302.14       Asia 1947-08-14
NGA     Nigeria   200.96    923.77    375.77     Africa 1960-10-01
BGD  Bangladesh   167.09    147.57    245.63       Asia 1971-03-26
RUS      Russia   146.79  17098.25   1530.75        NaN 1992-06-12
MEX      Mexico   126.58   1964.38   1158.23  N.America 1810-09-16
JPN       Japan   126.22    377.97   4872.42       Asia        NaT
DEU     Germany    83.02    357.11   3693.20     Europe        NaT
FRA      France    67.02    640.68   2582.49     Europe 1789-07-14
GBR          UK    66.44    242.50   2631.23     Europe        NaT
ITA       Italy    60.36    301.34   1943.84     Europe        NaT
ARG   Argentina    44.94   2780.40    637.49  S.America 1816-07-09
DZA     Algeria    43.38   2381.74    167.56     Africa 1962-07-05
CAN      Canada    37.59   9984.67   1647.12  N.America 1867-07-01
AUS   Australia    25.47   7692.02   1408.68    Oceania        NaT
KAZ  Kazakhstan    18.53   2724.90    159.41       Asia 1991-12-16

read_csv() decompresses the file before reading it into a DataFrame.

You can specify the type of compression with the optional parameter compression, which can take on any of the following values:

'infer'
'gzip'
'bz2'
'zip'
'xz'
None

The default value compression='infer' indicates that Pandas should deduce the compression type from the file extension.

Here’s how you would compress a pickle file:

>>>

>>> df = pd.DataFrame(data=data).T
>>> df.to_pickle('data.pickle.compress', compression='gzip')

You should get the file data.pickle.compress that you can later decompress and read:

>>>

>>> df = pd.read_pickle('data.pickle.compress', compression='gzip')

df again corresponds to the DataFrame with the same data as before.

You can give the other compression methods a try, as well. If you’re using pickle files, then keep in mind that the .zip format supports reading only.

Choose Columns

The Pandas read_csv() and read_excel() functions have the optional parameter usecols that you can use to specify the columns you want to load from the file. You can pass the list of column names as the corresponding argument:

>>>

>>> df = pd.read_csv('data.csv', usecols=['COUNTRY', 'AREA'])
>>> df
       COUNTRY      AREA
0        China   9596.96
1        India   3287.26
2           US   9833.52
3    Indonesia   1910.93
4       Brazil   8515.77
5     Pakistan    881.91
6      Nigeria    923.77
7   Bangladesh    147.57
8       Russia  17098.25
9       Mexico   1964.38
10       Japan    377.97
11     Germany    357.11
12      France    640.68
13          UK    242.50
14       Italy    301.34
15   Argentina   2780.40
16     Algeria   2381.74
17      Canada   9984.67
18   Australia   7692.02
19  Kazakhstan   2724.90

Now you have a DataFrame that contains less data than before. Here, there are only the names of the countries and their areas.

Instead of the column names, you can also pass their indices:

>>>

>>> df = pd.read_csv('data.csv',index_col=0, usecols=[0, 1, 3])
>>> df
        COUNTRY      AREA
CHN       China   9596.96
IND       India   3287.26
USA          US   9833.52
IDN   Indonesia   1910.93
BRA      Brazil   8515.77
PAK    Pakistan    881.91
NGA     Nigeria    923.77
BGD  Bangladesh    147.57
RUS      Russia  17098.25
MEX      Mexico   1964.38
JPN       Japan    377.97
DEU     Germany    357.11
FRA      France    640.68
GBR          UK    242.50
ITA       Italy    301.34
ARG   Argentina   2780.40
DZA     Algeria   2381.74
CAN      Canada   9984.67
AUS   Australia   7692.02
KAZ  Kazakhstan   2724.90

Expand the code block below to compare these results with the file 'data.csv':

,COUNTRY,POP,AREA,GDP,CONT,IND_DAY
CHN,China,1398.72,9596.96,12234.78,Asia,
IND,India,1351.16,3287.26,2575.67,Asia,1947-08-15
USA,US,329.74,9833.52,19485.39,N.America,1776-07-04
IDN,Indonesia,268.07,1910.93,1015.54,Asia,1945-08-17
BRA,Brazil,210.32,8515.77,2055.51,S.America,1822-09-07
PAK,Pakistan,205.71,881.91,302.14,Asia,1947-08-14
NGA,Nigeria,200.96,923.77,375.77,Africa,1960-10-01
BGD,Bangladesh,167.09,147.57,245.63,Asia,1971-03-26
RUS,Russia,146.79,17098.25,1530.75,,1992-06-12
MEX,Mexico,126.58,1964.38,1158.23,N.America,1810-09-16
JPN,Japan,126.22,377.97,4872.42,Asia,
DEU,Germany,83.02,357.11,3693.2,Europe,
FRA,France,67.02,640.68,2582.49,Europe,1789-07-14
GBR,UK,66.44,242.5,2631.23,Europe,
ITA,Italy,60.36,301.34,1943.84,Europe,
ARG,Argentina,44.94,2780.4,637.49,S.America,1816-07-09
DZA,Algeria,43.38,2381.74,167.56,Africa,1962-07-05
CAN,Canada,37.59,9984.67,1647.12,N.America,1867-07-01
AUS,Australia,25.47,7692.02,1408.68,Oceania,
KAZ,Kazakhstan,18.53,2724.9,159.41,Asia,1991-12-16

You can see the following columns:

The column at index 0 contains the row labels.
The column at index 1 contains the country names.
The column at index 3 contains the areas.

Simlarly, read_sql() has the optional parameter columns that takes a list of column names to read:

>>>

>>> df = pd.read_sql('data.db', con=engine, index_col='ID',
...                  columns=['COUNTRY', 'AREA'])
>>> df.index.name = None
>>> df
        COUNTRY      AREA
CHN       China   9596.96
IND       India   3287.26
USA          US   9833.52
IDN   Indonesia   1910.93
BRA      Brazil   8515.77
PAK    Pakistan    881.91
NGA     Nigeria    923.77
BGD  Bangladesh    147.57
RUS      Russia  17098.25
MEX      Mexico   1964.38
JPN       Japan    377.97
DEU     Germany    357.11
FRA      France    640.68
GBR          UK    242.50
ITA       Italy    301.34
ARG   Argentina   2780.40
DZA     Algeria   2381.74
CAN      Canada   9984.67
AUS   Australia   7692.02
KAZ  Kazakhstan   2724.90

Again, the DataFrame only contains the columns with the names of the countries and areas. If columns is None or omitted, then all of the columns will be read, as you saw before. The default behavior is columns=None.

Omit Rows

When you test an algorithm for data processing or machine learning, you often don’t need the entire dataset. It’s convenient to load only a subset of the data to speed up the process. The Pandas read_csv() and read_excel() functions have some optional parameters that allow you to select which rows you want to load:

skiprows: either the number of rows to skip at the beginning of the file if it’s an integer, or the zero-based indices of the rows to skip if it’s a list-like object
skipfooter: the number of rows to skip at the end of the file
nrows: the number of rows to read

Here’s how you would skip rows with odd zero-based indices, keeping the even ones:

>>>

>>> df = pd.read_csv('data.csv', index_col=0, skiprows=range(1, 20, 2))
>>> df
        COUNTRY      POP     AREA      GDP       CONT     IND_DAY
IND       India  1351.16  3287.26  2575.67       Asia  1947-08-15
IDN   Indonesia   268.07  1910.93  1015.54       Asia  1945-08-17
PAK    Pakistan   205.71   881.91   302.14       Asia  1947-08-14
BGD  Bangladesh   167.09   147.57   245.63       Asia  1971-03-26
MEX      Mexico   126.58  1964.38  1158.23  N.America  1810-09-16
DEU     Germany    83.02   357.11  3693.20     Europe         NaN
GBR          UK    66.44   242.50  2631.23     Europe         NaN
ARG   Argentina    44.94  2780.40   637.49  S.America  1816-07-09
CAN      Canada    37.59  9984.67  1647.12  N.America  1867-07-01
KAZ  Kazakhstan    18.53  2724.90   159.41       Asia  1991-12-16

In this example, skiprows is range(1, 20, 2) and corresponds to the values 1, 3, …, 19. The instances of the Python built-in class range behave like sequences. The first row of the file data.csv is the header row. It has the index 0, so Pandas loads it in. The second row with index 1 corresponds to the label CHN, and Pandas skips it. The third row with the index 2 and label IND is loaded, and so on.

If you want to choose rows randomly, then skiprows can be a list or NumPy array with pseudo-random numbers, obtained either with pure Python or with NumPy.

Force Less Precise Data Types

If you’re okay with less precise data types, then you can potentially save a significant amount of memory! First, get the data types with .dtypes again:

>>>

>>> df = pd.read_csv('data.csv', index_col=0, parse_dates=['IND_DAY'])
>>> df.dtypes
COUNTRY            object
POP               float64
AREA              float64
GDP               float64
CONT               object
IND_DAY    datetime64[ns]
dtype: object

The columns with the floating-point numbers are 64-bit floats. Each number of this type float64 consumes 64 bits or 8 bytes. Each column has 20 numbers and requires 160 bytes. You can verify this with .memory_usage():

>>>

>>> df.memory_usage()
Index      160
COUNTRY    160
POP        160
AREA       160
GDP        160
CONT       160
IND_DAY    160
dtype: int64

.memory_usage() returns an instance of Series with the memory usage of each column in bytes. You can conveniently combine it with .loc[] and .sum() to get the memory for a group of columns:

>>>

>>> df.loc[:, ['POP', 'AREA', 'GDP']].memory_usage(index=False).sum()
480

This example shows how you can combine the numeric columns 'POP', 'AREA', and 'GDP' to get their total memory requirement. The argument index=False excludes data for row labels from the resulting Series object. For these three columns, you’ll need 480 bytes.

You can also extract the data values in the form of a NumPy array with .to_numpy() or .values. Then, use the .nbytes attribute to get the total bytes consumed by the items of the array:

>>>

>>> df.loc[:, ['POP', 'AREA', 'GDP']].to_numpy().nbytes
480

The result is the same 480 bytes. So, how do you save memory?

In this case, you can specify that your numeric columns 'POP', 'AREA', and 'GDP' should have the type float32. Use the optional parameter dtype to do this:

>>>

>>> dtypes = {'POP': 'float32', 'AREA': 'float32', 'GDP': 'float32'}
>>> df = pd.read_csv('data.csv', index_col=0, dtype=dtypes,
...                  parse_dates=['IND_DAY'])

The dictionary dtypes specifies the desired data types for each column. It’s passed to the Pandas read_csv() function as the argument that corresponds to the parameter dtype.

Now you can verify that each numeric column needs 80 bytes, or 4 bytes per item:

>>>

>>> df.dtypes
COUNTRY            object
POP               float32
AREA              float32
GDP               float32
CONT               object
IND_DAY    datetime64[ns]
dtype: object
>>> df.memory_usage()
Index      160
COUNTRY    160
POP         80
AREA        80
GDP         80
CONT       160
IND_DAY    160
dtype: int64
>>> df.loc[:, ['POP', 'AREA', 'GDP']].memory_usage(index=False).sum()
240
>>> df.loc[:, ['POP', 'AREA', 'GDP']].to_numpy().nbytes
240

Each value is a floating-point number of 32 bits or 4 bytes. The three numeric columns contain 20 items each. In total, you’ll need 240 bytes of memory when you work with the type float32. This is half the size of the 480 bytes you’d need to work with float64.

In addition to saving memory, you can significantly reduce the time required to process data by using float32 instead of float64 in some cases.

Use Chunks to Iterate Through Files

Another way to deal with very large datasets is to split the data into smaller chunks and process one chunk at a time. If you use read_csv(), read_json() or read_sql(), then you can specify the optional parameter chunksize:

>>>

>>> data_chunk = pd.read_csv('data.csv', index_col=0, chunksize=8)
>>> type(data_chunk)
<class 'pandas.io.parsers.TextFileReader'>
>>> hasattr(data_chunk, '__iter__')
True
>>> hasattr(data_chunk, '__next__')
True

chunksize defaults to None and can take on an integer value that indicates the number of items in a single chunk. When chunksize is an integer, read_csv() returns an iterable that you can use in a for loop to get and process only a fragment of the dataset in each iteration:

>>>

>>> for df_chunk in pd.read_csv('data.csv', index_col=0, chunksize=8):
...     print(df_chunk, end='\n\n')
...     print('memory:', df_chunk.memory_usage().sum(), 'bytes',
...           end='\n\n\n')
...
        COUNTRY      POP     AREA       GDP       CONT     IND_DAY
CHN       China  1398.72  9596.96  12234.78       Asia         NaN
IND       India  1351.16  3287.26   2575.67       Asia  1947-08-15
USA          US   329.74  9833.52  19485.39  N.America  1776-07-04
IDN   Indonesia   268.07  1910.93   1015.54       Asia  1945-08-17
BRA      Brazil   210.32  8515.77   2055.51  S.America  1822-09-07
PAK    Pakistan   205.71   881.91    302.14       Asia  1947-08-14
NGA     Nigeria   200.96   923.77    375.77     Africa  1960-10-01
BGD  Bangladesh   167.09   147.57    245.63       Asia  1971-03-26

memory: 448 bytes


       COUNTRY     POP      AREA      GDP       CONT     IND_DAY
RUS     Russia  146.79  17098.25  1530.75        NaN  1992-06-12
MEX     Mexico  126.58   1964.38  1158.23  N.America  1810-09-16
JPN      Japan  126.22    377.97  4872.42       Asia         NaN
DEU    Germany   83.02    357.11  3693.20     Europe         NaN
FRA     France   67.02    640.68  2582.49     Europe  1789-07-14
GBR         UK   66.44    242.50  2631.23     Europe         NaN
ITA      Italy   60.36    301.34  1943.84     Europe         NaN
ARG  Argentina   44.94   2780.40   637.49  S.America  1816-07-09

memory: 448 bytes


        COUNTRY    POP     AREA      GDP       CONT     IND_DAY
DZA     Algeria  43.38  2381.74   167.56     Africa  1962-07-05
CAN      Canada  37.59  9984.67  1647.12  N.America  1867-07-01
AUS   Australia  25.47  7692.02  1408.68    Oceania         NaN
KAZ  Kazakhstan  18.53  2724.90   159.41       Asia  1991-12-16

memory: 224 bytes

In this example, the chunksize is 8. The first iteration of the for loop returns a DataFrame with the first eight rows of the dataset only. The second iteration returns another DataFrame with the next eight rows. The third and last iteration returns the remaining four rows.

Note: You can also pass iterator=True to force the Pandas read_csv() function to return an iterator object instead of a DataFrame object.

In each iteration, you get and process the DataFrame with the number of rows equal to chunksize. It’s possible to have fewer rows than the value of chunksize in the last iteration. You can use this functionality to control the amount of memory required to process data and keep that amount reasonably small.

Conclusion

You now know how to save the data and labels from Pandas DataFrame objects to different kinds of files. You also know how to load your data from files and create DataFrame objects.

You’ve used the Pandas read_csv() and .to_csv() methods to read and write CSV files. You also used similar methods to read and write Excel, JSON, HTML, SQL, and pickle files. These functions are very convenient and widely used. They allow you to save or load your data in a single function or method call.

You’ve also learned how to save time, memory, and disk space when working with large data files:

Compress or decompress files
Choose the rows and columns you want to load
Use less precise data types
Split data into chunks and process them one by one

You’ve mastered a significant step in the machine learning and data science process! If you have any questions or comments, then please put them in the comments section below.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python KeyError Exceptions and How to Handle Them

2019-11-26T14:00:00+00:00

Python’s KeyError exception is a common exception encountered by beginners. Knowing why a KeyError can be raised and some solutions to prevent it from stopping your program are essential steps to improving as a Python programmer.

By the end of this course, you’ll know:

What a Python KeyError usually means
Where else you might see a KeyError in the standard library
How to handle a KeyError when you see it
When to raise a Python KeyError in your code

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Threading in Python

2019-11-19T14:00:00+00:00

Python threading allows you to have different parts of your program run concurrently and can simplify your design. If you’ve got some experience in Python and want to speed up your program using threads, then this course is for you!

In this course, you’ll learn:

What threads are
How to create threads and wait for them to finish
How to use a ThreadPoolExecutor
How to avoid race conditions
How to use the common tools that Python threading provides

This course assumes you’ve got the Python basics down pat and that you’re using at least version 3.6 to run the examples. If you need a refresher, you can start with the Python Learning Paths and get up to speed.

If you’re not sure if you want to use Python threading, asyncio, or multiprocessing, then you can check out Speed Up Your Python Program With Concurrency.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Thinking Recursively in Python

2019-11-12T14:00:00+00:00

In this course, you’ll learn about recursion. Recursion is a powerful tool you can use to solve a problem that can be broken down into smaller variations of itself. You can create very complex recursive algorithms with only a few lines of code.

You’ll cover:

What recursion is
How to define a recursive function
How practical examples of recursive functions work
How to maintain state
How to optimize recursion

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Cool New Features in Python 3.8

2019-11-05T14:00:00+00:00

In this course, you’ll get a look into the newest version of Python. On October 14th, 2019 the first official version of Python 3.8 became ready.

You’ll learn about the following:

Using assignment expressions to simplify some code constructs
Enforcing positional-only arguments in your own functions
Specifying more precise type hints
Using f-strings for simpler debugging

With a few exceptions, Python 3.8 contains many small improvements over the earlier versions. Towards the end of the course, you’ll see many of these less attention-grabbing changes, as well as a discussion about some of the optimizations that make Python 3.8 faster than its predecessors.

If you want to learn more, additional resources will be referenced and linked to throughout the course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Exploring HTTPS With Python

What Is HTTP?

What Is HTTPS?

Why Is HTTPS Important?

Creating an Example Application

Setting Up Wireshark

Seeing That Your Data Is Not Safe

How Does Cryptography Help?

Understanding Cryptography Basics

Using Cryptography in Python HTTPS Applications

Seeing That Your Data Is Safe

How Are Keys Shared?

What Is HTTPS Like in the Real World?

What Does a Python HTTPS Application Look Like?

Becoming a Certificate Authority

Trusting Your Server

Conclusion

Python args and kwargs: Demystified

Using Pandas and Python to Explore Your Dataset

Setting Up Your Environment

Using the Pandas Python Library

Getting to Know Your Data

Displaying Data Types

Showing Basics Statistics

Exploring Your Dataset

Getting to Know Pandas’ Data Structures

Understanding Series Objects

Understanding DataFrame Objects

Accessing Series Elements

Using the Indexing Operator

Using .loc and .iloc

Accessing DataFrame Elements

Using the Indexing Operator

Using .loc and .iloc

Querying Your Dataset

Grouping and Aggregating Your Data

Manipulating Columns

Specifying Data Types

Cleaning Data

Missing Values

Invalid Values

Inconsistent Values

Combining Multiple Datasets

Visualizing Your Pandas DataFrame

Conclusion

Sorting Data With Python

Python Timer Functions: Three Ways to Monitor Your Code

Python Timers

Python Timer Functions

Example: Download Tutorials

Your First Python Timer

A Python Timer Class

Understanding Classes in Python

Creating a Python Timer Class

Using the Python Timer Class

Adding More Convenience and Flexibility

A Python Timer Context Manager

Understanding Context Managers in Python

Creating a Python Timer Context Manager

Using the Python Timer Context Manager

A Python Timer Decorator

Understanding Decorators in Python

Creating a Python Timer Decorator

Using the Python Timer Decorator

The Python Timer Code

Other Python Timer Functions

Using Alternative Python Timer Functions

Estimating Running Time With timeit

Finding Bottlenecks in Your Code With Profilers

Conclusion

Resources

Python Dictionary Iteration: Advanced Tips & Tricks

NumPy, SciPy, and Pandas: Correlation With Python

Correlation

Example: NumPy Correlation Calculation

Example: SciPy Correlation Calculation

Example: Pandas Correlation Calculation

Linear Correlation

Pearson Correlation Coefficient

Using `.loc` and `.iloc`

Using `.loc` and `.iloc`

Estimating Running Time With `timeit`