Real Python

Alexa Python Development: Build and Deploy an Alexa Skill

2020-03-04T14:00:00+00:00

Smart home speakers were a novel idea just a couple of years ago. Today, they’ve become a central part of many people’s homes and offices and their adoption is only expected to grow. Among the most popular of these devices are those controlled by Amazon Alexa. In this tutorial, you’ll become an Alexa Python developer by deploying your own Alexa skill, an application that users will interact with using voice commands to Amazon Alexa devices.

In this tutorial, you’ll learn:

What the main components of an Alexa skill are
How to set up an Alexa skill and create Intents
What the ask_sdk_core Alexa Python package is
How to use ask_sdk_core to create the business logic of your Alexa Python skill
How to build, deploy, and test your Alexa Python skill using the online developer console

Free Bonus: Click here to download a Python speech recognition sample project with full source code that you can use as a basis for your own speech recognition apps.

Getting Started With Alexa Python Development

To follow this tutorial, you’ll need to make a free Alexa developer account. On that page, you’ll take the following steps:

Click the Get Started button.
Click the Sign-Up button on the subsequent page.
Click Create your Amazon Account.
Fill out the form with the required details.
Click Submit to complete the sign-up process.

You’ll also need to be familiar with concepts such as lists and dictionaries in Python, as well as JavaScript Object Notation (JSON). If you’re new to JSON, then check out Working With JSON Data in Python.

Let’s get started!

Understanding Alexa Skills

An Alexa Python developer must be familiar with a number of different Alexa skill components, but the two most important components are the interface and the service:

The skill interface processes the user’s speech inputs and maps it to an intent.
The skill service contains all the business logic that determines the response for a given user input and returns it as a JSON object.

The skill interface will be the frontend of your Alexa skill. This is where you’ll define the intents and the invocation phrases that will perform a certain function. Essentially, this is the part of the skill that’s responsible for interacting with the users.

The skill service will be the backend of your Alexa skill. When a specific intent is triggered by the user, it will send that information as a request to the skill service. This will contain the business logic to be returned along with valuable information to the frontend, which will finally be relayed back to the user.

Setting Up Your Environment

It’s time to start building your first Alexa Python skill! Sign in to the Alexa developer console and click on the Create Skill button to get started. On the next page, enter the Skill name, which will be Joke Bot:

This will be the invocation phrase of your skill. It’s the phrase a user will speak to start using your Alexa skill. You can change this to something else later on if you’d like. Also, note that Alexa skills can interact in many languages, which you can see from the Default Language dropdown menu. For now, just set it to English (US).

Next, you’ll need to choose a model to add to your skill. These models are like templates that have been pre-designed by the Amazon team to help you get started with Alexa Python development, based on some common use cases. For this tutorial, you should select the Custom model.

Finally, you need to select a method to host the backend of your Alexa skill. This service will contain the business logic of your application.

Note: If you select the Provision your own option, then you’ll have to host your own backend for your Alexa Python projects. This can be an API built and hosted on a platform of your choice. The other option is to create a separate AWS Lambda function and link it to your Alexa skill. You can learn more about AWS Lambda pricing on their pricing page.

For now, select Alexa-Hosted (Python) as the backend for your Alexa skill. This will automatically provide you with a hosted backend within the AWS free tier so you don’t have to pay anything upfront or set up a complicated backend right now.

Finally, click the Create Skill button to proceed. You might be asked to fill out a CAPTCHA here, so complete that as well. After a minute or so, you should be redirected to the Build section of the developer console.

Understanding the Alexa Skill Model

Once you’ve logged into the Alexa developer console and selected or created a skill, you’ll be greeted with the Build section. This section provides you with a lot of options and controls to set up the interaction model of the skill. The components of this interaction model allow you to define how the users will interact with your skill. These properties can be accessed through the left-side panel, which looks something like this:

As an Alexa Python developer, there are a few components of an Alexa skill interaction model that you’ll need to know about. The first is the invocation. This is what users will say to begin interacting with your Alexa skill. For example, the user will say, “Joke Bot,” to invoke the Alexa skill you’ll build in this tutorial. You can change this from the Invocation section at any time.

Another component is the intent, which represents the core functionality of your application. Your app will have a set of intents that will represent what kinds of actions your skill can perform. To provide contextual information for a given intent, you’ll use a slot, which is a variable in an utterance phrase.

Consider the following example. A sample utterance to invoke the weather intent could be, “Tell me about the weather.” To make the skill more useful, you can set the intent to be, “Tell me about the weather in Chicago,” where the word “Chicago” will be passed as a slot variable, which improves the user experience.

Lastly, there are slot types, which define how data in a slot is handled and recognized. For example, the AMAZON.DATE slot type easily converts words that indicate a date—like “today, “tomorrow”, and others—into a standard date format (such as “2019-07-05”). You can check out the official slot type reference page to learn more.

Note: To learn more about the Alexa skill interaction model, check out the official documentation.

At this point, the Intents panel should be open. If it’s not, then you can open it by selecting Intents from the sidebar on the left. You’ll notice that there are five intents already set up by default:

The Intents panel includes a HelloWorldIntent and five Built-in Intents. The built-in intents are there to remind you to account for some common cases that are important to making a user-friendly bot. Here’s a brief overview:

AMAZON.CancelIntent lets the user cancel a transaction or task. Examples include, “Never mind,” “Forget it,” “Exit,” and “Cancel,” though there are others.
AMAZON.HelpIntent provides help about how to use the skill. This could be used to return a sentence that serves as a manual for the user on how to interact with your skill.
AMAZON.StopIntent allows the user to exit the skill.
AMAZON.NavigateHomeIntent navigates the user to the device home screen (if a screen is being used) and ends the skill session.

By default, there are no sample utterances assigned to trigger these intents, so you’ll have to add those as well. Consider it part of your training as an Alexa Python developer. You can learn more about these built-in intents in the official documentation.

Viewing a Sample Intent

Later in this tutorial, you’ll learn how to make a new intent, but for now, it’s a good idea to take a look at some existing intents that are part of every new skill you create. To start, click the HelloWorldIntent to see its properties:

You can see the sample utterances that a user can speak to invoke this intent. When this intent is invoked, this information is sent to the backend service of your Alexa skill, which will then execute the required business logic and return a response.

Below this, you have the option to set up the Dialog Delegation Strategy, which allows you to delegate a specific dialog that you define to a particular intent. While you won’t cover this in this tutorial, you can read more about it in the official documentation.

Next, you have the option to define slots for some particular data that your intent is supposed to collect. For example, if you were to create an intent that tells the weather for a given day, then you’d have a Date slot here that would collect the date information and send it to your backend service.

Note: In addition, the Intent Confirmation option can be useful in a case when you’re collecting a number of different data points from your user in a single intent and you want to prompt the user before sending it on for further processing.

Whenever you make changes to an intent, you need to click the Save Model button to save it. Then, you can click the Build Model button to go ahead and test your Alexa Python skill.

It’s helpful to know that the interaction model of a skill can be completely represented in a JSON format. To see the current structure of your Alexa skill, click the JSON Editor option from the left side panel of the console:

If you make a change directly using the JSON editor, then the changes are also reflected in the developer console UI. To test this behavior, add a new intent and click Save Model.

Once you’ve made all the necessary changes to the interaction model of your skill, you can open the Test section of the developer console to test out your skill. Testing is an important part of becoming an Alexa Python developer, so be sure not to skip this step! Click the Test button from the top navigation bar on the developer console. By default, testing will be disabled. From the drop-down menu, select Development to start testing:

Here, you have a number of ways that you can test out your Alexa Python skill. Let’s do a quick test so that you can get an idea of how your Alexa skill will respond to an utterance.

Select the Alexa Simulator option from the left side panel, then enter the phrase, “Hey Alexa, open Joke Bot.” You can do this either by typing it in the input box or by using the Mic option. After a couple of seconds, a response will be returned back to you:

In addition to the voice response, you can also see the JSON Input that was sent to the backend service of your Alexa skill, as well as the JSON Output that was received back to the console:

Here’s what’s happened so far:

The JSON input object was constructed from input data that the user entered through voice or text.
The Alexa simulator packaged up the input along with other relevant metadata and sent it to the backend service. You can see this in the JSON Input box.
The backend service received the input JSON object and parsed it to check the type of the request. Then, it passed the JSON to the relevant intent handler function.
The intent handler function processed the input and gathered the required response, which is sent back as a JSON response to the Alexa simulator. You can see this in the JSON Output box.
The Alexa simulator parsed this JSON and read the speech response back to you.

Note: You can read about the JSON request-response mechanism for Alexa skills in the official docs.

Now that you have an overview of the different components of an Alexa skill and how information flows from one part to the other, it’s time to start building your Joke Bot! In the next section, you’ll put your Alexa Python developer skills to the test by creating a new intent.

Creating New Intents

Let’s start by creating the JokeIntent, which will return a random joke from a list to the user. Open the Build section of your Alexa developer console. Then, click the Add button next to the Intents option from the left side panel:

With the Create custom intent option selected, set the name to JokeIntent and then click the Create custom intent button:

Next, you need to add sample utterances that the user will speak to invoke this intent. These can be phrases like “Tell me a joke” or “I want to hear a joke.” Type in a phrase and click the plus sign (+) to add it as a sample utterance. Here’s what that should look like:

You can add more sample utterances, but for now, these will do just fine. Lastly, click the Save Model button in the top-left corner of the window to save these changes.

Remember, you’ll need to build your model before you can test it out. Click the Build Model button to re-build the interaction model of your Alexa Python skill. You’ll see a progress notification on the bottom-right of your browser window. Once the build process is successful, you should see another pop-up notification indicating the status of the build process.

You can check to see if the JokeIntent is successfully triggered or not. Click the Evaluate Model button in the top-right corner of the developer console. A small window will pop in from the side allowing you to check what intent will be triggered by a given input utterance. Type in any of the sample utterances to make sure that the JokeIntent is being invoked successfully.

To get rid of the evaluate pop-up window, click the Evaluate Model button again.

Note: A key thing to remember here is that the model is very flexible in terms of the keywords that are part of the sample utterance phrases. For example, take the phrase, “Is this some kind of a joke?” Even this phrase will trigger the JokeIntent. As an Alexa Python developer, it’s important to select utterances that have a low probability of executing other intents in your skill.

Now that you’ve successfully created an intent, it’s time to write the Python code that will handle this intent and return back a joke as a response.

Building the Skill Backend

Now that you have an intent created that can be triggered by the user, you need to add functionality in the skill backend to handle this intent and return useful information. Open the Code section of the Alexa developer console to get started.

Note: Since you selected the Alexa-Hosted Python option during the setup process, you’re provided with a complete online code editor where you can write, test, build, and deploy the backend of your Alexa skill, all within the developer console.

When you open the Code section of the developer console, you can see an online code editor with some files already set up for you to get started. In particular, you’ll see the following three files in the lambda sub-directory:

lambda_function.py: This is the main entry point of the backend service. All the request data from the Alexa intent is received here and is supposed to be returned from this file only.
requirements.txt: This file contains the list of Python packages that are being used in this project. This is especially useful if you’re choosing to set up your own backend service instead of using what’s provided by Amazon. To learn more about requirements files, check out Using Requirements Files.
utils.py: This file contains some utility functions required for the lambda function to interact with the Amazon S3 service. It contains some sample code on how to fetch data from an Amazon S3 bucket, which you might find useful later on. Right now, this file is not being used in lambda_function.py.

For now, you’ll only be making changes in lambda_function.py, so let’s take a closer look at the structure of the file:

 7 import logging
 8 import ask_sdk_core.utils as ask_utils
 9 
10 from ask_sdk_core.skill_builder import SkillBuilder
11 from ask_sdk_core.dispatch_components import AbstractRequestHandler
12 from ask_sdk_core.dispatch_components import AbstractExceptionHandler
13 from ask_sdk_core.handler_input import HandlerInput
14 
15 from ask_sdk_model import Response
16 
17 logger = logging.getLogger(__name__)
18 logger.setLevel(logging.INFO)
19 
20 
21 class LaunchRequestHandler(AbstractRequestHandler):
22     """Handler for Skill Launch."""
23     def can_handle(self, handler_input):
24         # type: (HandlerInput) -> bool
25 
26         return ask_utils.is_request_type("LaunchRequest")(handler_input)
27 
28     def handle(self, handler_input):
29         # type: (HandlerInput) -> Response
30         speak_output = "Welcome, you can say Hello or Help. " \
31                        "Which would you like to try?"
32 
33         return (
34             handler_input.response_builder
35                 .speak(speak_output)
36                 .ask(speak_output)
37                 .response
38         )
39 ...

First, you import the necessary utilities that were provided in the ask_sdk_core Alexa Python package. Then, there are three main tasks you need to perform in lambda_function.py to handle a request from an intent received from the front-end of the Alexa skill:

Create an intent handler class, which inherits from the AbstractRequestHandler class, with functions can_handle() and handle(). There are already a couple of handler classes defined in lambda_function.py, such as LaunchRequestHandler, HelpIntentHandler, and so on. These handle the fundamental intents of an Alexa skill. An important point to note here is that you need to create a new intent handler class for each of the intents you define.
Create a SkillBuilder object, which acts as the entry point for your Alexa Python skill. This routes all the incoming request and response payloads to the intent handlers that you define.
Pass the intent handler class as an argument to .add_request_handler() so that they’re called in order whenever a new request is received. The SkillBuilder is a singleton, so only one instance of it is needed to handle the routing of all incoming requests.

This is a good time for you to go through lambda_function.py. You’ll notice that the same pattern is followed over and over again to handle different intents that can be triggered by your Alexa Python skill.

Now that you have a broad overview of all the different things you need to do to handle an intent in your backend, it’s time to write the code that will handle the JokeIntent that you built in the previous section.

Creating the JokeIntent Handler

Since the important utilities from the ask_sdk_core Alexa Python package have already been imported, you don’t need to import them again. If you want to learn more about these in-depth, then you can check out the official documentation.

Next, you’ll create a new intent handler that will handle the request received from the JokeIntent. In the code snippet below, the intent handler will simply return with a sample phrase. This indicates that the response to the JokeIntent was received from the backend. Add the following code to lambda_function.py above the class definition of LaunchRequestHandler():

20 class JokeIntentHandler(AbstractRequestHandler):
21     def can_handle(self, handler_input):
22         return ask_utils.is_intent_name("JokeIntent")(handler_input)
23 
24     def handle(self, handler_input):
25         speak_output = "Here's a sample joke for you."
26 
27         return (
28             handler_input.response_builder
29                 .speak(speak_output)
30                 .ask(speak_output)
31                 .response
32         )

Let’s take a look at what each section does. In line 20 you create a new intent handler class for the JokeIntent, which is a child class of the AbstractRequestHandler class. When you create an intent in the frontend, you need to create an intent handler class in the backend that can handle requests from Alexa. The code you write for this needs to do two things:

JokeIntentHandler.can_handle() recognizes each incoming request that Alexa sends.
JokeIntentHandler.handle() returns an appropriate response.

In line 21 you define .can_handle(). It takes in handler_input as a parameter, which is an object of type dict() that contains all the input request information. Then, it uses ask_utils.is_intent_name() or ask_utils.is_request_type() to check whether the JSON input it received can be handled by this intent handler function or not.

You use .is_intent_name() and pass in the name of the intent. This returns a predicate, which is a function object that returns True if the given handler_input originates from the indicated intent. If this is true, then the SkillBuilder object will call JokeIntentHandler.handle().

Note: If the JokeIntent is triggered from the Alexa skill frontend, then it will send a JSON object containing a key type in the body of request that indicates that the intent named JokeIntent was received as input.

This statement subsequently calls .handle(), which you define in line 24. This method receives the input request along with any other important information that might be needed. It contains the business logic that’s required to successfully handle a particular intent. In the case of the JokeIntent, this method is required to send a response containing a joke back to the Alexa frontend.

The speak_ouput variable contains the sentence which will be spoken back to the user by the Alexa skill frontend. speak(speak_output) indicates what the Alexa frontend will play to the user as speech. ask("Question to ask...") can be used to ask a follow-up question. In this method, an object of class response_builder returns the response back to the Alexa skill.

Note: A default response message (Sorry, I had trouble doing what you asked. Please try again.) will be sent back if .handle() does not exist.

Notice that the value of speak_output is set to a fixed response right now. You’ll change this later on to return a random joke from a list of jokes.

Here’s what your code looks like in an editor:

Once you’ve created an intent handler class, you need to pass it as an argument to SkillBuilder.add_request_handler. Scroll to the bottom of lambda_function.py and add the following line:

sb.add_request_handler(JokeIntentHandler())

An important thing to note here is that the placement of this line is important, as the code is processed from top to bottom. So, make sure that the call for your custom intent handler is above the call for the InstantReflectHandler() class. This is how it should look:

171 sb = SkillBuilder()
172 
173 sb.add_request_handler(LaunchRequestHandler())
174 sb.add_request_handler(JokeIntentHandler())
175 sb.add_request_handler(HelloWorldIntentHandler())
176 sb.add_request_handler(HelpIntentHandler())
177 sb.add_request_handler(CancelOrStopIntentHandler())
178 sb.add_request_handler(SessionEndedRequestHandler())
179 
180 # Make sure IntentReflectorHandler is last so it
181 # Doesn't override your custom intent handlers
182 sb.add_request_handler(IntentReflectorHandler())
183 
184 sb.add_exception_handler(CatchAllExceptionHandler())
185 
186 ...

Alright, it’s time to test your code! Click the Deploy button to save the changes and deploy the backend service. You’ll be checking whether it’s going to work as expected from the Alexa skill frontend.

Once the Deploy process is successful, head back to the Test section of the developer console and invoke the JokeIntent. Remember, enter the utterance phrase to invoke your Alexa Python skill, then input a phrase to execute an intent:

If you get a response similar to the one in the image above, then it means you’ve successfully created an intent handler for the JokeIntent in your skill’s backend service. Congratulations! Now, all that’s left to do is to return a random joke from a list back to the skill frontend.

Adding Jokes

Open the Code section of the developer console. Then, add the jokes variable in lambda_function.py:

15 from ask_sdk_model import Response
16 
17 logger = logging.getLogger(__name__)
18 logger.setLevel(logging.INFO)
19 
20 jokes = [
21         "Did you hear about the semi-colon that broke the law? He was given two consecutive sentences.",
22         "I ate a clock yesterday, it was very time-consuming.",
23         "I've just written a song about tortillas; actually, it's more of a rap.",
24         "I woke up this morning and forgot which side the sun rises from, then it dawned on me.",
25         "I recently decided to sell my vacuum cleaner as all it was doing was gathering dust.",
26         "If you shouldn't eat at night, why do they put a light in the fridge?",
27         ]
28 
29 class JokeIntentHandler(AbstractRequestHandler):
30 ...

Here, jokes is a variable of type list containing some one-liner jokes. Make sure to add this outside of a function or class definition so that it has global scope.

Note: Since this list will only be referenced by the JokeIntentHandler() class, it doesn’t really matter if you declare this in the body of a function or not. However, doing it this way does help the function body to be free of clutter.

Next, you’ll add the functionality that .handle() needs to randomly pick one joke from the list of jokes and return it to the user. Modify the body of JokeIntentHandler.handle() with the following code:

29 class JokeIntentHandler(AbstractRequestHandler):
30     def can_handle(self, handler_input):
31         return ask_utils.is_intent_name("JokeIntent")(handler_input)
32 
33     def handle(self, handler_input):
34         speak_output = random.choice(jokes)
35 
36         return (
37             handler_input.response_builder
38                 .speak(speak_output)
39                 .ask(speak_output)
40                 .response
41         )

In the body of .handle(), you select a random joke from the list jokes using random.choice() and return it back as a response to the Alexa skill frontend.

Finally, import the random package by adding an import statement to the top of lambda_function.py:

15 from ask_sdk_model import Response
16 
17 import random
18 
19 logger = logging.getLogger(__name__)
20 logger.setLevel(logging.INFO)
21 
22 ...

This is how the editor should look at this point:

There’s one final change to make before testing. You need to allow Alexa to give an acknowledgment that the skill has been triggered. To do this, look inside LaunchRequestHandler.handle() for the speak_output variable and set its value to the text in the highlighted line below:

45 class LaunchRequestHandler(AbstractRequestHandler):
46     """Handler for Skill Launch."""
47     def can_handle(self, handler_input):
48         # type: (HandlerInput) -> bool
49 
50         return ask_utils.is_request_type("LaunchRequest")(handler_input)
51 
52     def handle(self, handler_input):
53         # type: (HandlerInput) -> Response
54         speak_output = "Hey there! I am a Joke Bot. You can ask me to tell you a random Joke that might just make your day better!"
55 
56         return (
57             handler_input.response_builder
58                 .speak(speak_output)
59                 .ask(speak_output)
60                 .response
61         )
62 ...

Your Joke Bot is ready for final testing! Click the Deploy button to save the changes and head back to the Test section of the developer console. This time, you should see a new greeting message when your skill is first invoked. Then, when you ask the bot to tell you a joke, it should give you a different joke every time:

That’s it! You’ve successfully created your first skill as an Alexa Python developer!

Conclusion

Congratulations on taking your first steps into Alexa Python development! You’ve now successfully built your very own Alexa Python skill. You now know how to create a new skill, create intents, write Python code to handle those intents, and return valuable information back to the user.

Level-up your skills by trying some of the following:

Increase the list of jokes in the backend.
Create a new Intent named Trivia which will respond with a fun trivia fact.
Publish your skill to the Amazon Marketplace.

The possibilities are endless, so go ahead and dive right in! To learn more about Alexa Python development, check out the official docs. You can also check out How to Make a Twitter Bot in Python With Tweepy and How to Make a Discord Bot in Python to learn more about how you can make bots for different platforms using Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Implement a Python Stack

2020-03-03T14:00:00+00:00

Have you heard of stacks and wondered what they are? Do you have a general idea but are wondering how to implement a Python stack? You’ve come to the right place!

In this course, you’ll learn:

How to recognize when a stack is a good choice for a data structure
How to decide which implementation is best for your program
What extra considerations to make about stacks in a threading or multiprocessing environment

This course is for Pythonistas who are comfortable running scripts, know what a list is and how to use it, and are wondering how to implement Python stacks.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Bindings: Calling C or C++ From Python

2020-03-02T14:00:00+00:00

Are you a Python developer with a C or C++ library you’d like to use from Python? If so, then Python bindings allow you to call functions and pass data from Python to C or C++, letting you take advantage of the strengths of both languages. Throughout this tutorial, you’ll see an overview of some of the tools you can use to create Python bindings.

In this tutorial, you’ll learn about:

Why you want to call C or C++ from Python
How to pass data between C and Python
What tools and methods can help you create Python bindings

This tutorial is aimed at intermediate Python developers. It assumes basic knowledge of Python and some understanding of functions and data types in C or C++. You can get all of the example code you’ll see in this tutorial by clicking on the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about Python Bindings in this tutorial.

Let’s dive into looking at Python bindings!

Python Bindings Overview

Before you dive into how to call C from Python, it’s good to spend some time on why. There are several situations where creating Python bindings to call a C library is a great idea:

You already have a large, tested, stable library written in C++ that you’d like to take advantage of in Python. This may be a communication library or a library to talk to a specific piece of hardware. What it does is unimportant.
You want to speed up a particular section of your Python code by converting a critical section to C. Not only does C have faster execution speed, but it also allows you to break free from the limitations of the GIL, provided you’re careful.
You want to use Python test tools to do large-scale testing of their systems.

All of the above are great reasons to learn to create Python bindings to interface with your C library.

Note: Throughout this tutorial, you’ll be creating Python bindings to both C and C++. Most of the general concepts apply to both languages, and so C will be used unless there’s a specific difference between the two languages. In general, each of the tools will support either C or C++, but not both.

Let’s get started!

Marshalling Data Types

Wait! Before you start writing Python bindings, take a look at how Python and C store data and what types of issues this will cause. First, let’s define marshalling. This concept is defined by Wikipedia as follows:

The process of transforming the memory representation of an object to a data format suitable for storage or transmission. (Source)

For your purposes, marshalling is what the Python bindings are doing when they prepare data to move it from Python to C or vice versa. Python bindings need to do marshalling because Python and C store data in different ways. C stores data in the most compact form in memory possible. If you use an uint8_t, then it will only use 8 bits of memory total.

In Python, on the other hand, everything is an object. This means that each integer uses several bytes in memory. How many will depend on which version of Python you’re running, your operating system, and other factors. This means that your Python bindings will need to convert a C integer to a Python integer for each integer passed across the boundary.

Other data types have similar relationships between the two languages. Let’s look at each in turn:

Integers store counting numbers. Python stores integers with arbitrary precision, meaning that you can store very, very, large numbers. C specifies the exact sizes of integers. You need to be aware of data sizes when you’re moving between languages to prevent Python integer values from overflowing C integer variables.
Floating-point numbers are numbers with a decimal place. Python can store much larger (and much smaller) floating-point numbers than C. This means that you’ll also have to pay attention to those values to ensure they stay in range.
Complex numbers are numbers with an imaginary part. While Python has built-in complex numbers, and C has complex numbers, there’s no built-in method for marshalling between them. To marshal complex numbers, you’ll need to build a struct or class in the C code to manage them.
Strings are sequences of characters. For being such a common data type, strings will prove to be rather tricky when you’re creating Python bindings. As with other data types, Python and C store strings in quite different formats. (Unlike the other data types, this is an area where C and C++ differ as well, which adds to the fun!) Each of the solutions you’ll examine have slightly different methods for dealing with strings.
Boolean variables can have only two values. Since they’re supported in C, marshalling them will prove to be fairly straightforward.

Besides data type conversions, there are other issues you’ll need to think about as you build your Python bindings. Let’s keep exploring them.

Understanding Mutable and Immutable Values

In addition to all of these data types, you’ll also have to be aware of how Python objects can be mutable or immutable. C has a similar concept with function parameters when talking about pass-by-value or pass-by-reference. In C, all parameters are pass-by-value. If you want to allow a function to change a variable in the caller, then you need to pass a pointer to that variable.

You might be wondering if you can get around the immutable restriction by simply passing an immutable object to C using a pointer. Unless you go to ugly and non-portable extremes, Python won’t give you a pointer to an object, so this just doesn’t work. If you want to modify a Python object in C, then you’ll need to take extra steps to achieve this. These steps will be dependent on which tools you use, as you’ll see below.

So, you can add immutability to your checklist of items to consider as you create Python bindings. Your final stop on the grand tour of creating this checklist is how to handle the different ways in which Python and C deal with memory management.

Managing Memory

C and Python manage memory differently. In C, the developer must manage all memory allocations and ensure they’re freed once and only once. Python takes care of this for you using a garbage collector.

While each of these approaches has its advantages, it does add an extra wrinkle into creating Python bindings. You’ll need to be aware of where the memory for each object was allocated and ensure that it’s only freed on the same side of the language barrier.

For example, a Python object is created when you set x = 3. The memory for this is allocated on the Python side and needs to be garbage collected. Fortunately, with Python objects, it’s quite difficult to do anything else. Take a look at the converse in C, where you directly allocate a block of memory:

int* iPtr = (int*)malloc(sizeof(int));

When you do this, you need to ensure that this pointer is freed in C. This may mean manually adding code to your Python bindings to do this.

That rounds out your checklist of general topics. Let’s start setting up your system so you can write some code!

Setting Up Your Environment

For this tutorial, you’re going to be using pre-existing C and C++ libraries from the Real Python GitHub repo to show a test of each tool. The intent is that you’ll be able to use these ideas for any C library. To follow along with all of the examples here, you’ll need to have the following:

A C++ library installed and knowledge of the path for command-line invocation
Python development tools:
- For Linux, this is the python3-dev or python3-devel package, depending on your distro.
- For Windows, there are multiple options.
Python 3.6 or greater
A virtual environment (recommended, but not required)
The invoke tool

The last one might be new to you, so let’s take a closer look at it.

Using the `invoke` Tool

invoke is the tool you’ll be using to build and test your Python bindings in this tutorial. It has a similar purpose to make but uses Python instead of Makefiles. You’ll need to install invoke in your virtual environment using pip:

$ python3 -m pip install invoke

To run it, you type invoke followed by the task you wish to execute:

$ invoke build-cmult
==================================================
= Building C Library
* Complete

To see which tasks are available, you use the --list option:

$ invoke --list
Available tasks:

  all              Build and run all tests
  build-cffi       Build the CFFI Python bindings
  build-cmult      Build the shared library for the sample C code
  build-cppmult    Build the shared library for the sample C++ code
  build-cython     Build the cython extension module
  build-pybind11   Build the pybind11 wrapper library
  clean            Remove any built objects
  test-cffi        Run the script to test CFFI
  test-ctypes      Run the script to test ctypes
  test-cython      Run the script to test Cython
  test-pybind11    Run the script to test PyBind11

Note that when you look in the tasks.py file where the invoke tasks are defined, you’ll see that the name of the second task listed is build_cffi. However, the output from --list shows it as build-cffi. The minus sign (-) can’t be used as part of a Python name, so the file uses an underscore (_) instead.

For each of the tools you’ll examine, there will be a build- and a test- task defined. For example, to run the code for CFFI, you could type invoke build-cffi test-cffi. An exception is ctypes, as there’s no build phase for ctypes. In addition, there are two special tasks added for convenience:

invoke all runs the build and test tasks for all tools.
invoke clean removes any generated files.

Now that you’ve got a feeling for how to run the code, let’s take a peek at the C code you’ll be wrapping before hitting the tools overview.

C or C++ Source

In each of the example sections below, you’ll be creating Python bindings for the same function in either C or C++. These sections are intended to give you a taste of what each method looks like, rather than an in-depth tutorial on that tool, so the function you’ll wrap is small. The function you’ll create Python bindings for takes an int and a float as input parameters and returns a float that’s the product of the two numbers:

// cmult.c
float cmult(int int_param, float float_param) {
    float return_value = int_param * float_param;
    printf("    In cmult : int: %d float %.1f returning  %.1f\n", int_param,
            float_param, return_value);
    return return_value;
}

The C and C++ functions are almost identical, with minor name and string differences between them. You can get a copy of all of the code by clicking on the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about Python Bindings in this tutorial.

Now you’ve got the repo cloned and your tools installed, you can build and test the tools. So let’s dive into each section below!

`ctypes`

You’ll start with ctypes, which is a tool in the standard library for creating Python bindings. It provides a low-level toolset for loading shared libraries and marshalling data between Python and C.

How It’s Installed

One of the big advantages of ctypes is that it’s part of the Python standard library. It was added in Python version 2.5, so it’s quite likely you already have it. You can import it just like you do with the sys or time modules.

Calling the Function

All of the code to load your C library and call the function will be in your Python program. This is great since there are no extra steps in your process. You just run your program, and everything is taken care of. To create your Python bindings in ctypes, you need to do these steps:

Load your library.
Wrap some of your input parameters.
Tell ctypes the return type of your function.

You’ll look at each of these in turn.

Library Loading

ctypes provides several ways for you to load a shared library, some of which are platform-specific. For your example, you’ll create a ctypes.CDLL object directly by passing in the full path to the shared library you want:

# ctypes_test.py
import ctypes
import pathlib

if __name__ == "__main__":
    # Load the shared library into ctypes
    libname = pathlib.Path().absolute() / "libcmult.so"
    c_lib = ctypes.CDLL(libname)

This will work for cases when the shared library is in the same directory as your Python script, but be careful when you attempt to load libraries that are from packages other than your Python bindings. There are many details for loading libraries and finding paths in the ctypes documentation that are platform and situation-specific.

NOTE: Many platform-specific issues can arise during library loading. It’s best to make incremental changes once you get an example working.

Now that you have the library loaded into Python, you can try calling it!

Calling Your Function

Remember that the function prototype for your C function is as follows:

// cmult.h
float cmult(int int_param, float float_param);

You need to pass in an integer and a float and can expect to get a float returned. Integers and floats have native support in both Python and in C, so you expect this case will work for reasonable values.

Once you’ve loaded the library into your Python bindings, the function will be an attribute of c_lib, which is the CDLL object you created earlier. You can try to call it just like this:

x, y = 6, 2.3
answer = c_lib.cmult(x, y)

Oops! This doesn’t work. This line is commented out in the example repo because it fails. If you attempt to run with that call, then Python will complain with an error:

$ invoke test-ctypes
Traceback (most recent call last):
  File "ctypes_test.py", line 16, in <module>
    answer = c_lib.cmult(x, y)
ctypes.ArgumentError: argument 2: <class 'TypeError'>: Don't know how to convert parameter 2

It looks like you need to tell ctypes about any parameters that aren’t integers. ctypes doesn’t have any knowledge of the function unless you tell it explicitly. Any parameter that’s not marked otherwise is assumed to be an integer. ctypes doesn’t know how to convert the value 2.3 that’s stored in y to an integer, so it fails.

To fix this, you’ll need to create a c_float from the number. You can do that in the line where you’re calling the function:

# ctypes_test.py
answer = c_lib.cmult(x, ctypes.c_float(y))
print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Now, when you run this code, it returns the product of the two numbers you passed in:

$ invoke test-ctypes
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 48.0

Wait a minute… 6 multiplied by 2.3 is not 48.0!

It turns out that, much like the input parameters, ctypes assumes your function returns an int. In actuality, your function returns a float, which is getting marshalled incorrectly. Just like the input parameter, you need to tell ctypes to use a different type. The syntax here is slightly different:

# ctypes_test.py
c_lib.cmult.restype = ctypes.c_float
answer = c_lib.cmult(x, ctypes.c_float(y))
print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

That should do the trick. Let’s run the entire test-ctypes target and see what you’ve got. Remember, the first section of output is before you fixed the restype of the function to be a float:

$ invoke test-ctypes
==================================================
= Building C Library
* Complete
==================================================
= Testing ctypes Module
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 48.0

    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

That’s better! While the first, uncorrected version is returning the wrong value, your fixed version agrees with the C function. Both C and Python get the same result! Now that it’s working, take a look at why you may or may not want to use ctypes.

Strengths and Weaknesses

The biggest advantage ctypes has over the other tools you’ll examine here is that it’s built into the standard library. It also requires no extra steps, as all of the work is done as part of your Python program.

In addition, the concepts used are low-level, which makes exercises like the one you just did manageable. However, more complex tasks grow cumbersome with the lack of automation. In the next section, you’ll see a tool that adds some automation to the process.

`CFFI`

CFFI is the C Foreign Function Interface for Python. It takes a more automated approach to generate Python bindings. CFFI has multiple ways in which you can build and use your Python bindings. There are two different options to select from, which gives you four possible modes:

ABI vs API: API mode uses a C compiler to generate a full Python module, whereas ABI mode loads the shared library and interacts with it directly. Without running the compiler, getting the structures and parameters correct is error-prone. The documentation heavily recommends using the API mode.
in-line vs out-of-line: The difference between these two modes is a trade-off between speed and convenience:
- In-line mode compiles the Python bindings every time your script runs. This is convenient, as you don’t need an extra build step. It does, however, slow down your program.
- Out-of-line mode requires an extra step to generate the Python bindings a single time and then uses them each time the program is run. This is much faster, but that may not matter for your application.

For this example, you’ll use the API out-of-line mode, which produces the fastest code and, in general, looks similar to other Python bindings you’ll create later in this tutorial.

How It’s Installed

Since CFFI is not a part of the standard library, you’ll need to install it on your machine. It’s recommended that you create a virtual environment for this. Fortunately, CFFI installs with pip:

$ python3 -m pip install cffi

This will install the package into your virtual environment. If you’ve already installed from the requirements.txt, then this should be taken care of. You can take a look at requirements.txt by accessing the repo at the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about Python Bindings in this tutorial.

Now that you have CFFI installed, it’s time to take it for a spin!

Calling the Function

Unlike ctypes, with CFFI you’re creating a full Python module. You’ll be able to import the module like any other module in the standard library. There is some extra work you’ll have to do to build your Python module. To use your CFFI Python bindings, you’ll need to take the following steps:

Write some Python code describing the bindings.
Run that code to generate a loadable module.
Modify the calling code to import and use your newly created module.

That might seem like a lot of work, but you’ll walk through each of these steps and see how it works.

Write the Bindings

CFFI provides methods to read a C header file to do most of the work when generating Python bindings. In the documentation for CFFI, the code to do this is placed in a separate Python file. For this example, you’ll place that code directly into the build tool invoke, which uses Python files as input. To use CFFI, you start by creating a cffi.FFI object, which provides the three methods you need:

# tasks.py
import cffi
...
""" Build the CFFI Python bindings """
print_banner("Building CFFI Module")
ffi = cffi.FFI()

Once you have the FFI, you’ll use .cdef() to process the contents of the header file automatically. This creates wrapper functions for you to marshal data from Python:

# tasks.py
this_dir = pathlib.Path().absolute()
h_file_name = this_dir / "cmult.h"
with open(h_file_name) as h_file:
    ffi.cdef(h_file.read())

Reading and processing the header file is the first step. After that, you need to use .set_source() to describe the source file that CFFI will generate:

# tasks.py
ffi.set_source(
    "cffi_example",
    # Since you're calling a fully-built library directly, no custom source
    # is necessary. You need to include the .h files, though, because behind
    # the scenes cffi generates a .c file that contains a Python-friendly
    # wrapper around each of the functions.
    '#include "cmult.h"',
    # The important thing is to include the pre-built lib in the list of
    # libraries you're linking against:
    libraries=["cmult"],
    library_dirs=[this_dir.as_posix()],
    extra_link_args=["-Wl,-rpath,."],
)

Here’s a breakdown of the parameters you’re passing in:

"cffi_example" is the base name for the source file that will be created on your file system. CFFI will generate a .c file, compile it to a .o file, and link it to a .<system-description>.so or .<system-description>.dll file.
'#include "cmult.h"' is the custom C source code that will be included in the generated source before it’s compiled. Here, you just include the .h file for which you’re generating bindings, but this can be used for some interesting customizations.
libraries=["cmult"] tells the linker the name of your pre-existing C library. This is a list, so you can specify several libraries if required.
library_dirs=[this_dir.as_posix(),] is a list of directories that tells the linker where to look for the above list of libraries.
extra_link_args=['-Wl,-rpath,.'] is a set of options that generate a shared object, which will look in the current path (.) for other libraries it needs to load.

Build the Python Bindings

Calling .set_source() doesn’t build the Python bindings. It only sets up the metadata to describe what will be generated. To build the Python bindings, you need to call .compile():

# tasks.py
ffi.compile()

This wraps things up by generating the .c file, .o file, and the shared library. The invoke task you just walked through can be run on the command line to build the Python bindings:

$ invoke build-cffi
==================================================
= Building C Library
* Complete
==================================================
= Building CFFI Module
* Complete

You have your CFFI Python bindings, so it’s time to run this code!

Calling Your Function

After all of the work you did to configure and run the CFFI compiler, using the generated Python bindings looks just like using any other Python module:

# cffi_test.py
import cffi_example

if __name__ == "__main__":
    # Sample data for your call
    x, y = 6, 2.3

    answer = cffi_example.lib.cmult(x, y)
    print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

You import the new module, and then you can call cmult() directly. To test it out, use the test-cffi task:

$ invoke test-cffi
==================================================
= Testing CFFI Module
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

This runs your cffi_test.py program, which tests out the new Python bindings you’ve created with CFFI. That completes the section on writing and using your CFFI Python bindings.

Strengths and Weaknesses

It might seem that ctypes requires less work than the CFFI example you just saw. While this is true for this use case, CFFI scales to larger projects much better than ctypes due to automation of much of the function wrapping.

CFFI also produces quite a different user experience. ctypes allows you to load a pre-existing C library directly into your Python program. CFFI, on the other hand, creates a new Python module that can be loaded like other Python modules.

What’s more, with the out-of-line-API method you used above, the time penalty for creating the Python bindings is done once when you build it and doesn’t happen each time you run your code. For small programs, this might not be a big deal, but CFFI scales better to larger projects in this way, as well.

Like ctypes, using CFFI only allows you to interface with C libraries directly. C++ libraries require a good deal of work to use. In the next section, you’ll see a Python bindings tool that focuses on C++.

`PyBind11`

PyBind11 takes a quite different approach to create Python bindings. In addition to shifting the focus from C to C++, it also uses C++ to specify and build the module, allowing it to take advantage of the metaprogramming tools in C++. Like CFFI, the Python bindings generated from PyBind11 are a full Python module that can be imported and used directly.

PyBind11 is modeled after the Boost::Python library and has a similar interface. It restricts its use to C++11 and newer, however, which allows it to simplify and speed things up compared to Boost, which supports everything.

How It’s Installed

The First Steps section of the PyBind11 documentation walks you through how to download and build the test cases for PyBind11. While this doesn’t appear to be strictly required, working through these steps will ensure you’ve got the proper C++ and Python tools set up.

Note: Most of the examples for PyBind11 use cmake, which is a fine tool for building C and C++ projects. For this demo, however, you’ll continue to use the invoke tool, which follows the instructions in the Building Manually section of the docs.

You’ll want to install this tool into your virtual environment:

$ python3 -m pip install pybind11

PyBind11 is an all-header library, similar to much of Boost. This allows pip to install the actual C++ source for the library directly into your virtual environment.

Calling the Function

Before you dive in, please note that you’re using a different C++ source file, cppmult.cpp, instead of the C file you used for the previous examples. The function is essentially the same in both languages.

Writing the Bindings

Similar to CFFI, you need to create some code to tell the tool how to build your Python bindings. Unlike CFFI, this code will be in C++ instead of Python. Fortunately, there’s a minimal amount of code required:

// pybind11_wrapper.cpp
#include <pybind11/pybind11.h>
#include <cppmult.hpp>

PYBIND11_MODULE(pybind11_example, m) {
    m.doc() = "pybind11 example plugin"; // Optional module docstring
    m.def("cpp_function", &cppmult, "A function that multiplies two numbers");
}

Let’s look at this a piece at a time, as PyBind11 packs a lot of information into a few lines.

The first two lines include the pybind11.h file and the header file for your C++ library, cppmult.hpp. After that, you have the PYBIND11_MODULE macro. This expands into a block of C++ code that’s well described in the PyBind11 source:

This macro creates the entry point that will be invoked when the Python interpreter imports an extension module. The module name is given as the first argument and it should not be in quotes. The second macro argument defines a variable of type py::module which can be used to initialize the module. (Source)

What this means for you is that, for this example, you’re creating a module called pybind11_example and that the rest of the code will use m as the name of the py::module object. On the next line, inside the C++ function you’re defining, you create a docstring for the module. While this is optional, it’s a nice touch to make your module more Pythonic.

Finally, you have the m.def() call. This will define a function that’s exported by your new Python bindings, meaning it will be visible from Python. In this example, you’re passing three parameters:

cpp_function is the exported name of the function that you’ll use in Python. As this example shows, it doesn’t need to match the name of the C++ function.
&cppmult takes the address of the function to be exported.
"A function..." is an optional docstring for the function.

Now that you have the code for the Python bindings, take a look at how you can build this into a Python module.

Build the Python Bindings

The tool you use to build the Python bindings in PyBind11 is the C++ compiler itself. You may need to modify the defaults for your compiler and operating system.

To begin, you must build the C++ library for which you’re creating bindings. For an example this small, you could build the cppmult library directly into the Python bindings library. However, for most real-world examples, you’ll have a pre-existing library you want to wrap, so you’ll build the cppmult library separately. The build is a standard call to the compiler to build a shared library:

# tasks.py
invoke.run(
    "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC cppmult.cpp "
    "-o libcppmult.so "
)

Running this with invoke build-cppmult produces libcppmult.so:

$ invoke build-cppmult
==================================================
= Building C++ Library
* Complete

The build for the Python bindings, on the other hand, requires some special details:

 1 # tasks.py
 2 invoke.run(
 3     "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
 4     "`python3 -m pybind11 --includes` "
 5     "-I /usr/include/python3.7 -I .  "
 6     "{0} "
 7     "-o {1}`python3.7-config --extension-suffix` "
 8     "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
 9 )

Let’s walk through this line-by-line. Line 3 contains fairly standard C++ compiler flags that indicate several details, including that you want all warnings caught and treated as errors, that you want a shared library, and that you’re using C++11.

Line 4 is the first step of the magic. It calls the pybind11 module to have it produce the proper include paths for PyBind11. You can run this command directly on the console to see what it does:

$ python3 -m pybind11 --includes
-I/home/jima/.virtualenvs/realpython/include/python3.7m
-I/home/jima/.virtualenvs/realpython/include/site/python3.7

Your output should be similar but show different paths.

In line 5 of your compilation call, you can see that you’re also adding the path to the Python dev includes. While it’s recommended that you don’t link against the Python library itself, the source needs some code from Python.h to work its magic. Fortunately, the code it uses is fairly stable across Python versions.

Line 5 also uses -I . to add the current directory to the list of include paths. This allows the #include <cppmult.hpp> line in your wrapper code to be resolved.

Line 6 specifies the name of your source file, which is pybind11_wrapper.cpp. Then, on line 7 you see some more build magic happening. This line specifies the name of the output file. Python has some particular ideas on module naming, which include the Python version, the machine architecture, and other details. Python also provides a tool to help with this called python3.7-config:

$ python3.7-config --extension-suffix
.cpython-37m-x86_64-linux-gnu.so

You may need to modify the command if you’re using a different version of Python. Your results will likely change if you’re using a different version of Python or are on a different operating system.

The final line of your build command, line 8, points the linker at the libcppmult library you built earlier. The rpath section tells the linker to add information to the shared library to help the operating system find libcppmult at runtime. Finally, you’ll notice that this string is formatted with the cpp_name and the extension_name. You’ll be using this function again when you build your Python bindings module with Cython in the next section.

Run this command to build your bindings:

$ invoke build-pybind11
==================================================
= Building C++ Library
* Complete
==================================================
= Building PyBind11 Module
* Complete

That’s it! You’ve built your Python bindings with PyBind11. It’s time to test it out!

Calling Your Function

Similar to the CFFI example above, once you’ve done the heavy lifting of creating the Python bindings, calling your function looks like normal Python code:

# pybind11_test.py
import pybind11_example

if __name__ == "__main__":
    # Sample data for your call
    x, y = 6, 2.3

    answer = pybind11_example.cpp_function(x, y)
    print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Since you used pybind11_example as the name of your module in the PYBIND11_MODULE macro, that’s the name you import. In the m.def() call you told PyBind11 to export the cppmult function as cpp_function, so that’s what you use to call it from Python.

You can test it with invoke as well:

$ invoke test-pybind11
==================================================
= Testing PyBind11 Module
    In cppmul: int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

That’s what PyBind11 looks like. Next, you’ll see when and why PyBind11 is the right tool for the job.

Strengths and Weaknesses

PyBind11 is focused on C++ instead of C, which makes it different from ctypes and CFFI. It has several features that make it quite attractive for C++ libraries:

It supports classes.
It handles polymorphic subclassing.
It allows you to add dynamic attributes to objects from Python and many other tools, which would be quite difficult to do from the C-based tools you’ve examined.

That being said, there’s a fair bit of setup and configuration you need to do to get PyBind11 up and running. Getting the installation and build correct can be a bit finicky, but once that’s done, it seems fairly solid. Also, PyBind11 requires that you use at least C++11 or newer. This is unlikely to be a big restriction for most projects, but it may be a consideration for you.

Finally, the extra code you need to write to create the Python bindings is in C++ and not Python. This may or may not be an issue for you, but it is different than the other tools you’ve looked at here. In the next section, you’ll move on to Cython, which takes quite a different approach to this problem.

`Cython`

The approach Cython takes to creating Python bindings uses a Python-like language to define the bindings and then generates C or C++ code that can be compiled into the module. There are several methods for building Python bindings with Cython. The most common one is to use setup from distutils. For this example, you’ll stick with the invoke tool, which will allow you to play with the exact commands that are run.

How It’s Installed

Cython is a Python module that can be installed into your virtual environment from PyPI:

$ python3 -m pip install cython

Again, if you’ve installed the requirements.txt file into your virtual environment, then this will already be there. You can grab a copy of requirements.txt by clicking on the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about Python Bindings in this tutorial.

That should have you ready to work with Cython!

Calling the Function

To build your Python bindings with Cython, you’ll follow similar steps to those you used for CFFI and PyBind11. You’ll write the bindings, build them, and then run Python code to call them. Cython can support both C and C++. For this example, you’ll use the cppmult library that you used for the PyBind11 example above.

Write the Bindings

The most common form of declaring a module in Cython is to use a .pyx file:

 1 # cython_example.pyx
 2 """ Example cython interface definition """
 3 
 4 cdef extern from "cppmult.hpp":
 5     float cppmult(int int_param, float float_param)
 6 
 7 def pymult( int_param, float_param ):
 8     return cppmult( int_param, float_param )

There are two sections here:

Lines 3 and 4 tell Cython that you’re using cppmult() from cppmult.hpp.
Lines 6 and 7 create a wrapper function, pymult(), to call cppmult().

The language used here is a special mix of C, C++, and Python. It will look fairly familiar to Python developers, though, as the goal is to make the process easier.

The first section with cdef extern... tells Cython that the function declarations below are also found in the cppmult.hpp file. This is useful for ensuring that your Python bindings are built against the same declarations as your C++ code. The second section looks like a regular Python function—because it is! This section creates a Python function that has access to the C++ function cppmult.

Now that you’ve got the Python bindings defined, it’s time to build them!

Build the Python Bindings

The build process for Cython has similarities to the one you used for PyBind11. You first run Cython on the .pyx file to generate a .cpp file. Once you’ve done this, you compile it with the same function you used for PyBind11:

 1 # tasks.py
 2 def compile_python_module(cpp_name, extension_name):
 3     invoke.run(
 4         "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
 5         "`python3 -m pybind11 --includes` "
 6         "-I /usr/include/python3.7 -I .  "
 7         "{0} "
 8         "-o {1}`python3.7-config --extension-suffix` "
 9         "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
10     )
11 
12 def build_cython(c):
13     """ Build the cython extension module """
14     print_banner("Building Cython Module")
15     # Run cython on the pyx file to create a .cpp file
16     invoke.run("cython --cplus -3 cython_example.pyx -o cython_wrapper.cpp")
17 
18     # Compile and link the cython wrapper library
19     compile_python_module("cython_wrapper.cpp", "cython_example")
20     print("* Complete")

You start by running cython on your .pyx file. There are a few options you use on this command:

--cplus tells the compiler to generate a C++ file instead of a C file.
-3 switches Cython to generate Python 3 syntax instead of Python 2.
-o cython_wrapper.cpp specifies the name of the file to generate.

Once the C++ file is generated, you use the C++ compiler to generate the Python bindings, just as you did for PyBind11. Note that the call to produce the extra include paths using the pybind11 tool is still in that function. It won’t hurt anything here, as your source will not need those.

Running this task in invoke produces this output:

$ invoke build-cython
==================================================
= Building C++ Library
* Complete
==================================================
= Building Cython Module
* Complete

You can see that it builds the cppmult library and then builds the cython module to wrap it. Now you have the Cython Python bindings. (Try saying that quickly…) It’s time to test it out!

Calling Your Function

The Python code to call your new Python bindings is quite similar to what you used to test the other modules:

 1 # cython_test.py
 2 import cython_example
 3 
 4 # Sample data for your call
 5 x, y = 6, 2.3
 6 
 7 answer = cython_example.pymult(x, y)
 8 print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Line 2 imports your new Python bindings module, and you call pymult() on line 7. Remember that the .pyx file provided a Python wrapper around cppmult() and renamed it to pymult. Using invoke to run your test produces the following:

$ invoke test-cython
==================================================
= Testing Cython Module
    In cppmul: int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

You get the same result as before!

Strengths and Weaknesses

Cython is a relatively complex tool that can provide you a deep level of control when creating Python bindings for either C or C++. Though you didn’t cover it in depth here, it provides a Python-esque method for writing code that manually controls the GIL, which can significantly speed up certain types of problems.

That Python-esque language is not quite Python, however, so there’s a slight learning curve when you’re coming up to speed in figuring out which parts of C and Python fit where.

Conclusion

Congrats! You’ve now had an overview of several different options for creating Python bindings. You’ve learned about marshalling data and issues you need to consider when creating bindings. You’ve seen what it takes to be able to call a C or C++ function from Python using the following tools:

ctypes
CFFI
PyBind11
Cython

You now know that, while ctypes allow you to load a DLL or shared library directly, the other three tools take an extra step, but still create a full Python module. As a bonus, you’ve also played a little with the invoke tool to run command-line tasks from Python. You can get all of the code you saw in this tutorial by clicking the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about Python Bindings in this tutorial.

Now pick your favorite tool and start building those Python bindings! Special thanks to Loic Domaigne for the extra technical review of this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Beginner's Guide to Python Turtle

2020-02-26T14:00:00+00:00

When I was a kid, I used to learn Logo, a programming language that involved a turtle that you could move around the screen with just a few commands. I remember feeling like a computer genius as I controlled this little object on my screen, and this was what got me interested in programming in the first place. The Python turtle library comes with a similar interactive feature that gives new programmers a taste of what it’s like to work with Python.

In this tutorial, you will:

Understand what the Python turtle library is
Learn how to set turtle up on your computer
Program with the Python turtle library
Grasp some important Python concepts and turtle commands
Develop a short but entertaining game using what you’ve learned

If you’re a beginner to Python, then this tutorial will help you as you take your first steps into the world of programming with the help of the Python turtle library!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Getting to Know the Python `turtle` Library

turtle is a pre-installed Python library that enables users to create pictures and shapes by providing them with a virtual canvas. The onscreen pen that you use for drawing is called the turtle and this is what gives the library its name. In short, the Python turtle library helps new programmers get a feel for what programming with Python is like in a fun and interactive way.

turtle is mainly used to introduce children to the world of computers. It’s a straightforward yet versatile way to understand the concepts of Python. This makes it a great avenue for kids to take their first steps in Python programming. That being said, the Python turtle library is not restricted to little ones alone! It’s also proved extremely useful for adults who are trying their hands at Python, which makes it great for Python beginners.

With the Python turtle library, you can draw and create various types of shapes and images. Here’s a sample of the kinds of drawings you can make with turtle:

Cool, right? This is just one of many different drawings you can make using the Python turtle library. Most developers use turtle to draw shapes, create designs, and make images. Others use turtle to create mini-games and animations, just like the one you saw above.

Getting Started With `turtle`

Before you continue, there are two important things that you’ll need to do to make the most of this tutorial:

Python Environment: Make sure that you’re familiar with your programming environment. You can use applications like IDLE or Jupyter Notebook to program with turtle. However, if you’re not comfortable with them, then you can program with the REPL, which you’ll use in this tutorial.
Python Version: Ensure that you have version 3 of Python on your computer. If not, then you can download it from the Python website. For help setting things up, check out Python 3 Installation & Setup Guide.

The good thing about turtle is that it’s a built-in library, so you don’t need to install any new packages. All you need to do is import the library into your Python environment, which in this case would be the REPL. Once you open your REPL application, you can run Python 3 on it by typing the following line of code:

>>>

>>> python3

This calls Python 3 into your REPL application and opens up the environment for you.

Before you begin your Python programming, you need to understand what a library is. In the non-computer world, a library is a place where different types of books are stored. You can access these books at any time, take whatever information you need from them, and return them to the same place.

In the computer world, a library works similarly. By definition, a library is a set of important functions and methods that you can access to make your programming easier. The Python turtle library contains all the methods and functions that you’ll need to create your images. To access a Python library, you need to import it into your Python environment, like this:

>>>

>>> import turtle

Now that you have turtle in your Python environment, you can begin programming with it. turtle is a graphical library, which means you’ll need to create a separate window (called the screen) to carry out each drawing command. You can create this screen by initializing a variable for it.

In Python, you use variables to store information that you’ll use later on in your program. You initialize a variable when you assign a starting value to it. Since the value of the variable isn’t constant, it can change several times during the execution of your program.

Now, to open the turtle screen, you initialize a variable for it in the following way:

>>>

>>> s = turtle.getscreen()

You should see a separate window open up:

This window is called the screen. It’s where you can view the output of your code. The little black triangular shape in the middle of the screen is called the turtle.

Note: Keep in mind that when you name a variable, you need to pick a name that can be easily understood by anyone who’s looking at your program. However, you must also choose a name that’s convenient for you to use, especially because you’ll be calling it very often throughout your program!

For example, choosing a name like my_turtle_screen_name would be tedious to keep typing, while a name like Joe or a would appear to be very random. Using a single alphabet character, like s in this case, would be much more suitable. That’s because it’s short and sweet, and it’s clear to remember that the letter s refers to the screen.

Next, you initialize the variable t, which you’ll then use throughout the program to refer to the turtle:

>>>

>>> t = turtle.Turtle()

Just like for the screen, you can also give this variable another name like a or Jane or even my_turtle, but in this case, you’ll stick with t.

You now have your screen and your turtle. The screen acts as a canvas, while the turtle acts like a pen. You can program the turtle to move around the screen. The turtle has certain changeable characteristics, like size, color, and speed. It always points in a specific direction, and will move in that direction unless you tell it otherwise:

When it’s up, it means that no line will be drawn when it moves.
When it’s down, it means that a line will be drawn when it moves.

In the next section, you’ll explore the different ways of programming with the Python turtle library.

Programming With `turtle`

The first thing you’ll learn when it comes to programming with the Python turtle library is how to make the turtle move in the direction you want it to go. Next, you’ll learn how to customize your turtle and its environment. Finally, you’ll learn a couple of extra commands with which you can perform some special tasks.

Moving the Turtle

There are four directions that a turtle can move in:

Forward
Backward
Left
Right

The turtle moves .forward() or .backward() in the direction that it’s facing. You can change this direction by turning it .left() or .right() by a certain degree. You can try each of these commands like so:

>>>

>>> t.right(90)
>>> t.forward(100)
>>> t.left(90)
>>> t.backward(100)

When you run these commands, the turtle will turn right by ninety degrees, go forward by a hundred units, turn left by ninety degrees, and move backward by a hundred units. You can see how this looks in the image below:

You can use the shortened versions of these commands as well:

t.rt() instead of t.right()
t.fd() instead of t.forward()
t.lt() instead of t.left()
t.bk() instead of t.backward()

You can also draw a line from your current position to any other arbitrary position on the screen. This is done with the help of coordinates:

The screen is divided into four quadrants. The point where the turtle is initially positioned at the beginning of your program is (0,0). This is called Home. To move the turtle to any other area on the screen, you use .goto() and enter the coordinates like this:

>>>

>>> t.goto(100,100)

Your output will look like this:

You’ve drawn a line from your current position to the point (100,100) on the screen.

To bring the turtle back to its home position, you type the following:

>>>

>>> t.home()

This is like a shortcut command that sends the turtle back to the point (0,0). It’s quicker than typing t.goto(0,0).

Drawing a Shape

Now that you know the movements of the turtle, you can move on to making actual shapes. You can start by drawing polygons since they all consist of straight lines connected at certain angles. Here’s an example that you can try:

>>>

>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)

Your output will look like this:

Well done! You’ve just drawn a square. In this way, the turtle can be programmed to create different shapes and images.

Now, try drawing a rectangle, using this code as a template. Remember, in a rectangle, all four sides are not equal. You’ll need to change the code accordingly. Once you do that, you can even try creating other polygons by increasing the number of sides and changing the angles.

Drawing Preset Figures

Suppose you want to draw a circle. If you attempt to draw it in the same way as you drew the square, then it would be extremely tedious, and you’d have to spend a lot of time just for that one shape. Thankfully, the Python turtle library provides a solution for this. You can use a single command to draw a circle:

>>>

>>> t.circle(60)

You’ll get an output like this:

The number within the brackets is the radius of the circle. You can increase or decrease the size of the circle by changing the value of its radius.

In the same way, you can also draw a dot, which is nothing but a filled-in circle. Type in this command:

>>>

>>> t.dot(20)

You’ll get a filled-in circle like this:

The number within the brackets is the diameter of the dot. Just like with the circle, you can increase or decrease the size of the dot by changing the value of its diameter.

Great job so far! You’ve learned how to move the turtle around and create different shapes with it. In the next few sections, you’ll see how you can customize your turtle and its environment, based on your requirements.

Changing the Screen Color

By default, turtle always opens up a screen with a white background. However, you can change the color of the screen at any time using the following command:

>>>

>>> turtle.bgcolor("blue")

You can replace "blue" with any other color. Try "green" or "red". You’ll get a result like this:

You can use a variety of colors for your screen just by typing in their hex code number. To learn more about using different colors, check out the Python turtle library documentation.

Changing the Screen Title

Sometimes, you may want to change the title of your screen. You can make it more personal, like "My Turtle Program", or more suitable to what you’re working on, like "Drawing Shapes With Turtle". You can change the title of your screen with the help of this command:

>>>

>>> turtle.title("My Turtle Program")

Your title bar will now display this:

In this way, you can change the heading of your screen according to your preference.

Changing the Turtle Size

You can increase or decrease the size of the onscreen turtle to make it bigger or smaller. This changes only the size of the shape without affecting the output of the pen as it draws on the screen. Try typing in the following commands:

>>>

>>> t.shapesize(1,5,10)
>>> t.shapesize(10,5,1)
>>> t.shapesize(1,10,5)
>>> t.shapesize(10,1,5)

Your outputs will look like this:

The numbers given are the parameters for the size of the turtle:

Stretch length
Stretch width
Outline width

You can change these according to your preference. In the example given above, you can see a visible difference in the appearance of the turtle. For more information on how you can change the size of the turtle, check out the Python turtle library documentation.

Changing the Pen Size

The previous command changed the size of the turtle’s shape only. However, sometimes, you may need to increase or decrease the thickness of your pen. You can do this using the following command:

>>>

>>> t.pensize(5)
>>> t.forward(100)

This results in an outcome like this:

As you can see, the size of your pen is now five times the original size (which was one). Try drawing some more lines of various sizes, and compare the difference in thickness between them.

Changing the Turtle and Pen Color

When you first open a new screen, the turtle starts out as a black figure and draws with black ink. Based on your requirements, you can do two things:

Change the color of the turtle: This changes the fill color.
Change the color of the pen: This changes the outline or the ink color.

You can even choose both of these if you wish. Before you change the colors, increase the size of your turtle to help you see the color difference more clearly. Type in this code:

>>>

>>> t.shapesize(3,3,3)

Now, to change the color of the turtle (or the fill), you type the following:

>>>

>>> t.fillcolor("red")

Your turtle will look like this:

To change the color of the pen (or the outline), you type the following:

>>>

>>> t.pencolor("green")

Your turtle will look like this:

To change the color of both, you type the following:

>>>

>>> t.color("green", "red")

Your turtle will look like this:

Here, the first color is for the pen, and the second is for the fill. Note that changing the color of the pen and the fill also changes the color of the onscreen turtle accordingly.

Filling in an Image

Coloring in an image usually makes it look better, doesn’t it? The Python turtle library gives you the option to add color to your drawings. Try typing in the following code and see what happens:

>>>

>>> t.begin_fill()
>>> t.fd(100)
>>> t.lt(120)
>>> t.fd(100)
>>> t.lt(120)
>>> t.fd(100)
>>> t.end_fill()

When you execute this code, you’ll get a triangle that’s filled in with a solid color, like this:

When you use .beginfill(), you’re telling your program that you’re going to be drawing a closed shape which will need to be filled in. Then, you use .endfill() to indicate that you’re done creating your shape and it can now be filled in.

Changing the Turtle Shape

The initial shape of the turtle isn’t really a turtle, but a triangular figure. However, you can change the way the turtle looks, and you do have a couple of options when it comes to doing so. You can have a look at some of them by typing in the following commands:

>>>

>>> t.shape("turtle")
>>> t.shape("arrow")
>>> t.shape("circle")

The shape of the turtle will change accordingly, like this:

You have a couple of other options that you can try as well:

Square
Arrow
Circle
Turtle
Triangle
Classic

The classic shape is the original shape. Check out the Python turtle library documentation to learn more about the types of shapes that you can use.

Changing the Pen Speed

The turtle generally moves at a moderate pace. If you want to decrease or increase the speed to make your turtle move slower or faster, then you can do so by typing the following:

>>>

>>> t.speed(1)
>>> t.forward(100)
>>> t.speed(10)
>>> t.forward(100)

This code will first decrease the speed and move the turtle forward, then increase the speed and move the turtle forward again, like this:

The speed can be any number ranging from 0 (the slowest speed) to 10 (the highest speed). You can play around with your code to see how fast or slow the turtle will go.

Customizing in One Line

Suppose you want to set your turtle’s characteristics to the following:

Pen color: purple
Fill color: orange
Pen size: 10
Pen speed: 9

From what you’ve just learned, the code should look something like this:

>>>

>>> t.pencolor("purple")
>>> t.fillcolor("orange")
>>> t.pensize(10)
>>> t.speed(9)
>>> t.begin_fill()
>>> t.circle(90)
>>> t.end_fill()

It’s pretty long, but not that bad, right?

Now, just imagine if you had ten different turtles. Changing all of their characteristics would be extremely tiresome for you to do! The good news is that you can reduce your workload by altering the parameters in just a single line of code, like this:

>>>

>>> t.pen(pencolor="purple", fillcolor="orange", pensize=10, speed=9)
>>> t.begin_fill()
>>> t.circle(90)
>>> t.end_fill()

This will give you a result like this:

This single line of code changed the entire pen, without you having to change each characteristic individually. To learn more about this command, check out the Python turtle library documentation.

Great job! Now that you’ve learned to customize your turtle and the screen, take a look at some other important commands that are required while drawing with the Python turtle library.

Picking the Pen Up and Down

Sometimes, you may want to move your turtle to another point on the screen without drawing anything on the screen itself. To do this, you use .penup(). Then, when you want to start drawing again, you use .pendown(). Give it a shot using the code that you used previously to draw a square. Try typing the following code:

>>>

>>> t.fd(100)
>>> t.rt(90)
>>> t.penup()
>>> t.fd(100)
>>> t.rt(90)
>>> t.pendown()
>>> t.fd(100)
>>> t.rt(90)
>>> t.penup()
>>> t.fd(100)
>>> t.pendown()

When you run this code, your output will look like this:

Here, you’ve obtained two parallel lines instead of a square by adding some extra commands in between the original program.

Undoing Changes

No matter how careful you are, there’s always a possibility of making a mistake. Don’t worry, though! The Python turtle library gives you the option to undo what you’ve done. If you want to undo the very last thing you did, then type in the following:

>>>

>>> t.undo()

This undoes the last command that you ran. If you want to undo your last three commands, then you would type t.undo() three times.

Clearing the Screen

Right now, you probably have a lot on your screen since you’ve started this tutorial. To make room for more, just type in the following command:

>>>

>>> t.clear()

This will clean up your screen so that you can continue drawing. Note here that your variables will not change, and the turtle will remain in the same position. If you have other turtles on your screen other than the original turtle, then their drawings will not be cleared out unless you specifically call them out in your code.

Resetting the Environment

You also have the option to start on a clean slate with a reset command. The screen will get cleared up, and the turtle’s settings will all be restored to their default parameters. All you need to to do is type in the following command:

>>>

>>> t.reset()

This clears the screen and takes the turtle back to its home position. Your default settings, like the turtle’s size, shape, color, and other features, will also be restored.

Now that you’ve learned the fundamentals of programming with the Python turtle library, you’ll check out some bonus features that you may want to use while programming.

Leaving a Stamp

You have the option of leaving a stamp of your turtle on the screen, which is nothing but an imprint of the turtle. Try typing in this code to see how it works:

>>>

>>> t.stamp()
8
>>> t.fd(100)
>>> t.stamp()
9
>>> t.fd(100)

Your output will look like this:

The numbers that appear are the turtle’s location or stamp ID. Now, if you want to remove a particular stamp, then just use the following:

>>>

>>> t.clearstamp(8)

This will clear the one with the stamp ID of 8.

Cloning Your Turtle

Sometimes, you may need to have more than one turtle on your screen. You’ll see an example of this later on in the final project. For now, you can get another turtle by cloning your current turtle into your environment. Try running this code to create a clone turtle, c, and then move both the turtles on the screen:

>>>

>>> c = t.clone()
>>> t.color("magenta")
>>> c.color("red")
>>> t.circle(100)
>>> c.circle(60)

The output will look like this:

Awesome!

Now that you have an idea of some important commands from the Python turtle library, you’re ready to move on to a few more concepts that you’ll need to understand. These concepts are very much needed when it comes to programming in any language.

Using Loops and Conditional Statements

When you get into higher-level programming, you’ll find yourself using loops and conditional statements very often. That’s why, in this section, you’ll be going through a couple of turtle programs that make use of these types of commands. This will give you a practical approach when it comes to understanding these concepts. Before you begin, however, here are three definitions for you to keep in mind:

Loops are a set of instructions that are continuously repeated until a particular condition is satisfied.
Conditional statements carry out a certain task based on a condition that’s satisfied.
Indentations are used to define blocks of code, especially when using loops and conditional statements. In general, you create an indentation by tapping the Tab key on the keyboard.

Now, let’s go ahead and explore these commands!

`for` Loops

Do you remember the program that you used to create a square? You had to repeat the same line of code four times, like this:

>>>

>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)
>>> t.rt(90)
>>> t.fd(100)
>>> t.rt(90)

A much shorter way to do this is with the help of a for loop. Try running this code:

>>>

>>> for i in range(4):
...     t.fd(100)
...     t.rt(90)

Here, the i is like a counter that starts from zero and keeps increasing by 1. When you say in range(4), you’re telling the program that the value of this i should be less than 4. It will terminate the program before i reaches 4.

Here’s a breakdown of how the program works:

At i = 0, the turtle moves forward by 100 units and then turns 90 degrees to the right.
At i = 0 + 1 = 1, the turtle moves forward by 100 units and then turns 90 degrees to the right.
At i = 1 + 1 = 2, the turtle moves forward by 100 units and then turns 90 degrees to the right.
At i = 2 + 1 = 3, the turtle moves forward by 100 units and then turns 90 degrees to the right.

The turtle will then exit the loop. To check the value of i, type i and then press the Enter key. You’ll get the value of i equal to 3:

>>>

>>> i
3

Note that the whitespace that comes before line 2 and line 3 in the program is the indentation. This indicates that all 3 lines form a single block of code. To learn more about for loops in Python, check out Python “for” Loops (Definite Iteration).

`while` Loops

The while loop is used to perform a certain task while a condition is still satisfied. If the condition is no longer satisfied, then your code will terminate the process. You can use a while loop to create a series of circles by typing in this code:

>>>

>>> n=10
>>> while n <= 40:
...     t.circle(n)
...     n = n+10

When you run this code, you’ll see the circles appearing one after the other, and each new circle will be larger than the previous one:

Here, n is used as a counter. You’ll need to specify by how much you want the value of n to increase in each loop. Take a look at this mini walk-through to see how the program works:

At n = 10, the turtle draws a circle with a radius of 10 units. After that, the value of n is increased by 10.
At n = 20, the turtle draws a circle with a radius of 20 units. Once again, the value of n is increased by 10.
At n = 30, the turtle draws a circle with a radius of 30 units. For the third time, the value of n is increased by 10.
At n = 40, the turtle draws a circle with a radius of 40 units. For the last time, the value of n is increased by 10.
At n = 50, n is no longer less than or equal to 40. The loop is terminated.

To read more about while loops, check out Python “while” Loops (Indefinite Iteration).

Conditional Statements

You use conditional statements to check if a given condition is true. If it is, then the corresponding command is executed. Try typing in this program:

>>>

>>> u = input("Would you like me to draw a shape? Type yes or no: ")
>>> if u == "yes":
...     t.circle(50)

input() is used to obtain input from the user. Here, it will store the user’s response under the variable u. Next, it will compare the value of u with the condition provided and check whether the value of u is "yes". If it’s "yes", then your program draws a circle. If the user types in anything else, then the program won’t do anything.

Note: The comparison operator == indicates a comparison. It’s used to check if the value of something is equal to something else. The assignment operator = is used to assign a value to something. To learn more about the differences between the two, check out Operators and Expressions in Python.

When you add an else clause to an if statement, you can specify two results based on whether the condition is true or false. Let’s see this in a program:

>>>

>>> u = input("Would you like me to draw a shape? Type yes or no: ")
>>> if u == "yes":
...     t.circle(50)
>>> else:
...     print("Okay")

Here, you tell the program to display a particular output even when the user does not say "yes". You use print() to display some pre-defined characters on the screen.

Note that the user doesn’t need to type "no". They can type anything else, in which case, the result will always be "Okay", because you’re not explicitly telling the program that the user needs to type "no". Not to worry, however, as that can be fixed. You can add an elif clause to provide the program with several conditions and their respective outputs, as you can observe here:

>>>

>>> u = input("Would you like me to draw a shape? Type yes or no: ")
>>> if u == "yes":
...     t.circle(50)
>>> elif u == "no":
...     print("Okay")
>>> else:
...     print("Invalid Reply")

As you can see, this program now has more than one outcome, depending on the input it receives. Here’s how this code works:

If you type in "yes", then the code processes the input and draws a circle, as per your instructions.
If you type in "no", then the code prints out "Okay" and your program is terminated.
If you type in anything else, like "Hello" or "Sandwich", then the code prints "Invalid Reply" and your program is terminated.

Note that this program is case-sensitive, so when you’re trying it out, be sure to put the strings in upper-case or lower-case accordingly.

To learn more about conditional statements, check out Conditional Statements in Python.

Final Project: The Python Turtle Race

So far, you’ve learned how to customize your turtle environment, program your turtle to move around the screen, and use loops and conditional statements to improve your code. Now it’s time for the most important part of your programming journey. In this section, you’ll be implementing all that you’ve learned into a single program by creating a fun game that you can play with your friends.

Before you begin, here’s what you need to know about the game:

The Objective: The player whose turtle reaches its home first wins the game.
How to Play:
- Each player rolls a dice to get a number.
- The player then moves their turtle by that many steps.
- The players alternate turns until one of them wins.
The Structure:
- Each player had a turtle indicated by a different color. You can have more than two players, but for the sake of this tutorial, you’ll be creating a two-player game.
- Each turtle has a home position that it must reach.
- Each player uses a die to choose a value at random for their turn. In your program, the die is represented by a list of numbers from 1 to 6.

Now that you’ve understood the logic of the game, you can go ahead and begin creating it! First, you’ll need to set up the environment.

Setting Up the Game Environment

Start by importing the Python turtle library. After this, import the built-in random library, which you’ll use randomly select an item from a list:

>>>

>>> import turtle
>>> import random

Once these libraries are successfully called into your environment, you can proceed with the rest of your program.

Setting Up the Turtles and Homes

You now have to create the two turtles that will represent the players. Each turtle will be a different color, corresponding to the different players. Here, player one is green and player two is blue:

>>>

>>> player_one = turtle.Turtle()
>>> player_one.color("green")
>>> player_one.shape("turtle")
>>> player_one.penup()
>>> player_one.goto(-200,100)
>>> player_two = player_one.clone()
>>> player_two.color("blue")
>>> player_two.penup()
>>> player_two.goto(-200,-100)

One you’ve created the turtles, you place them at their starting positions and make sure that these positions are aligned. Note that you created player two’s turtle by cloning player one’s turtle, changing its color, and placing it at a different starting point.

You now need to set up homes for the turtles. These homes will act as the finishing points for each turtle. Each of the turtles’ homes will be represented by a circle. Here, you need to make sure that both homes are equidistant from the starting point:

>>>

>>> player_one.goto(300,60)
>>> player_one.pendown()
>>> player_one.circle(40)
>>> player_one.penup()
>>> player_one.goto(-200,100)
>>> player_two.goto(300,-140)
>>> player_two.pendown()
>>> player_two.circle(40)
>>> player_two.penup()
>>> player_two.goto(-200,-100)

After drawing the respective homes, you send the turtles back to their starting positions:

Awesome! The visual aspects of your game are complete. You can now create the die that you’ll be using to play the game.

Creating the Die

You can create a virtual die for your game with a list, which is an ordered sequence of items. In real life, you might prepare grocery lists and to-do lists to help you stay organized. In Python, lists work in a similar way.

In this case, you’ll be using a list to create your die. First, you define your list of numbers in ascending order from 1 to 6. You can define a list by giving it a name and then enclosing its items within square brackets, like this:

>>>

>>> die = [1,2,3,4,5,6]

This list has now become your die. To roll the dice, all you have to do is program your system to randomly select a number from it. The number that is selected will be considered as the output of the die.

Developing the Game

It’s time to develop the code for the rest of the game. You’ll be using loops and conditional statements here, so you need to be careful with the indentations and spaces. To start, take a look at the steps your program will need to take to run the game:

Step 1: You’ll start by telling your program to check if either turtle has reached its home.
Step 2: If they haven’t, then you’ll tell your program to allow the players to continue trying.
Step 3: In each loop, you tell your program to roll the die by randomly picking a number from the list.
Step 4: You then tell it to move the respective turtle accordingly, with the number of steps based on the outcome of this random selection.

The program keeps repeating this process, and stops once one of the turtles reaches the goal. Here’s how the code looks:

>>>

 1 >>> for i in range(20):
 2 ...     if player_one.pos() >= (300,100):
 3 ...             print("Player One Wins!")
 4 ...             break
 5 ...     elif player_two.pos() >= (300,-100):
 6 ...             print("Player Two Wins!")
 7 ...             break
 8 ...     else:
 9 ...             player_one_turn = input("Press 'Enter' to roll the die ")
10 ...             die_outcome = random.choice(die)
11 ...             print("The result of the die roll is: ")
12 ...             print(die_outcome)
13 ...             print("The number of steps will be: ")
14 ...             print(20*die_outcome)
15 ...             player_one.fd(20*die_outcome)
16 ...             player_two_turn = input("Press 'Enter' to roll the die ")
17 ...             d = random.choice(die)
18 ...             print("The result of the die roll is: ")
19 ...             print(die_outcome)
20 ...             print("The number of steps will be: ")
21 ...             print(20*die_outcome)
22 ...             player_two.fd(20*die_outcome)

Your final output will look a little something like this:

In summary, this is what the code is doing:

Line 1 sets up a for loop with a range from 1 to 20.
Lines 2 through 7 check if either player has reached their goal. If one of them has, then the program prints out the corresponding statement and breaks the loop.
Line 8 moves the program on to the next set of steps if neither player has won.
Line 9 prints out a statement asking player one to press the Enter key to roll the die.
Line 10 takes a random value from the list die and stores it in dice_outcome.
Line 11 prints a statement prior to displaying the outcome of the dice roll.
Line 12 prints the dice outcome.
Line 14 multiplies this value by 20 to reduce the overall number of steps required to complete the game.
Line 15 moves player one’s turtle forward by this number of steps.
Lines 16 to 22 repeat these steps for player two.

The entire for loop is repeated until one of the player’s turtles reaches the final position.

Note: In Python, you use the asterisk (*) to indicate multiplication. This is known as an arithmetic operator. You can also use the plus sign (+) for addition, the minus sign (-) for subtraction, and a slash (/) for division. To learn more about arithmetic operators, check out the Arithmetic Operators section of Operators and Expressions in Python.

Remember, you can customize the game however you want, so go ahead and play around with it! You can add more turtles, change the colors, change the speed, or even create some obstacles to challenge your players. It’s all up to you as the developer of the game!

Conclusion

In this tutorial, you’ve learned how to program with the Python turtle library and grasped some very important programming concepts. You know how to deal with variable initialization, loops, conditional statements, indentations, lists, and operators. This is a great start for you, especially if you’re new to the Python programming language!

Now you can:

Set up the Python turtle library
Move your turtle around
Customize your turtle and its environment
Program your turtle
Use basic programming concepts
Create a game that you can play with friends

Now you’re ready to venture into some higher-level Python programming. To progress further in your Python journey, check out Introduction to Python and 11 Beginner Tips for Learning Python Programming. Just remember to work hard and keep practicing, and you’ll find that you’re a Python expert in no time!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Work With a PDF in Python

2020-02-25T14:00:00+00:00

The Portable Document Format or PDF is a file format that can be used to present and exchange documents reliably across operating systems. While the PDF was originally invented by Adobe, it is now an open standard that is maintained by the International Organization for Standardization (ISO). You can work with a preexisting PDF in Python by using the PyPDF2 package.

PyPDF2 is a pure-Python package that you can use for many different types of PDF operations.

By the end of this course, you’ll know how to:

Extract document information from a PDF in Python
Rotate pages
Merge PDFs
Split PDFs
Add watermarks
Encrypt a PDF

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Python SQL Libraries

2020-02-24T14:00:00+00:00

All software applications interact with data, most commonly through a database management system (DBMS). Some programming languages come with modules that you can use to interact with a DBMS, while others require the use of third-party packages. In this tutorial, you’ll explore the different Python SQL libraries that you can use. You’ll develop a straightforward application to interact with SQLite, MySQL, and PostgreSQL databases.

In this tutorial, you’ll learn how to:

Connect to different database management systems with Python SQL libraries
Interact with SQLite, MySQL, and PostgreSQL databases
Perform common database queries using a Python application
Develop applications across different databases using a Python script

To get the most out of this tutorial, you should have knowledge of basic Python, SQL, and working with database management systems. You should also be able to download and import packages in Python and know how to install and run different database servers locally or remotely.

Free PDF Download: Python 3 Cheat Sheet

Understanding the Database Schema

In this tutorial, you’ll develop a very small database for a social media application. The database will consist of four tables:

users
posts
comments
likes

A high-level diagram of the database schema is shown below:

Both users and posts will have a one-to-many relationship since one user can like many posts. Similarly, one user can post many comments, and one post can also have multiple comments. So, both users and posts will also have one-to-many relationships with the comments table. This also applies to the likes table, so both users and posts will have a one-to-many relationship with the likes table.

Using Python SQL Libraries to Connect to a Database

Before you interact with any database through a Python SQL Library, you have to connect to that database. In this section, you’ll see how to connect to SQLite, MySQL, and PostgreSQL databases from within a Python application.

Note: You’ll need MySQL and PostgreSQL servers up and running before you execute the scripts in the MySQL and PostgreSQL database sections. For a quick intro on how to start a MySQL server, check out the MySQL section of Starting a Django Project. To learn how to create a database in PostgreSQL, check out the Setting Up a Database section of Preventing SQL Injection Attacks With Python.

It’s recommended that you create three different Python files, so you have one for each of the three databases. You’ll execute the script for each database in its corresponding file.

SQLite

SQLite is probably the most straightforward database to connect to with a Python application since you don’t need to install any external Python SQL modules to do so. By default, your Python installation contains a Python SQL library named sqlite3 that you can use to interact with an SQLite database.

What’s more, SQLite databases are serverless and self-contained, since they read and write data to a file. This means that, unlike with MySQL and PostgreSQL, you don’t even need to install and run an SQLite server to perform database operations!

Here’s how you use sqlite3 to connect to an SQLite database in Python:

 1 import sqlite3
 2 from sqlite3 import Error
 3 
 4 def create_connection(path):
 5     connection = None
 6     try:
 7         connection = sqlite3.connect(path)
 8         print("Connection to SQLite DB successful")
 9     except Error as e:
10         print(f"The error '{e}' occurred")
11 
12     return connection

Here’s how this code works:

Lines 1 and 2 import sqlite3 and the module’s Error class.
Line 4 defines a function .create_connection() that accepts the path to the SQLite database.
Line 7 uses .connect() from the sqlite3 module and takes the SQLite database path as a parameter. If the database exists at the specified location, then a connection to the database is established. Otherwise, a new database is created at the specified location, and a connection is established.
Line 8 prints the status of the successful database connection.
Line 9 catches any exception that might be thrown if .connect() fails to establish a connection.
Line 10 displays the error message in the console.

sqlite3.connect(path) returns a connection object, which is in turn returned by create_connection(). This connection object can be used to execute queries on an SQLite database. The following script creates a connection to the SQLite database:

connection = create_connection("E:\\sm_app.sqlite")

Once you execute the above script, you’ll see that a database file sm_app.sqlite is created in the root directory. Note that you can change the location to match your setup.

MySQL

Unlike SQLite, there’s no default Python SQL module that you can use to connect to a MySQL database. Instead, you’ll need to install a Python SQL driver for MySQL in order to interact with a MySQL database from within a Python application. One such driver is mysql-connector-python. You can download this Python SQL module with pip:

$ pip install mysql-connector-python

Note that MySQL is a server-based database management system. One MySQL server can have multiple databases. Unlike SQLite, where creating a connection is tantamount to creating a database, a MySQL database has a two-step process for database creation:

Make a connection to a MySQL server.
Execute a separate query to create the database.

Define a function that connects to the MySQL database server and returns the connection object:

 1 import mysql.connector
 2 from mysql.connector import Error
 3 
 4 def create_connection(host_name, user_name, user_password):
 5     connection = None
 6     try:
 7         connection = mysql.connector.connect(
 8             host=host_name,
 9             user=user_name,
10             passwd=user_password
11         )
12         print("Connection to MySQL DB successful")
13     except Error as e:
14         print(f"The error '{e}' occurred")
15 
16     return connection
17 
18 connection = create_connection("localhost", "root", "")

In the above script, you define a function create_connection() that accepts three parameters:

host_name
user_name
user_password

The mysql.connector Python SQL module contains a method .connect() that you use in line 7 to connect to a MySQL database server. Once the connection is established, the connection object is returned to the calling function. Finally, in line 18 you call create_connection() with the host name, username, and password.

So far, you’ve only established the connection. The database is not yet created. To do this, you’ll define another function create_database() that accepts two parameters:

connection is the connection object to the database server that you want to interact with.
query is the query that creates the database.

Here’s what this function looks like:

def create_database(connection, query):
    cursor = connection.cursor()
    try:
        cursor.execute(query)
        print("Database created successfully")
    except Error as e:
        print(f"The error '{e}' occurred")

To execute queries, you use the cursor object. The query to be executed is passed to cursor.execute() in string format.

Create a database named sm_app for your social media app in the MySQL database server:

create_database_query = "CREATE DATABASE sm_app"
create_database(connection, create_database_query)

Now you’ve created a database sm_app on the database server. However, the connection object returned by the create_connection() is connected to the MySQL database server. You need to connect to the sm_app database. To do so, you can modify create_connection() as follows:

 1 def create_connection(host_name, user_name, user_password, db_name):
 2     connection = None
 3     try:
 4         connection = mysql.connector.connect(
 5             host=host_name,
 6             user=user_name,
 7             passwd=user_password,
 8             database=db_name
 9         )
10         print("Connection to MySQL DB successful")
11     except Error as e:
12         print(f"The error '{e}' occurred")
13 
14     return connection

You can see in line 8 that create_connection() now accepts an additional parameter called db_name. This parameter specifies the name of the database that you want to connect to. You can pass in the name of the database you want to connect to when you call this function:

connection = create_connection("localhost", "root", "", "sm_app")

The above script successfully calls create_connection() and connects to the sm_app database.

PostgreSQL

Like MySQL, there’s no default Python SQL library that you can use to interact with a PostgreSQL database. Instead, you need to install a third-party Python SQL driver to interact with PostgreSQL. One such Python SQL driver for PostgreSQL is psycopg2. Execute the following command on your terminal to install the psycopg2 Python SQL module:

$ pip install psycopg2

Like with the SQLite and MySQL databases, you’ll define create_connection() to make a connection with your PostgreSQL database:

import psycopg2
from psycopg2 import OperationalError

def create_connection(db_name, db_user, db_password, db_host, db_port):
    connection = None
    try:
        connection = psycopg2.connect(
            database=db_name,
            user=db_user,
            password=db_password,
            host=db_host,
            port=db_port,
        )
        print("Connection to PostgreSQL DB successful")
    except OperationalError as e:
        print(f"The error '{e}' occurred")
    return connection

You use psycopg2.connect() to connect to a PostgreSQL server from within your Python application.

You can then use create_connection() to create a connection to a PostgreSQL database. First, you’ll make a connection with the default database postgres by using the following string:

connection = create_connection(
    "postgres", "postgres", "abc123", "127.0.0.1", "5432"
)

Next, you have to create the database sm_app inside the default postgres database. You can define a function to execute any SQL query in PostgreSQL. Below, you define create_database() to create a new database in the PostgreSQL database server:

def create_database(connection, query):
    connection.autocommit = True
    cursor = connection.cursor()
    try:
        cursor.execute(query)
        print("Query executed successfully")
    except OperationalError as e:
        print(f"The error '{e}' occurred")

create_database_query = "CREATE DATABASE sm_app"
create_database(connection, create_database_query)

Once you run the script above, you’ll see the sm_app database in your PostgreSQL database server.

Before you execute queries on the sm_app database, you need to connect to it:

connection = create_connection(
    "sm_app", "postgres", "abc123", "127.0.0.1", "5432"
)

Once you execute the above script, a connection will be established with the sm_app database located in the postgres database server. Here, 127.0.0.1 refers to the database server host IP address, and 5432 refers to the port number of the database server.

Creating Tables

In the previous section, you saw how to connect to SQLite, MySQL, and PostgreSQL database servers using different Python SQL libraries. You created the sm_app database on all three database servers. In this section, you’ll see how to create tables inside these three databases.

As discussed earlier, you’ll create four tables:

users
posts
comments
likes

You’ll start with SQLite.

SQLite

To execute queries in SQLite, use cursor.execute(). In this section, you’ll define a function execute_query() that uses this method. Your function will accept the connection object and a query string, which you’ll pass to cursor.execute().

.execute() can execute any query passed to it in the form of string. You’ll use this method to create tables in this section. In the upcoming sections, you’ll use this same method to execute update and delete queries as well.

Note: This script should be executed in the same file where you created the connection for your SQLite database.

Here’s your function definition:

def execute_query(connection, query):
    cursor = connection.cursor()
    try:
        cursor.execute(query)
        connection.commit()
        print("Query executed successfully")
    except Error as e:
        print(f"The error '{e}' occurred")

This code tries to execute the given query and prints an error message if necessary.

Next, write your query:

create_users_table = """
CREATE TABLE IF NOT EXISTS users (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL,
  age INTEGER,
  gender TEXT,
  nationality TEXT
);
"""

This says to create a table users with the following five columns:

id
name
age
gender
nationality

Finally, you’ll call execute_query() to create the table. You’ll pass in the connection object that you created in the previous section, along with the create_users_table string that contains the create table query:

execute_query(connection, create_users_table)

The following query is used to create the posts table:

create_posts_table = """
CREATE TABLE IF NOT EXISTS posts(
  id INTEGER PRIMARY KEY AUTOINCREMENT, 
  title TEXT NOT NULL, 
  description TEXT NOT NULL, 
  user_id INTEGER NOT NULL, 
  FOREIGN KEY (user_id) REFERENCES users (id)
);
"""

Since there’s a one-to-many relationship between users and posts, you can see a foreign key user_id in the posts table that references the id column in the users table. Execute the following script to create the posts table:

execute_query(connection, create_posts_table)

Finally, you can create the comments and likes tables with the following script:

create_comments_table = """
CREATE TABLE IF NOT EXISTS comments (
  id INTEGER PRIMARY KEY AUTOINCREMENT, 
  text TEXT NOT NULL, 
  user_id INTEGER NOT NULL, 
  post_id INTEGER NOT NULL, 
  FOREIGN KEY (user_id) REFERENCES users (id) FOREIGN KEY (post_id) REFERENCES posts (id)
);
"""

create_likes_table = """
CREATE TABLE IF NOT EXISTS likes (
  id INTEGER PRIMARY KEY AUTOINCREMENT, 
  user_id INTEGER NOT NULL, 
  post_id integer NOT NULL, 
  FOREIGN KEY (user_id) REFERENCES users (id) FOREIGN KEY (post_id) REFERENCES posts (id)
);
"""

execute_query(connection, create_comments_table)  
execute_query(connection, create_likes_table)

You can see that creating tables in SQLite is very similar to using raw SQL. All you have to do is store the query in a string variable and then pass that variable to cursor.execute().

MySQL

You’ll use the mysql-connector-python Python SQL module to create tables in MySQL. Just like with SQLite, you need to pass your query to cursor.execute(), which is returned by calling .cursor() on the connection object. You can create another function execute_query() that accepts the connection and query string:

 1 def execute_query(connection, query):
 2     cursor = connection.cursor()
 3     try:
 4         cursor.execute(query)
 5         connection.commit()
 6         print("Query executed successfully")
 7     except Error as e:
 8         print(f"The error '{e}' occurred")

In line 4, you pass the query to cursor.execute().

Now you can create your users table using this function:

create_users_table = """
CREATE TABLE IF NOT EXISTS users (
  id INT AUTO_INCREMENT, 
  name TEXT NOT NULL, 
  age INT, 
  gender TEXT, 
  nationality TEXT, 
  PRIMARY KEY (id)
) ENGINE = InnoDB
"""

execute_query(connection, create_users_table)

The query for implementing the foreign key relation is slightly different in MySQL as compared to SQLite. What’s more, MySQL uses the AUTO_INCREMENT keyword (compared to the SQLite AUTOINCREMENT keyword) to create columns where the values are automatically incremented when new records are inserted.

The following script creates the posts table, which contains a foreign key user_id that references the id column of the users table:

create_posts_table = """
CREATE TABLE IF NOT EXISTS posts (
  id INT AUTO_INCREMENT, 
  title TEXT NOT NULL, 
  description TEXT NOT NULL, 
  user_id INTEGER NOT NULL, 
  FOREIGN KEY fk_user_id (user_id) REFERENCES users(id), 
  PRIMARY KEY (id)
) ENGINE = InnoDB
"""

execute_query(connection, create_posts_table)

Similarly, to create the comments and likes tables, you can pass the corresponding CREATE queries to execute_query().

PostgreSQL

Like with SQLite and MySQL databases, the connection object that’s returned by psycopg2.connect() contains a cursor object. You can use cursor.execute() to execute Python SQL queries on your PostgreSQL database.

Define a function execute_query():

def execute_query(connection, query):
    connection.autocommit = True
    cursor = connection.cursor()
    try:
        cursor.execute(query)
        print("Query executed successfully")
    except OperationalError as e:
        print(f"The error '{e}' occurred")

You can use this function to create tables, insert records, modify records, and delete records in your PostgreSQL database.

Now create the users table inside the sm_app database:

create_users_table = """
CREATE TABLE IF NOT EXISTS users (
  id SERIAL PRIMARY KEY,
  name TEXT NOT NULL, 
  age INTEGER,
  gender TEXT,
  nationality TEXT
)
"""

execute_query(connection, create_users_table)

You can see that the query to create the users table in PostgreSQL is slightly different than SQLite and MySQL. Here, the keyword SERIAL is used to create columns that increment automatically. Recall that MySQL uses the keyword AUTO_INCREMENT.

In addition, foreign key referencing is also specified differently, as shown in the following script that creates the posts table:

create_posts_table = """
CREATE TABLE IF NOT EXISTS posts (
  id SERIAL PRIMARY KEY, 
  title TEXT NOT NULL, 
  description TEXT NOT NULL, 
  user_id INTEGER REFERENCES users(id)
)
"""

execute_query(connection, create_posts_table)

To create the comments table, you’ll have to write a CREATE query for the comments table and pass it to execute_query(). The process for creating the likes table is the same. You only have to modify the CREATE query to create the likes table instead of the comments table.

Inserting Records

In the previous section, you saw how to create tables in your SQLite, MySQL, and PostgreSQL databases by using different Python SQL modules. In this section, you’ll see how to insert records into your tables.

SQLite

To insert records into your SQLite database, you can use the same execute_query() function that you used to create tables. First, you have to store your INSERT INTO query in a string. Then, you can pass the connection object and query string to execute_query(). Let’s insert five records into the users table:

create_users = """
INSERT INTO
  users (name, age, gender, nationality)
VALUES
  ('James', 25, 'male', 'USA'),
  ('Leila', 32, 'female', 'France'),
  ('Brigitte', 35, 'female', 'England'),
  ('Mike', 40, 'male', 'Denmark'),
  ('Elizabeth', 21, 'female', 'Canada');
"""

execute_query(connection, create_users)

Since you set the id column to auto-increment, you don’t need to specify the value of the id column for these users. The users table will auto-populate these five records with id values from 1 to 5.

Now insert six records into the posts table:

create_posts = """
INSERT INTO
  posts (title, description, user_id)
VALUES
  ("Happy", "I am feeling very happy today", 1),
  ("Hot Weather", "The weather is very hot today", 2),
  ("Help", "I need some help with my work", 2),
  ("Great News", "I am getting married", 1),
  ("Interesting Game", "It was a fantastic game of tennis", 5),
  ("Party", "Anyone up for a late-night party today?", 3);
"""

execute_query(connection, create_posts)

It’s important to mention that the user_id column of the posts table is a foreign key that references the id column of the users table. This means that the user_id column must contain a value that already exists in the id column of the users table. If it doesn’t exist, then you’ll see an error.

Similarly, the following script inserts records into the comments and likes tables:

create_comments = """
INSERT INTO
  comments (text, user_id, post_id)
VALUES
  ('Count me in', 1, 6),
  ('What sort of help?', 5, 3),
  ('Congrats buddy', 2, 4),
  ('I was rooting for Nadal though', 4, 5),
  ('Help with your thesis?', 2, 3),
  ('Many congratulations', 5, 4);
"""

create_likes = """
INSERT INTO
  likes (user_id, post_id)
VALUES
  (1, 6),
  (2, 3),
  (1, 5),
  (5, 4),
  (2, 4),
  (4, 2),
  (3, 6);
"""

execute_query(connection, create_comments)
execute_query(connection, create_likes)

In both cases, you store your INSERT INTO query as a string and execute it with execute_query().

MySQL

There are two ways to insert records into MySQL databases from a Python application. The first approach is similar to SQLite. You can store the INSERT INTO query in a string and then use cursor.execute() to insert records.

Earlier, you defined a wrapper function execute_query() that you used to insert records. You can use this same function now to insert records into your MySQL table. The following script inserts records into the users table using execute_query():

create_users = """
INSERT INTO
  `users` (`name`, `age`, `gender`, `nationality`)
VALUES
  ('James', 25, 'male', 'USA'),
  ('Leila', 32, 'female', 'France'),
  ('Brigitte', 35, 'female', 'England'),
  ('Mike', 40, 'male', 'Denmark'),
  ('Elizabeth', 21, 'female', 'Canada');
"""

execute_query(connection, create_users)

The second approach uses cursor.executemany(), which accepts two parameters:

The query string containing placeholders for the records to be inserted
The list of records that you want to insert

Look at the following example, which inserts two records into the likes table:

sql = "INSERT INTO likes ( user_id, post_id ) VALUES ( %s, %s )"
val = [(4, 5), (3, 4)]

cursor = connection.cursor()
cursor.executemany(sql, val)
connection.commit()

It’s up to you which approach you choose to insert records into your MySQL table. If you’re an expert in SQL, then you can use .execute(). If you’re not much familiar with SQL, then it may be more straightforward for you to use .executemany(). With either of the two approaches, you can successfully insert records into the posts, comments, and likes tables.

PostgreSQL

In the previous section, you saw two approaches for inserting records into SQLite database tables. The first uses an SQL string query, and the second uses .executemany(). psycopg2 follows this second approach, though .execute() is used to execute a placeholder-based query.

You pass the SQL query with the placeholders and the list of records to .execute(). Each record in the list will be a tuple, where tuple values correspond to the column values in the database table. Here’s how you can insert user records into the users table in a PostgreSQL database:

users = [
    ("James", 25, "male", "USA"),
    ("Leila", 32, "female", "France"),
    ("Brigitte", 35, "female", "England"),
    ("Mike", 40, "male", "Denmark"),
    ("Elizabeth", 21, "female", "Canada"),
]

user_records = ", ".join(["%s"] * len(users))

insert_query = (
    f"INSERT INTO users (name, age, gender, nationality) VALUES {user_records}"
)

connection.autocommit = True
cursor = connection.cursor()
cursor.execute(insert_query, users)

The script above creates a list users that contains five user records in the form of tuples. Next, you create a placeholder string with five placeholder elements (%s) that correspond to the five user records. The placeholder string is concatenated with the query that inserts records into the users table. Finally, the query string and the user records are passed to .execute(). The above script successfully inserts five records into the users table.

Take a look at another example of inserting records into a PostgreSQL table. The following script inserts records into the posts table:

posts = [
    ("Happy", "I am feeling very happy today", 1),
    ("Hot Weather", "The weather is very hot today", 2),
    ("Help", "I need some help with my work", 2),
    ("Great News", "I am getting married", 1),
    ("Interesting Game", "It was a fantastic game of tennis", 5),
    ("Party", "Anyone up for a late-night party today?", 3),
]

post_records = ", ".join(["%s"] * len(posts))

insert_query = (
    f"INSERT INTO posts (title, description, user_id) VALUES {post_records}"
)

connection.autocommit = True
cursor = connection.cursor()
cursor.execute(insert_query, posts)

You can insert records into the comments and likes tables with the same approach.

Selecting Records

In this section, you’ll see how to select records from database tables using the different Python SQL modules. In particular, you’ll see how to perform SELECT queries on your SQLite, MySQL, and PostgreSQL databases.

SQLite

To select records using SQLite, you can again use cursor.execute(). However, after you’ve done this, you’ll need to call .fetchall(). This method returns a list of tuples where each tuple is mapped to the corresponding row in the retrieved records.

To simplify the process, you can create a function execute_read_query():

def execute_read_query(connection, query):
    cursor = connection.cursor()
    result = None
    try:
        cursor.execute(query)
        result = cursor.fetchall()
        return result
    except Error as e:
        print(f"The error '{e}' occurred")

This function accepts the connection object and the SELECT query and returns the selected record.

`SELECT`

Let’s now select all the records from the users table:

select_users = "SELECT * from users"
users = execute_read_query(connection, select_users)

for user in users:
    print(user)

In the above script, the SELECT query selects all the users from the users table. This is passed to the execute_read_query(), which returns all the records from the users table. The records are then traversed and printed to the console.

Note: It’s not recommended to use SELECT * on large tables since it can result in a large number of I/O operations that increase the network traffic.

The output of the above query looks like this:

(1, 'James', 25, 'male', 'USA')
(2, 'Leila', 32, 'female', 'France')
(3, 'Brigitte', 35, 'female', 'England')
(4, 'Mike', 40, 'male', 'Denmark')
(5, 'Elizabeth', 21, 'female', 'Canada')

In the same way, you can retrieve all the records from the posts table with the below script:

select_posts = "SELECT * FROM posts"
posts = execute_read_query(connection, select_posts)

for post in posts:
    print(post)

The output looks like this:

(1, 'Happy', 'I am feeling very happy today', 1)
(2, 'Hot Weather', 'The weather is very hot today', 2)
(3, 'Help', 'I need some help with my work', 2)
(4, 'Great News', 'I am getting married', 1)
(5, 'Interesting Game', 'It was a fantastic game of tennis', 5)
(6, 'Party', 'Anyone up for a late-night party today?', 3)

The result shows all the records in the posts table.

`JOIN`

You can also execute complex queries involving JOIN operations to retrieve data from two related tables. For instance, the following script returns the user ids and names, along with the description of the posts that these users posted:

select_users_posts = """
SELECT
  users.id,
  users.name,
  posts.description
FROM
  posts
  INNER JOIN users ON users.id = posts.user_id
"""

users_posts = execute_read_query(connection, select_users_posts)

for users_post in users_posts:
    print(users_post)

Here’s the output:

(1, 'James', 'I am feeling very happy today')
(2, 'Leila', 'The weather is very hot today')
(2, 'Leila', 'I need some help with my work')
(1, 'James', 'I am getting married')
(5, 'Elizabeth', 'It was a fantastic game of tennis')
(3, 'Brigitte', 'Anyone up for a late night party today?')

You can also select data from three related tables by implementing multiple JOIN operators. The following script returns all posts, along with the comments on the posts and the names of the users who posted the comments:

select_posts_comments_users = """
SELECT
  posts.description as post,
  text as comment,
  name
FROM
  posts
  INNER JOIN comments ON posts.id = comments.post_id
  INNER JOIN users ON users.id = comments.user_id
"""

posts_comments_users = execute_read_query(
    connection, select_posts_comments_users
)

for posts_comments_user in posts_comments_users:
    print(posts_comments_user)

The output looks like this:

('Anyone up for a late night party today?', 'Count me in', 'James')
('I need some help with my work', 'What sort of help?', 'Elizabeth')
('I am getting married', 'Congrats buddy', 'Leila')
('It was a fantastic game of tennis', 'I was rooting for Nadal though', 'Mike')
('I need some help with my work', 'Help with your thesis?', 'Leila')
('I am getting married', 'Many congratulations', 'Elizabeth')

You can see from the output that the column names are not being returned by .fetchall(). To return column names, you can use the .description attribute of the cursor object. For instance, the following list returns all the column names for the above query:

cursor = connection.cursor()
cursor.execute(select_posts_comments_users)
cursor.fetchall()

column_names = [description[0] for description in cursor.description]
print(column_names)

The output looks like this:

['post', 'comment', 'name']

You can see the names of the columns for the given query.

`WHERE`

Now you’ll execute a SELECT query that returns the post, along with the total number of likes that the post received:

select_post_likes = """
SELECT
  description as Post,
  COUNT(likes.id) as Likes
FROM
  likes,
  posts
WHERE
  posts.id = likes.post_id
GROUP BY
  likes.post_id
"""

post_likes = execute_read_query(connection, select_post_likes)

for post_like in post_likes:
    print(post_like)

The output is as follows:

('The weather is very hot today', 1)
('I need some help with my work', 1)
('I am getting married', 2)
('It was a fantastic game of tennis', 1)
('Anyone up for a late night party today?', 2)

By using a WHERE clause, you’re able to return more specific results.

MySQL

The process of selecting records in MySQL is absolutely identical to selecting records in SQLite. You can use cursor.execute() followed by .fetchall(). The following script creates a wrapper function execute_read_query() that you can use to select records:

def execute_read_query(connection, query):
    cursor = connection.cursor()
    result = None
    try:
        cursor.execute(query)
        result = cursor.fetchall()
        return result
    except Error as e:
        print(f"The error '{e}' occurred")

Now select all the records from the users table:

select_users = "SELECT * FROM users"
users = execute_read_query(connection, select_users)

for user in users:
    print(user)

The output will be similar to what you saw with SQLite.

PostgreSQL

The process of selecting records from a PostgreSQL table with the psycopg2 Python SQL module is similar to what you did with SQLite and MySQL. Again, you’ll use cursor.execute() followed by .fetchall() to select records from your PostgreSQL table. The following script selects all the records from the users table and prints them to the console:

def execute_read_query(connection, query):
    cursor = connection.cursor()
    result = None
    try:
        cursor.execute(query)
        result = cursor.fetchall()
        return result
    except OperationalError as e:
        print(f"The error '{e}' occurred")

select_users = "SELECT * FROM users"
users = execute_read_query(connection, select_users)

for user in users:
    print(user)

Again, the output will be similar to what you’ve seen before.

Updating Table Records

In the last section, you saw how to select records from SQLite, MySQL, and PostgreSQL databases. In this section, you’ll cover the process for updating records using the Python SQL libraries for SQLite, PostgresSQL, and MySQL.

SQLite

Updating records in SQLite is pretty straightforward. You can again make use of execute_query(). As an example, you can update the description of the post with an id of 2. First, SELECT the description of this post:

select_post_description = "SELECT description FROM posts WHERE id = 2"

post_description = execute_read_query(connection, select_post_description)

for description in post_description:
    print(description)

You should see the following output:

('The weather is very hot today',)

The following script updates the description:

update_post_description = """
UPDATE
  posts
SET
  description = "The weather has become pleasant now"
WHERE
  id = 2
"""

execute_query(connection, update_post_description)

Now, if you execute the SELECT query again, you should see the following result:

('The weather has become pleasant now',)

The output has been updated.

MySQL

The process of updating records in MySQL with mysql-connector-python is also a carbon copy of the sqlite3 Python SQL module. You need to pass the string query to cursor.execute(). For example, the following script updates the description of the post with an id of 2:

update_post_description = """
UPDATE
  posts
SET
  description = "The weather has become pleasant now"
WHERE
  id = 2
"""

execute_query(connection,  update_post_description)

Again, you’ve used your wrapper function execute_query() to update the post description.

PostgreSQL

The update query for PostgreSQL is similar to what you’ve seen with SQLite and MySQL. You can use the above scripts to update records in your PostgreSQL table.

Deleting Table Records

In this section, you’ll see how to delete table records using the Python SQL modules for SQLite, MySQL, and PostgreSQL databases. The process of deleting records is uniform for all three databases since the DELETE query for the three databases is the same.

SQLite

You can again use execute_query() to delete records from YOUR SQLite database. All you have to do is pass the connection object and the string query for the record you want to delete to execute_query(). Then, execute_query() will create a cursor object using the connection and pass the string query to cursor.execute(), which will delete the records.

As an example, try to delete the comment with an id of 5:

delete_comment = "DELETE FROM comments WHERE id = 5"
execute_query(connection, delete_comment)

Now, if you select all the records from the comments table, you’ll see that the fifth comment has been deleted.

MySQL

The process for deletion in MySQL is also similar to SQLite, as shown in the following example:

delete_comment = "DELETE FROM comments WHERE id = 2"
execute_query(connection, delete_comment)

Here, you delete the second comment from the sm_app database’s comments table in your MySQL database server.

PostgreSQL

The delete query for PostgreSQL is also similar to SQLite and MySQL. You can write a delete query string by using the DELETE keyword and then passing the query and the connection object to execute_query(). This will delete the specified records from your PostgreSQL database.

Conclusion

In this tutorial, you’ve learned how to use three common Python SQL libraries. sqlite3, mysql-connector-python, and psycopg2 allow you to connect a Python application to SQLite, MySQL, and PostgreSQL databases, respectively.

Now you can:

Interact with SQLite, MySQL, or PostgreSQL databases
Use three different Python SQL modules
Execute SQL queries on various databases from within a Python application

However, this is just the tip of the iceberg! There are also Python SQL libraries for object-relational mapping, such as SQLAlchemy and Django ORM, that automate the task of database interaction in Python. You’ll learn more about these libraries in other tutorials in our Python databases section.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Null in Python: Understanding Python's NoneType Object

2020-02-19T14:00:00+00:00

If you have experience with other programming languages, like C or Java, then you’ve probably heard of the concept of null. Many languages use this to represent a pointer that doesn’t point to anything, to denote when a variable is empty, or to mark default parameters that you haven’t yet supplied. null is often defined to be 0 in those languages, but null in Python is different.

Python uses the keyword None to define null objects and variables. While None does serve some of the same purposes as null in other languages, it’s another beast entirely. As the null in Python, None is not defined to be 0 or any other value. In Python, None is an object and a first-class citizen!

In this tutorial, you’ll learn:

What None is and how to test for it
When and why to use None as a default parameter
What None and NoneType mean in your traceback
How to use None in type checking
How null in Python works under the hood

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Understanding Null in Python

None is the value a function returns when there is no return statement in the function:

>>>

>>> def has_no_return():
...     pass
>>> has_no_return()
>>> print(has_no_return())
None

When you call has_no_return(), there’s no output for you to see. When you print a call to it, however, you’ll see the hidden None it returns.

In fact, None so frequently appears as a return value that the Python REPL won’t print None unless you explicitly tell it to:

>>>

>>> None
>>> print(None)
None

None by itself has no output, but printing it displays None to the console.

Interestingly, print() itself has no return value. If you try to print a call to print(), then you’ll get None:

>>>

>>> print(print("Hello, World!"))
Hello, World!
None

It may look strange, but print(print("...")) shows you the None that the inner print() returns.

None also often used as a signal for missing or default parameters. For instance, None appears twice in the docs for list.sort:

>>>

>>> help(list.sort)
Help on method_descriptor:

sort(...)
    L.sort(key=None, reverse=False) -> None -- stable sort *IN PLACE*

Here, None is the default value for the key parameter as well as the type hint for the return value. The exact output of help can vary from platform to platform. You may get different output when you run this command in your interpreter, but it will be similar.

Using Python’s Null Object `None`

Often, you’ll use None as part of a comparison. One example is when you need to check and see if some result or parameter is None. Take the result you get from re.match. Did your regular expression match a given string? You’ll see one of two results:

Return a Match object: Your regular expression found a match.
Return a None object: Your regular expression did not find a match.

In the code block below, you’re testing if the pattern "Goodbye" matches a string:

>>>

>>> import re
>>> match = re.match(r"Goodbye", "Hello, World!")
>>> if match is None:
...     print("It doesn't match.")
It doesn't match.

Here, you use is None to test if the pattern matches the string "Hello, World!". This code block demonstrates an important rule to keep in mind when you’re checking for None:

Do use the identity operators is and is not.
Do not use the equality operators == and !=.

The equality operators can be fooled when you’re comparing user-defined objects that override them:

>>>

>>> class BrokenComparison:
...     def __eq__(self, other):
...         return True
>>> b = BrokenComparison()
>>> b == None  # Equality operator
True
>>> b is None  # Identity operator
False

Here, the equality operator == returns the wrong answer. The identity operator is, on the other hand, can’t be fooled because you can’t override it.

Note: For more info on how to compare with None, check out Do’s and Dont’s: Python Programming Recommendations.

None is falsy, which means not None is True. If all you want to know is whether a result is falsy, then a test like the following is sufficient:

>>>

>>> some_result = None
>>> if some_result:
...     print("Got a result!")
... else:
...     print("No result.")
...
No result.

The output doesn’t show you that some_result is exactly None, only that it’s falsy. If you must know whether or not you have a None object, then use is and is not.

The following objects are all falsy as well:

Empty lists
Empty dictionaries
Empty sets
Empty strings
0
False

For more on comparisons, truthy, and falsy values, check out How to Use the Python or Operator.

Declaring Null Variables in Python

In some languages, variables come to life from a declaration. They don’t have to have an initial value assigned to them. In those languages, the initial default value for some types of variables might be null. In Python, however, variables come to life from assignment statements. Take a look at the following code block:

>>>

>>> print(bar)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'bar' is not defined
>>> bar = None
>>> print(bar)
None

Here, you can see that a variable with the value None is different from an undefined variable. All variables in Python come into existence by assignment. A variable will only start life as null in Python if you assign None to it.

Using `None` as a Default Parameter

Very often, you’ll use None as the default value for an optional parameter. There’s a very good reason for using None here rather than a mutable type such as a list. Imagine a function like this:

def bad_function(new_elem, starter_list=[]):
    starter_list.append(new_elem)
    return starter_list

bad_function() contains a nasty surprise. It works fine when you call it with an existing list:

>>>

>>> my_list = ['a', 'b', 'c']
>>> bad_function('d', my_list)
['a', 'b', 'c', 'd']

Here, you add `’d” to the end of the list with no problems.

But if you call this function a couple times with no starter_list parameter, then you start to see incorrect behavior:

>>>

>>> bad_function('a')
['a']
>>> bad_function('b')
['a', 'b']
>>> bad_function('c')
['a', 'b', 'c']

The default value for starter_list evaluates only once at the time the function is defined, so the code reuses it every time you don’t pass an existing list.

The right way to build this function is to use None as the default value, then test for it and instantiate a new list as needed:

>>>

 1 >>> def good_function(new_elem, starter_list=None):
 2 ...     if starter_list is None:
 3 ...         starter_list = []
 4 ...     starter_list.append(new_elem)
 5 ...     return starter_list
 6 ...
 7 >>> good_function('e', my_list)
 8 ['a', 'b', 'c', 'd', 'e']
 9 >>> good_function('a')
10 ['a']
11 >>> good_function('b')
12 ['b']
13 >>> good_function('c')
14 ['c']

good_function() behaves as you want by making a new list with each call where you don’t pass an existing list. It works because your code will execute lines 2 and 3 every time it calls the function with the default parameter.

Using `None` as a Null Value in Python

What do you do when None is a valid input object? For instance, what if good_function() could either add an element to the list or not, and None was a valid element to add? In this case, you can define a class specifically for use as a default, while being distinct from None:

>>>

>>> class DontAppend: pass
...
>>> def good_function(new_elem=DontAppend, starter_list=None):
...     if starter_list is None:
...         starter_list = []
...     if new_elem is not DontAppend:
...         starter_list.append(new_elem)
...     return starter_list
...
>>> good_function(starter_list=my_list)
['a', 'b', 'c', 'd', 'e']
>>> good_function(None, my_list)
['a', 'b', 'c', 'd', 'e', None]

Here, the class DontAppend serves as the signal not to append, so you don’t need None for that. That frees you to add None when you want.

You can use this technique when None is a possibility for return values, too. For instance, dict.get returns None by default if a key is not found in the dictionary. If None was a valid value in your dictionary, then you could call dict.get like this:

>>>

>>> class KeyNotFound: pass
...
>>> my_dict = {'a':3, 'b':None}
>>> for key in ['a', 'b', 'c']:
...     value = my_dict.get(key, KeyNotFound)
...     if value is not KeyNotFound:
...         print(f"{key}->{value}")
...
a->3
b->None

Here you’ve defined a custom class KeyNotFound. Now, instead of returning None when a key isn’t in the dictionary, you can return KeyNotFound. That frees you to return None when that’s the actual value in the dictionary.

Deciphering `None` in Tracebacks

When NoneType appears in your traceback, it means that something you didn’t expect to be None actually was None, and you tried to use it in a way that you can’t use None. Almost always, it’s because you’re trying to call a method on it.

For instance, you called append() on my_list many times above, but if my_list somehow became anything other than a list, then append() would fail:

>>>

>>> my_list.append('f')
>>> my_list
['a', 'b', 'c', 'd', 'e', None, 'f']
>>> my_list = None
>>> my_list.append('g')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'append'

Here, your code raises the very common AttributeError because the underlying object, my_list, is not a list anymore. You’ve set it to None, which doesn’t know how to append(), and so the code throws an exception.

When you see a traceback like this in your code, look for the attribute that raised the error first. Here, it’s append(). From there, you’ll see the object you tried to call it on. In this case, it’s my_list, as you can tell from the code just above the traceback. Finally, figure out how that object got to be None and take the necessary steps to fix your code.

Checking for Null in Python

There are two type checking cases where you’ll care about null in Python. The first case is when you’re returning None:

>>>

>>> def returns_None() -> None:
...     pass

This case is similar to when you have no return statement at all, which returns None by default.

The second case is a bit more challenging. It’s where you’re taking or returning a value that might be None, but also might be some other (single) type. This case is like what you did with re.match above, which returned either a Match object or None.

The process is similar for parameters:

from typing import Any, List, Optional
def good_function(new_elem:Any, starter_list:Optional[List]=None) -> List:
    pass

You modify good_function() from above and import Optional from typing to return an Optional[Match].

Taking a Look Under the Hood

In many other languages, null is just a synonym for 0, but null in Python is a full-blown object:

>>>

>>> type(None)
<class 'NoneType'>

This line shows that None is an object, and its type is NoneType.

None itself is built into the language as the null in Python:

>>>

>>> dir(__builtins__)
['ArithmeticError', ..., 'None', ..., 'zip']

Here, you can see None in the list of __builtins__ which is the dictionary the interpreter keeps for the builtins module.

None is a keyword, just like True and False. But because of this, you can’t reach None directly from __builtins__ as you could, for instance, ArithmeticError. However, you can get it with a getattr() trick:

>>>

>>> __builtins__.ArithmeticError
<class 'ArithmeticError'>
>>> __builtins__.None
  File "<stdin>", line 1
    __builtins__.None
                    ^
SyntaxError: invalid syntax
>>> print(getattr(__builtins__, 'None'))
None

When you use getattr(), you can fetch the actual None from __builtins__, which you can’t do by simply asking for it with __builtins__.None.

Even though Python prints the word NoneType in many error messages, NoneType is not an identifier in Python. It’s not in builtins. You can only reach it with type(None).

None is a singleton. That is, the NoneType class only ever gives you the same single instance of None. There’s only one None in your Python program:

>>>

>>> my_None = type(None)()  # Create a new instance
>>> print(my_None)
None
>>> my_None is None
True

Even though you try to create a new instance, you still get the existing None.

You can prove that None and my_None are the same object by using id():

>>>

>>> id(None)
4465912088
>>> id(my_None)
4465912088

Here, the fact that id outputs the same integer value for both None and my_None means they are, in fact, the same object.

Note: The actual value produced by id will vary across systems, and even between program executions. Under CPython, the most popular Python runtime, id() does its job by reporting the memory address of an object. Two objects that live at the same memory address are the same object.

If you try to assign to None, then you’ll get a SyntaxError:

>>>

>>> None = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SyntaxError: can't assign to keyword
>>> None.age = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'age'
>>> setattr(None, 'age', 5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'age'
>>> setattr(type(None), 'age', 5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'NoneType'

All the examples above show that you can’t modify None or NoneType. They are true constants.

You can’t subclass NoneType, either:

>>>

>>> class MyNoneType(type(None)):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'NoneType' is not an acceptable base type

This traceback shows that the interpreter won’t let you make a new class that inherits from type(None).

Conclusion

None is a powerful tool in the Python toolbox. Like True and False, None is an immutable keyword. As the null in Python, you use it to mark missing values and results, and even default parameters where it’s a much better choice than mutable types.

Now you can:

Test for None with is and is Not
Choose when None is a valid value in your code
Use None and its alternatives as default parameters
Decipher None and NoneType in your tracebacks
Use None and Optional in type hints

How do you use the null in Python? Leave a comment down in the comments section below!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Finding the Perfect Python Code Editor

2020-02-18T14:00:00+00:00

Find your perfect Python development setup with this review of Python IDEs and code editors. Writing Python using IDLE or the Python REPL is great for simple things, but not ideal for larger programming projects. With this course you’ll get an overview of the most common Python coding environments to help you make an informed decision.

By the end of this course, you’ll know how to:

Choose the Python editing environment that’s right for you
Perform common tasks like creating, running, and debugging code
Dig deeper into optimizing your favorite editing setup

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Guide to the Newer Python String Format Techniques

2020-02-17T14:00:00+00:00

In the previous tutorial in this introductory series, you learned how to format string data using the string modulo operator. The string modulo operator is useful, and it’s good for you to be familiar with it because you’re likely to encounter it in older Python code. However, there are two newer ways that you can use Python to format strings that are arguably more preferable.

In this tutorial, you’ll learn about:

The string .format() method
The formatted string literal, or f-string

You’ll learn about these formatting techniques in detail and add them to your Python string formatting toolkit. Note that there’s a standard module called string containing a class called Template, which provides some string formatting through interpolation. The string modulo operator provides more or less the same functionality, so you won’t cover string.Template here.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

The Python String `.format()` Method

The Python string .format() method was introduced in version 2.6. It’s similar in many ways to the string modulo operator, but .format() goes well beyond in versatility. The general form of a Python .format() call is shown below:

<template>.format(<positional_argument(s)>, <keyword_argument(s)>)

Note that this is a method, not an operator. You call the method on <template>, which is a string containing replacement fields. The <positional_arguments> and <keyword_arguments> to the method specify values that are inserted into <template> in place of the replacement fields. The resulting formatted string is the method’s return value.

In the <template> string, replacement fields are enclosed in curly braces ({}). Anything not contained in curly braces is literal text that’s copied directly from the template to the output. If you need to include a literal curly bracket character, like { or }, in the template string, then you can escape this character by doubling it:

>>>

>>> '{{ {0} }}'.format('foo')
'{ foo }'

Now the curly braces are included in your output.

The String `.format()` Method: Arguments

Let’s start with a quick example to get you acquainted before you dive into more detail on how to use this method in Python to format strings. For review, here’s the first example from the previous tutorial on the string modulo operator:

>>>

>>> print('%d %s cost $%.2f' % (6, 'bananas', 1.74))
6 bananas cost $1.74

Here, you used the string modulo operator in Python to format the string. Now, you can use Python’s string .format() method to obtain the same result, like this:

>>>

>>> print('{0} {1} cost ${2}'.format(6, 'bananas', 1.74))
6 bananas cost $1.74

In this example, <template> is the string '{0} {1} cost ${2}'. The replacement fields are {0}, {1}, and {2}, which contain numbers that correspond to the zero-based positional arguments 6, 'bananas', and 1.74. Each positional argument is inserted into the template in place of its corresponding replacement field:

Using The String .format() Method in Python to Format a String With Positional Arguments

The next example uses keyword arguments instead of positional parameters to produce the same result:

>>>

>>> print('{quantity} {item} cost ${price}'.format(
...     quantity=6,
...     item='bananas',
...     price=1.74))
6 bananas cost $1.74

In this case, the replacement fields are {quantity}, {item}, and {price}. These fields specify keywords that correspond to the keyword arguments quantity=6, item='bananas', and price=1.74. Each keyword value is inserted into the template in place of its corresponding replacement field:

Using The String .format() Method in Python to Format a String With Keyword Arguments

You’ll learn more about positional and keywords arguments there in the next tutorial in this introductory series, which explores functions and argument passing. For now, the two sections that follow will show you how these are used with the Python .format() method.

Positional Arguments

Positional arguments are inserted into the template in place of numbered replacement fields. Like list indexing, the numbering of replacement fields is zero-based. The first positional argument is numbered 0, the second is numbered 1, and so on:

>>>

>>> '{0}/{1}/{2}'.format('foo', 'bar', 'baz')
'foo/bar/baz'

Note that replacement fields don’t have to appear in the template in numerical order. They can be specified in any order, and they can appear more than once:

>>>

>>> '{2}.{1}.{0}/{0}{0}.{1}{1}.{2}{2}'.format('foo', 'bar', 'baz')
'baz.bar.foo/foofoo.barbar.bazbaz'

When you specify a replacement field number that’s out of range, you’ll get an error. In the following example, the positional arguments are numbered 0, 1, and 2, but you specify {3} in the template:

>>>

>>> '{3}'.format('foo', 'bar', 'baz')
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    '{3}'.format('foo', 'bar', 'baz')
IndexError: tuple index out of range

This raises an IndexError exception.

Starting with Python 3.1, you can omit the numbers in the replacement fields, in which case the interpreter assumes sequential order. This is referred to as automatic field numbering:

>>>

>>> '{}/{}/{}'.format('foo', 'bar', 'baz')
'foo/bar/baz'

When you specify automatic field numbering, you must provide at least as many arguments as there are replacement fields:

>>>

>>> '{}{}{}{}'.format('foo','bar','baz')
Traceback (most recent call last):
  File "<pyshell#27>", line 1, in <module>
    '{}{}{}{}'.format('foo','bar','baz')
IndexError: tuple index out of range

In this case, there are four replacement fields in the template but only three arguments, so an IndexError exception occurs. On the other hand, it’s fine if the arguments outnumber the replacement fields. The excess arguments simply aren’t used:

>>>

>>> '{}{}'.format('foo', 'bar', 'baz')
'foobar'

Here, the argument 'baz' is ignored.

Note that you can’t intermingle these two techniques:

>>>

>>> '{1}{}{0}'.format('foo','bar','baz')
Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    '{1}{}{0}'.format('foo','bar','baz')
ValueError: cannot switch from manual field specification to automatic field
 numbering

When you use Python to format strings with positional arguments, you must choose between either automatic or explicit replacement field numbering.

Keyword Arguments

Keyword arguments are inserted into the template string in place of keyword replacement fields with the same name:

>>>

>>> '{x}/{y}/{z}'.format(x='foo', y='bar', z='baz')
'foo/bar/baz'

In this example, the values of the keyword arguments x, y, and z take the place of the replacement fields {x}, {y}, and {z}, respectively.

If you refer to a keyword argument that’s missing, then you’ll see an error:

>>>

>>> '{x}/{y}/{w}'.format(x='foo', y='bar', z='baz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'w'

Here, you specify replacement field {w}, but there’s no corresponding keyword argument named w. Python raises a KeyError exception.

While you have to specify positional arguments in sequential order, but you can specify keyword arguments in any arbitrary order:

>>>

>>> '{0}/{1}/{2}'.format('foo', 'bar', 'baz')
'foo/bar/baz'
>>> '{0}/{1}/{2}'.format('bar', 'baz', 'foo')
'bar/baz/foo'

>>> '{x}/{y}/{z}'.format(x='foo', y='bar', z='baz')
'foo/bar/baz'
>>> '{x}/{y}/{z}'.format(y='bar', z='baz', x='foo')
'foo/bar/baz'

You can specify both positional and keyword arguments in one Python .format() call. Just note that, if you do so, then all the positional arguments must appear before any of the keyword arguments:

>>>

>>> '{0}{x}{1}'.format('foo', 'bar', x='baz')
'foobazbar'
>>> '{0}{x}{1}'.format('foo', x='baz', 'bar')
  File "<stdin>", line 1
SyntaxError: positional argument follows keyword argument

In fact, the requirement that all positional arguments appear before any keyword arguments doesn’t apply only to Python format methods. This is generally true of any function or method call. You’ll learn more about this in the next tutorial in this series, which explores functions and function calls.

In all the examples shown so far, the values you passed to the Python .format() method have been literal values, but you may specify variables as well:

>>>

>>> x = 'foo'
>>> y = 'bar'
>>> z = 'baz'
>>> '{0}/{1}/{s}'.format(x, y, s=z)
'foo/bar/baz'

In this case, you pass the variables x and y as positional parameter values and z as a keyword parameter value.

The String `.format()` Method: Simple Replacement Fields

As you’ve seen, when you call Python’s .format() method, the <template> string contains replacement fields. These indicate where in the template to insert the arguments to the method. A replacement field consists of three components:

{[<name>][!<conversion>][:<format_spec>]}

These components are interpreted as follows:

Component	Meaning
`<name>`	Specifies the source of the value to be formatted
`<conversion>`	Indicates which standard Python function to use to perform the conversion
`<format_spec>`	Specifies more detail about how the value should be converted

Each component is optional and may be omitted. Let’s take a look at each component in more depth.

The `<name>` Component

The <name> component is the first portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

<name> indicates which argument from the argument list is inserted into the Python format string in the given location. It’s either a number for a positional argument or a keyword for a keyword argument. In the following example, the <name> components of the replacement fields are 0, 1, and baz, respectively:

>>>

>>> x, y, z = 1, 2, 3
>>> '{0}, {1}, {baz}'.format(x, y, baz=z)
'1, 2, 3'

If an argument is a list, then you can use indices with <name> to access the list’s elements:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> '{0[0]}, {0[2]}'.format(a)
'foo, baz'
>>> '{my_list[0]}, {my_list[2]}'.format(my_list=a)
'foo, baz'

Similarly, you can use a key reference with <name> if the corresponding argument is a dictionary:

>>>

>>> d = {'key1': 'foo', 'key2': 'bar'}
>>> d['key1']
'foo'
>>> '{0[key1]}'.format(d)
'foo'
>>> d['key2']
'bar'
>>> '{my_dict[key2]}'.format(my_dict=d)
'bar'

You can also reference object attributes from within a replacement field. In the previous tutorial in this series, you learned that virtually every item of data in Python is an object. Objects may have attributes assigned to them that are accessed using dot notation:

obj.attr

Here, obj is an object with an attribute named attr. You use dot notation to access the object’s attribute. Let’s see an example. Complex numbers in Python have attributes named .real and .imag that represent the real and imaginary portions of the number. You can access these using dot notation as well:

>>>

>>> z = 3+5j
>>> type(z)
<class 'complex'>
>>> z.real
3.0
>>> z.imag
5.0

There are several upcoming tutorials in this series on object-oriented programming, in which you’ll learn a great deal more about object attributes like these.

The relevance of object attributes in this context is that you can specify them in a Python .format() replacement field:

>>>

>>> z
(3+5j)
>>> 'real = {0.real}, imag = {0.imag}'.format(z)
'real = 3.0, imag = 5.0'

As you can see, it’s relatively straightforward in Python to format components of complex objects using the .format() method.

The `<conversion>` Component

The <conversion> component is the middle portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

Python can format an object as a string using three different built-in functions:

str()
repr()
ascii()

By default, the Python .format() method uses str(), but in some instances, you may want to force .format() to use one of the other two. You can do this with the <conversion> component of a replacement field. The possible values for <conversion> are shown in the table below:

Value	Meaning
`!s`	Convert with `str()`
`!r`	Convert with `repr()`
`!a`	Convert with `ascii()`

The following examples force Python to perform string conversion using str(), repr(), and ascii(), respectively:

>>>

>>> '{0!s}'.format(42)
'42'
>>> '{0!r}'.format(42)
'42'
>>> '{0!a}'.format(42)
'42'

In many cases, the result is the same regardless of which conversion function you use, as you can see in the example above. That being said, you won’t often need the <conversion> component, so you won’t spend a lot of time on it here. However, there are situations where it makes a difference, so it’s good to be aware that you have the capability to force a specific conversion function if you need to.

The `<format_spec>` Component

The <format_spec> component is the last portion of a replacement field:

{[<name>][!<conversion>][:<format_spec>]}

<format_spec> represents the guts of the Python .format() method’s functionality. It contains information that exerts fine control over how values are formatted prior to being inserted into the template string. The general form is this:

:[[<fill>]<align>][<sign>][#][0][<width>][<group>][.<prec>][<type>]

The ten subcomponents of <format_spec> are specified in the order shown. They control formatting as described in the table below:

Subcomponent	Effect
`:`	Separates the `<format_spec>` from the rest of the replacement field
`<fill>`	Specifies how to pad values that don’t occupy the entire field width
`<align>`	Specifies how to justify values that don’t occupy the entire field width
`<sign>`	Controls whether a leading sign is included for numeric values
`#`	Selects an alternate output form for certain presentation types
`0`	Causes values to be padded on the left with zeros instead of ASCII space characters
`<width>`	Specifies the minimum width of the output
`<group>`	Specifies a grouping character for numeric output
`.<prec>`	Specifies the number of digits after the decimal point for floating-point presentation types, and the maximum output width for string presentations types
`<type>`	Specifies the presentation type, which is the type of conversion performed on the corresponding argument

These functions are analogous to the components you’ll find in the string modulo operator’s conversion specifier, but with somewhat greater capability. You’ll see their capabilities explained more fully in the following sections.

The `<type>` Subcomponent

Let’s start with <type>, which is the final portion of <format_spec>. The <type> subcomponent specifies the presentation type, which is the type of conversion that’s performed on the corresponding value to produce the output. The possible values are shown below:

Value	Presentation Type
`b`	Binary integer
`c`	Single character
`d`	Decimal integer
`e` or `E`	Exponential
`f` or `F`	Floating point
`g` or `G`	Floating point or Exponential
`o`	Octal integer
`s`	String
`x` or `X`	Hexadecimal integer
`%`	Percentage

These are like the conversion types used with the string modulo operator, and in many cases, they function the same. The following examples demonstrate the similarity:

>>>

>>> '%d' % 42
'42'
>>> '{:d}'.format(42)
'42'

>>> '%f' % 2.1
'2.100000'
>>> '{:f}'.format(2.1)
'2.100000'

>>> '%s' % 'foobar'
'foobar'
>>> '{:s}'.format('foobar')
'foobar'

>>> '%x' % 31
'1f'
>>> '{:x}'.format(31)
'1f'

However, there are some minor differences between some of the Python .format() presentation types and the string modulo operator conversion types:

Type	`.format()` Method	String Modulo Operator
b	Designates binary integer conversion	Not supported
i, u	Not supported	Designates integer conversion
c	Designates character conversion, and the corresponding value must be an integer	Designates character conversion, but the corresponding value may be either an integer or a single-character string
g, G	Chooses between floating point or exponential output, but the rules governing the choice are slightly more complicated	Chooses between floating point or exponential output, depending on the magnitude of the exponent and the value specified for `<prec>`
r, a	Not supported (though the functionality is provided by the `!r` and `!a` conversion components in the replacement field)	Designates conversion with `repr()` or `ascii()`, respectively
%	Converts a numeric argument to a percentage	Inserts a literal `'%'` character into the output

Next, you’ll see several examples illustrating these differences, as well as some of the added features of the Python .format() method presentation types. The first presentation type you’ll see is b, which designates binary integer conversion:

>>>

>>> '{:b}'.format(257)
'100000001'

The modulo operator doesn’t support binary conversion type at all:

>>>

>>> '%b' % 257
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unsupported format character 'b' (0x62) at index 1

However, the modulo operator does allow decimal integer conversion with any of the d, i, and u types. Only the d presentation type indicates decimal integer conversion with the Python .format() method. The i and u presentation types aren’t supported and aren’t really necessary.

Next up is single character conversion. The modulo operator allows either an integer or single character value with the c conversion type:

>>>

>>> '%c' % 35
'#'
>>> '%c' % '#'
'#'

On the other hand, Python’s .format() method requires that the value corresponding to the c presentation type be an integer:

>>>

>>> '{:c}'.format(35)
'#'
>>> '{:c}'.format('#')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'c' for object of type 'str'

When you try to pass a value of a different type, you’ll get a ValueError.

For both the string modulo operator and Python’s .format() method, the g conversion type chooses either floating-point or exponential output, depending on the magnitude of the exponent and the value specified for <prec>:

>>>

>>> '{:g}'.format(3.14159)
'3.14159'

>>> '{:g}'.format(-123456789.8765)
'-1.23457e+08'

The exact rules governing the choice are slightly more complicated with .format() than they are with the modulo operator. Generally, you can trust that the choice will make sense.

G is identical to g except for when the output is exponential, in which case the 'E' will be in uppercase:

>>>

>>> '{:G}'.format(-123456789.8765)
'-1.23457E+08'

The result is the same as in the previous example, but this time with an uppercase 'E'.

Note: There are a couple of other situations where you’ll see a difference between the g and G presentations types.

Under some circumstances, a floating-point operation can result in a value that’s essentially infinite. The string representation of such a number in Python is 'inf'. It may also happen that a floating-point operation produces a value that can’t be represented as a number. Python represents this with the string 'NaN', which stands for Not a Number.

When you pass these values to Python’s .format() method, the g presentation type produces lowercase output, and G produces uppercase output:

>>>

>>> x = 1e300 * 1e300
>>> x
inf

>>> '{:g}'.format(x)
'inf'
>>> '{:g}'.format(x * 0)
'nan'

>>> '{:G}'.format(x)
'INF'
>>> '{:G}'.format(x * 0)
'NAN'

You’ll see similar behavior with the f and F presentations types as well:

>>>

>>> '{:f}'.format(x)
'inf'
>>> '{:F}'.format(x)
'INF'

>>> '{:f}'.format(x * 0)
'nan'
>>> '{:F}'.format(x * 0)
'NAN'

For more information on floating-point representation, inf, and NaN, check out the Wikipedia page on IEEE 754-1985.

The modulo operator supports r and a conversion types to force conversion by repr() and ascii(), respectively. Python’s .format() method doesn’t support r and a presentation types, but you can accomplish the same thing with the !r and !a conversion components in the replacement field.

Finally, you can use the % conversion type with the modulo operator to insert a literal '%' character into the output:

>>>

>>> '%f%%' % 65.0
'65.000000%'

You don’t need anything special to insert a literal '%' character into the Python .format() method’s output, so the % presentation type serves a different handy purpose for percent output. It multiplies the specified value by 100 and appends a percent sign:

>>>

>>> '{:%}'.format(0.65)
'65.000000%'

The remaining parts of <format_spec> indicate how the chosen presentation type is formatted, in much the same way as the string modulo operator’s width and precision specifiers and conversion flags. These are described more fully in the following sections.

The `<fill>` and `<align>` Subcomponents

<fill> and <align> control how formatted output is padded and positioned within the specified field width. These subcomponents only have meaning when the formatted field value doesn’t occupy the entire field width, which can only happen if a minimum field width is specified with <width>. If <width> isn’t specified, then <fill> and <align> are effectively ignored. You’ll cover <width> later on in this tutorial.

Here are the possible values for the <align> subcomponent:

A value using the less than sign (<) indicates that the output is left-justified:

>>>

>>> '{0:<8s}'.format('foo')
'foo     '
>>> '{0:<8d}'.format(123)
'123     '

This behavior is the default for string values.

A value using the greater than sign (>) indicates that the output should be right-justified:

>>>

>>> '{0:>8s}'.format('foo')
'     foo'
>>> '{0:>8d}'.format(123)
'     123'

This behavior is the default for numeric values.

A value using a caret (^) indicates that the output should be centered in the output field:

>>>

>>> '{0:^8s}'.format('foo')
'  foo   '
>>> '{0:^8d}'.format(123)
'  123   '

Finally, you can also specify a value using the equals sign (=) for the <align> subcomponent. This only has meaning for numeric values, and only when a sign is included in the output.

When numeric output includes a sign, it’s normally placed directly to the left of the first digit in the number, as shown above. If <align> is set to the equals sign (=), then the sign appears at the left edge of the output field, and padding is placed in between the sign and the number:

>>>

>>> '{0:+8d}'.format(123)
'    +123'
>>> '{0:=+8d}'.format(123)
'+    123'

>>> '{0:+8d}'.format(-123)
'    -123'
>>> '{0:=+8d}'.format(-123)
'-    123'

You’ll cover the <sign> component in detail in the next section.

<fill> specifies how to fill in extra space when the formatted value doesn’t completely fill the output width. It can be any character except for curly braces ({}). (If you really feel compelled to pad a field with curly braces, then you’ll just have to find another way!)

Some examples of the use of <fill> are shown below:

>>>

>>> '{0:->8s}'.format('foo')
'-----foo'

>>> '{0:#<8d}'.format(123)
'123#####'

>>> '{0:*^8s}'.format('foo')
'**foo***'

If you specify a value for <fill>, then you should also include a value for <align> as well.

The `<sign>` Subcomponent

You can control whether a sign appears in numeric output with the <sign> component. For example, in the following, the plus sign (+) specified in the <format_spec> indicates that the value should always be displayed with a leading sign:

>>>

>>> '{0:+6d}'.format(123)
'  +123'
>>> '{0:+6d}'.format(-123)
'  -123'

Here, you use the plus sign (+), so a sign will always be included for both positive and negative values. If you use the minus sign (-), then only negative numeric values will include a leading sign, and positive values won’t:

>>>

>>> '{0:-6d}'.format(123)
'   123'
>>> '{0:-6d}'.format(-123)
'  -123'

When you use a single space (' '), it means a sign is included for negative values, and an ASCII space character for positive values:

>>>

>>> '{0:*> 6d}'.format(123)
'** 123'

>>> '{0:*>6d}'.format(123)
'***123'

>>> '{0:*> 6d}'.format(-123)
'**-123'

Since the space character is the default fill character, you’d only notice the effect of this if an alternate <fill> character is specified.

Lastly, recall from above that when you specify the equals sign (=) for <align> and you include a <sign> specifier, the padding goes between the sign and the value, rather than to the left of the sign.

The `#` Subcomponent

When you specify a hash character (#) in <format_spec>, Python will select an alternate output form for certain presentation types. This is analogous to the # conversion flag for the string modulo operator. For binary, octal, and hexadecimal presentation types, the hash character (#) causes inclusion of an explicit base indicator to the left of the value:

>>>

>>> '{0:b}, {0:#b}'.format(16)
'10000, 0b10000'
>>> '{0:o}, {0:#o}'.format(16)
'20, 0o20'
>>> '{0:x}, {0:#x}'.format(16)
'10, 0x10'

As you can see, the base indicator can be 0b, 0o, or 0x.

For floating-point or exponential presentation types, the hash character forces the output to contain a decimal point, even if the output consists of a whole number:

>>>

>>> '{0:.0f}, {0:#.0f}'.format(123)
'123, 123.'
>>> '{0:.0e}, {0:#.0e}'.format(123)
'1e+02, 1.e+02'

For any presentation type other than those shown above, the hash character (#) has no effect.

The `0` Subcomponent

If output is smaller than the indicated field width and you specify the digit zero (0) in <format_spec>, then values will be padded on the left with zeros instead of ASCII space characters:

>>>

>>> '{0:05d}'.format(123)
'00123'
>>> '{0:08.1f}'.format(12.3)
'000012.3'

You’ll typically use this for numeric values, as shown above. However, it works for string values as well:

>>>

>>> '{0:>06s}'.format('foo')
'000foo'

If you specify both <fill> and <align>, then <fill> overrides 0:

>>>

>>> '{0:*>05d}'.format(123)
'**123'

<fill> and 0 essentially control the same thing, so there really isn’t any need to specify both. In fact, 0 is really superfluous, and was probably included as a convenience for developers who are familiar with the string modulo operator’s similar 0 conversion flag.

The `<width>` Subcomponent

<width> specifies the minimum width of the output field:

>>>

>>> '{0:8s}'.format('foo')
'foo     '
>>> '{0:8d}'.format(123)
'     123'

Note that this is a minimum field width. Suppose you specify a value that’s longer than the minimum:

>>>

>>> '{0:2s}'.format('foobar')
'foobar'

In this case, <width> is effectively ignored.

The `<group>` Subcomponent

<group> allows you to include a grouping separator character in numeric output. For decimal and floating-point presentation types, <group> may be specified as either a comma character (,) or an underscore character (_). That character then separates each group of three digits in the output:

>>>

>>> '{0:,d}'.format(1234567)
'1,234,567'
>>> '{0:_d}'.format(1234567)
'1_234_567'

>>> '{0:,.2f}'.format(1234567.89)
'1,234,567.89'
>>> '{0:_.2f}'.format(1234567.89)
'1_234_567.89'

A <group> value using an underscore (_) may also be specified with the binary, octal, and hexadecimal presentation types. In that case, each group of four digits is separated by an underscore character in the output:

>>>

>>> '{0:_b}'.format(0b111010100001)
'1110_1010_0001'
>>> '{0:#_b}'.format(0b111010100001)
'0b1110_1010_0001'

>>> '{0:_x}'.format(0xae123fcc8ab2)
'ae12_3fcc_8ab2'
>>> '{0:#_x}'.format(0xae123fcc8ab2)
'0xae12_3fcc_8ab2'

If you try to specify <group> with any presentation type other than those listed above, then your code will raise an exception.

The `.<prec>` Subcomponent

.<prec> specifies the number of digits after the decimal point for floating point presentation types:

>>>

>>> '{0:8.2f}'.format(1234.5678)
' 1234.57'
>>> '{0:8.4f}'.format(1.23)
'  1.2300'

>>> '{0:8.2e}'.format(1234.5678)
'1.23e+03'
>>> '{0:8.4e}'.format(1.23)
'1.2300e+00'

For string types, .<prec> specifies the maximum width of the converted output:

>>>

>>> '{:.4s}'.format('foobar')
'foob'

If the output would be longer than the value specified, then it will be truncated.

The String `.format()` Method: Nested Replacement Fields

Recall that you can specify either <width> or <prec> by an asterisk with the string modulo operator:

>>>

>>> w = 10
>>> p = 2
>>> '%*.*f' % (w, p, 123.456)  # Width is 10, precision is 2
'    123.46'

The associated values are then taken from the argument list. This allows <width> and <prec> to be evaluated dynamically at run-time, as shown in the example above. Python’s .format() method provides similar capability using nested replacement fields.

Inside a replacement field, you can specify a nested set of curly braces ({}) that contains a name or number referring to one of the method’s positional or keyword arguments. That portion of the replacement field will then be evaluated at run-time and replaced using the corresponding argument. You can accomplish the same effect as the above string modulo operator example with nested replacement fields:

>>>

>>> w = 10
>>> p = 2
>>> '{2:{0}.{1}f}'.format(w, p, 123.456)
'    123.46'

Here, the <name> component of the replacement field is 2, which indicates the third positional parameter whose value is 123.456. This is the value to be formatted. The nested replacement fields {0} and {1} correspond to the first and second positional parameters, w and p. These occupy the <width> and <prec> locations in <format_spec> and allow field width and precision to be evaluated at run-time.

You can use keyword arguments with nested replacement fields as well. This example is functionally equivalent to the previous one:

>>>

>>> w = 10
>>> p = 2
>>> '{val:{wid}.{pr}f}'.format(wid=w, pr=p, val=123.456)
'    123.46'

In either case, the values of w and p are evaluated at run-time and used to modify the <format_spec>. The result is effectively the same as this:

>>>

>>> '{0:10.2f}'.format(123.456)
'    123.46'

The string modulo operator only allows <width> and <prec> to be evaluated at run-time in this way. By contrast, with Python’s .format() method you can specify any portion of <format_spec> using nested replacement fields.

In the following example, the presentation type <type> is specified by a nested replacement field and determined dynamically:

>>>

>>> bin(10), oct(10), hex(10)
('0b1010', '0o12', '0xa')
>>> for t in ('b', 'o', 'x'):
...     print('{0:#{type}}'.format(10, type=t))
...
0b1010
0o12
0xa

Here, the grouping character <group> is nested:

>>>

>>> '{0:{grp}d}'.format(123456789, grp='_')
'123_456_789'
>>> '{0:{grp}d}'.format(123456789, grp=',')
'123,456,789'

Whew! That was an adventure. The specification of the template string is virtually a language unto itself!

As you can see, string formatting can be very finely tuned when you use Python’s .format() method. Next, you’ll see one more technique for string and output formatting that affords all the advantages of .format(), but with more direct syntax.

The Python Formatted String Literal (f-String)

In version 3.6, a new Python string formatting syntax was introduced, called the formatted string literal. These are also informally called f-strings, a term that was initially coined in PEP 498, where they were first proposed.

f-String Syntax

An f-string looks very much like a typical Python string except that it’s prepended by the character f:

>>>

>>> f'foo bar baz'
'foo bar baz'

You can also use an uppercase F:

>>>

>>> s = F'qux quux'
>>> s
'qux quux'

The effect is exactly the same. Just like with any other type of string, you can use single, double, or triple quotes to define an f-string:

>>>

>>> f'foo'
'foo'
>>> f"bar"
'bar'
>>> f'''baz'''
'baz'

The magic of f-strings is that you can embed Python expressions directly inside them. Any portion of an f-string that’s enclosed in curly braces ({}) is treated as an expression. The expression is evaluated and converted to string representation, and the result is interpolated into the original string in that location:

>>>

>>> s = 'bar'
>>> print(f'foo.{s}.baz')
foo.bar.baz

The interpreter treats the remainder of the f-string—anything not inside curly braces—just as it would an ordinary string. For example, escape sequences are processed as expected:

>>>

>>> s = 'bar'
>>> print(f'foo\n{s}\nbaz')
foo
bar
baz

Here’s the example from earlier using an f-string:

>>>

>>> quantity = 6
>>> item = 'bananas'
>>> price = 1.74
>>> print(f'{quantity} {item} cost ${price}')
6 bananas cost $1.74

This is equivalent to the following:

>>>

>>> quantity = 6
>>> item = 'bananas'
>>> price = 1.74
>>> print('{0} {1} cost ${2}'.format(quantity, item, price))
6 bananas cost $1.74

Expressions embedded in f-strings can be almost arbitrarily complex. The examples below show some of the possibilities:

Variables:

>>>

>>> quantity, item, price = 6, 'bananas', 1.74
>>> f'{quantity} {item} cost ${price}'
'6 bananas cost $1.74'

Arithmetic expressions:

>>>

>>> quantity, item, price = 6, 'bananas', 1.74
>>> print(f'Price per item is ${price/quantity}')
Price per item is $0.29

>>> x = 6
>>> print(f'{x} cubed is {x**3}')
6 cubed is 216

Objects of composite types:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> d = {'foo': 1, 'bar': 2}

>>> print(f'a = {a} | d = {d}')
a = ['foo', 'bar', 'baz'] | d = {'foo': 1, 'bar': 2}

Indexing, slicing, and key references:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> d = {'foo': 1, 'bar': 2}

>>> print(f'First item in list a = {a[0]}')
First item in list a = foo

>>> print(f'Last two items in list a = {a[-2:]}')
Last two items in list a = ['bar', 'baz']

>>> print(f'List a reversed = {a[::-1]}')
List a reversed = ['baz', 'bar', 'foo']

>>> print(f"Dict value for key 'bar' is {d['bar']}")
Dict value for key 'bar' is 2

Function and method calls:

>>>

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
>>> print(f'List a has {len(a)} items')
List a has 5 items

>>> s = 'foobar'
>>> f'--- {s.upper()} ---'
'--- FOOBAR ---'

>>> d = {'foo': 1, 'bar': 2}
>>> print(f"Dict value for key 'bar' is {d.get('bar')}")
Dict value for key 'bar' is 2

Conditional expressions:

>>>

>>> x = 3
>>> y = 7
>>> print(f'The larger of {x} and {y} is {x if x > y else y}')
The larger of 3 and 7 is 7

>>> age = 13
>>> f'I am {"a minor" if age < 18 else "an adult"}.'
'I am a minor.'

Object attributes:

>>>

>>> z = 3+5j
>>> z
(3+5j)

>>> print(f'real = {z.real}, imag = {z.imag}')
real = 3.0, imag = 5.0

To include a literal curly brace in an f-string, escape it by doubling it, the same as you would in a template string for Python’s .format() method:

>>>

>>> z = 'foobar'
>>> f'{{ {z[::-1]} }}'
'{ raboof }'

You may prefix an f-string with 'r' or 'R' to indicate that it is a raw f-string. In that case, backslash sequences are left intact, just like with an ordinary string:

>>>

>>> z = 'bar'
>>> print(f'foo\n{z}\nbaz')
foo
bar
baz
>>> print(rf'foo\n{z}\nbaz')
foo\nbar\nbaz
>>> print(fr'foo\n{z}\nbaz')
foo\nbar\nbaz

Note that you can specify the 'r' first and then the 'f', or vice-versa.

f-String Expression Limitations

There are a few minor restrictions on f-string expression. These aren’t too limiting, but you should know what they are. First, an f-string expression can’t be empty:

>>>

>>> f'foo{}bar'
  File "<stdin>", line 1
SyntaxError: f-string: empty expression not allowed

It isn’t obvious why you’d want to do this. But if you feel compelled to try, then just know that it won’t work.

Additionally, an f-string expression can’t contain a backslash (\) character. Among other things, that means you can’t use a backslash escape sequence in an f-string expression:

>>>

>>> print(f'foo{\n}bar')
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash
>>> print(f'foo{\'}bar')
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash

You can get around this limitation by creating a temporary variable that contains the escape sequence you want to insert:

>>>

>>> nl = '\n'
>>> print(f'foo{nl}bar')
foo
bar
>>> quote = '\''
>>> print(f'foo{quote}bar')
foo'bar

Lastly, an expression in an f-string that is triple-quoted can’t contain comments:

>>>

>>> z = 'bar'

>>> print(f'''foo{
... z
... }baz''')
foobarbaz

>>> print(f'''foo{
... z    # Comment
... }''')
  File "<stdin>", line 3
SyntaxError: f-string expression part cannot include '#'

Note, however, that the multiline f-string may contain embedded newlines.

f-String Formatting

Like Python’s .format() method, f-strings support extensive modifiers that control the final appearance of the output string. There’s more good news, too. If you’ve mastered the Python .format() method, then you already know how to use Python to format f-strings!

Expressions in f-strings can be modified by a <conversion> or <format_spec>, just like replacement fields used in the .format() template. The syntax is identical. In fact, in both cases Python will format the replacement field using the same internal function. In the following example, !r is specified as a <conversion> component in the .format() template string:

>>>

>>> s = 'foo'
>>> '{0!r}'.format(s)
"'foo'"

This forces conversion to be performed by repr(). You can get essentially the same code using an f-string instead:

>>>

>>> s = 'foo'
>>> f'{s!r}'
"'foo'"

All the <format_spec> components that work with .format() also work with f-strings:

>>>

>>> n = 123
>>> '{:=+8}'.format(n)
'+    123'
>>> f'{n:=+8}'
'+    123'

>>> s = 'foo'
>>> '{0:*^8}'.format(s)
'**foo***'
>>> f'{s:*^8}'
'**foo***'

>>> n = 0b111010100001
>>> '{0:#_b}'.format(n)
'0b1110_1010_0001'
>>> f'{n:#_b}'
'0b1110_1010_0001'

Nesting works as well, like nested replacement fields with Python’s .format() method:

>>>

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
>>> w = 4
>>> f'{len(a):0{w}d}'
'0005'

>>> n = 123456789
>>> sep = '_'
>>> f'{(n * n):{sep}d}'
'15_241_578_750_190_521'

This means formatting items can be evaluated at run-time.

f-strings and Python’s .format() method are, more or less, two different ways of doing the same thing, with f-strings being a more concise shorthand. The following expressions are essentially the same:

f'{<expr>!<conversion>:<format_spec>}'

'{0!<conversion>:<format_spec>}'.format(<expr>)

If you want to explore f-strings further, then check out Python 3’s f-Strings: An Improved String Formatting Syntax (Course).

Conclusion

In this tutorial, you mastered two additional techniques that you can use in Python to format string data. You should now have all the tools you need to prepare string data for output or display!

You might be wondering which Python formatting technique you should use. Under what circumstances would you choose .format() over the f-string? See Python String Formatting Best Practices for some considerations to take into account.

In the next tutorial, you’re going to learn more about functions in Python. Throughout this tutorial series, you’ve seen many examples of Python’s built-in functions. In Python, as in most programming languages, you can define your own custom user-defined functions as well. If you can’t wait to learn how then continue on to the next tutorial!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Brett Slatkin

2020-02-12T14:00:00+00:00

Today I’m speaking to Brett Slatkin, a principal software engineer at Google and the author of the Python programming book Effective Python. Join us as we discuss Brett’s experience working with Python at Google, refactoring, and the challenges he faced when writing the second edition of his book. Without any further ado, let’s get into it!

Ricky: Welcome to Real Python, Brett. I’m glad you could join me for this interview. Let’s start the same way we do with all our guests. How’d you get into programming, and when did you start using Python?

Brett: Thanks for having me! My path to programming was long and varied. Python was a surprising twist at the end of my education. It was something that I never expected, but it was a blessing. Once I learned the language, I fell in love with it.

When I was growing up, my family had computers in our house, which was an enormous privilege. I always loved taking things apart to see how they worked, and I don’t remember a time when I wasn’t using computers and trying to figure them out. One of my late grandfather’s favorite stories was about the time he was having a printing problem and I fixed it for him. (I was 3 years old.)

In the 5th grade, my whole class went to a computer lab one day and programmed with Logo on an Apple IIgs. I’m lucky that my mom encouraged my interest in programming after that by buying me books and enrolling me in nerdy summer camps. I used Lego Technic Control Center kits to program small machines. These were the precursor to today’s Lego Mindstorms.

I also used Borland Delphi to build simple GUI apps, like guess the number games. Then, the staff at my local internet cafe showed me Unix, Linux, and C, which blew my mind. That’s when I knew I wanted to be a programmer when I grew up.

During high school, I took free classes at my local community college and studied C, C++, x86 assembly, and game development. I read Scott Meyers’ second edition of Effective C++ and became a huge fan of C++ and all of its arcane features. One summer, I had an unpaid internship with a dot-com bubble startup where I wrote web apps in ASP and VBScript. I was most excited about network programming and tried to build my own versions of DikuMUD and Hotline.

In college, I majored in computer engineering because I wanted to understand how computers worked at the transistor level. Most of my coursework was electrical engineering, where I used programming tools like SPICE, VHDL, and MATLAB. For computer science classes, I learned LISP and wrote a lot of Java. I was a teaching assistant for a variety of classes and really enjoyed teaching others.

Finally, my first encounter with Python was in 2003 when I was looking into how the BitTorrent client worked. (It was open source back then.) I thought the code was ugly because of all the __dunder__ special methods, and I ignored the language for all of my projects!

I went to Google in 2005 after graduating from college because it seemed like the best place to learn more about network programming. But in those earlier days of the company, you didn’t get to pick what you worked on as a new hire. You were just assigned to a team.

On my first day, my boss gave me a copy of the book Python in a Nutshell and told me to go fix a ~25KLOC codebase that was broken. I thought that in the job I would be writing C++ for networked systems, but I ended up having to do something completely different. What made matters worse was that I was on my own because the original programmer had gone on mandatory vacation due to burnout!

Luckily, Alex Martelli was a colleague on the team I joined. He was the author of the book Python in a Nutshell and he helped me learn the language. I was able to rewrite most of the codebase I inherited, fix its fundamental issues, and scale it to Google’s entire production fleet of machines. Believe it or not, this code is still in use today, nearly 15 years later.

The productivity I gained from using Python was dramatic compared to my experience with C++ and Java. My perspective on programming and languages had completely changed. I was now excited about Python!

Ricky: You’ve already touched on using Python at Google, where you’re now a principal software engineer, but in your tenure, you launched Google’s first cloud product, the App Engine. Now you’re the technical lead on Google Surveys, among other projects. How instrumental was Python in helping you develop these, and what does the future of Python look like at Google?

Brett: There’s no doubt that I owe my career at Google to Python, but there’s more to the story. As background, it’s important for you to know that the five engineers who founded App Engine were originally using JavaScript, similar to Netscape’s Livewire server from 1995. However, this was before the release of Chrome and the large performance improvements of V8, which led to NodeJS.

The founders were concerned that server-side JavaScript would make the system seem too niche. They wanted to provide the LAMP stack, which was very popular at the time. The “P” in LAMP stood for Perl, PHP, or Python. Perl was generally on its way out of style, and PHP wasn’t a language Google used internally. But Python was, thanks to Peter Norvig, and so it was the natural choice.

I started on the App Engine team with a 20% project in late 2006 when I implemented the dev_appserver in Python. Before that, there was no way to locally run an app before deploying it. I was so excited by the product’s potential and its use of Python that I moved over to work on it full-time shortly after.

Guido van Rossum, who also worked at Google then, soon joined us. I had the exciting privilege of doing the live demo during the launch event in 2008, where I wrote a web app in Python from scratch and deployed it in just a few minutes. I collaborated with Guido on a variety of projects over the years, the most interesting of which was ndb (the precursor to asyncio) that I beta tested extensively.

What I enjoyed the most was writing full applications with Python in the App Engine environment to push the product’s limits. I would frequently break the system with my demo apps by hitting new barriers that we didn’t realize we had. This would lead us to build infrastructure improvements to ensure that the next app that needed to scale, such as Snapchat, wouldn’t hit the same issues.

My favorite demo app was the PubSubHubbub protocol and hub that Brad Fitzpatrick and I developed together. The protocol turned RSS feeds into real-time streams. It brought the liveliness of services like Twitter to the open web.

At its height, our hub app was processing millions of feeds and thousands of requests per second, all using Python on App Engine. Since then, the hub has been integrated into Google’s standard web crawling infrastructure. PubSubHubbub (now called WebSub) became part of a larger group of specifications that made up OStatus, which helped get Mastadon off the ground.

Google Surveys started as another Python demo app on App Engine. I prototyped the whole system end-to-end in just a few weeks. My co-founder and I had the chance to show it off to Larry Page, and he greenlit the project. We launched a year later with a lean team, thanks to the leverage that Python provides.

Our services now handle 500K+ requests per second and impact the majority of Google’s users and revenue. The codebase has grown from ~10KLOC to over 1MLOC, with the majority of it being in Python. We’ve had to migrate some services over to Go, C++, and Java as we’ve hit various CPU cost and latency constraints, but Python is still at the core of everything we do.

Python has found its place at Google primarily as the tool for data science, machine learning, and DevOps / SRE. Right now, engineers across the company are working towards deprecating Python 2 and moving the whole monorepo over to Python 3 to stay in line with the Python community and open source packages.

Ricky: On top of what you just talked about, you’re also an author of the popular Python programming book, Effective Python, which we previously reviewed in The Best Python Books. You’ve recently published a second edition, which has substantial updates. What do the updates include, and what’s changed since the first edition?

Brett: Thanks for considering my book one of the best! The response to the first edition has been wonderful. It was even translated into eight languages! There’s a lot of info about the new book on the official website, including the full table of contents with all of the item titles, a variety of samples, and the example code.

The second edition of Effective Python is nearly twice the length of the original and significantly revises all of the items of advice, in addition to providing 30+ new ones. The first edition of the book was written back in 2014 with Python 2.7 and Python 3.4 in mind. So much has been added in Python 3.5 through Python 3.8 over the past 5 years, so there was a lot to write about! The general style is the same and it still includes multi-color syntax highlighting.

The new book covers the walrus operator, f-strings, best practices for dictionaries, class decorators, and all of the important features of functions (including asynchronous execution). I get into the details of unpacking generalizations, customized sorting, testing facilities, and algorithmic performance in the standard library. I also explain how to leverage more advanced tools such as the typing module and the memoryview type.

In this new edition, my advice about metaclasses has completely changed, thanks to the addition of __init_subclass__ and __set_name__ to the language. My recommendation for advanced generator usage with send() and throw() is now the opposite of what it was in the first edition. (Now I say to avoid them.) I compare the benefits of asyncio to threads and processes, which was entirely missing before. I provide guidance on how to convert synchronous code to asynchronous code, as well as how to mix asyncio with threads.

Anyone out there who reads the book, please feel free to send me questions or feedback!

Ricky: At PyCon 2016, you gave a talk titled Refactoring Python: Why and How to Restructure Your Code, which made our Top 10 Must-Watch PyCon Talks list, too. I’m curious to know your motivations for giving the talk. What was it about refactoring (or the lack thereof) that inspired you to give a talk about it? Do people seem to struggle with refactoring? Are they falling for the same common pitfalls?

Brett: Thanks for including my talk in your top ten list!

When I put this talk together, I had been working in the same codebase for about six years. Any code of that age needs to be refactored constantly and updated to prevent it from rotting and becoming unusable.

The rot I’m talking about happens when dependencies change in backward-incompatible ways, when the underlying infrastructure shifts to have different performance characteristics, or when product decisions break a large number of prior assumptions. So, refactoring was an ongoing necessity and it was on my mind frequently. This experience gave me a new appreciation for how crucial the skill of refactoring is.

People often talk about how testing, tooling, and frameworks make them more effective programmers. But I believe refactoring is the big underappreciated fundamental skill that every programmer needs to focus on improving. Why? In my view, the best code is code that doesn’t exist and has been deleted or was never written. The best program you can have is an empty .py file. The question is how do you get from where you are in a project to closer to that ideal? By refactoring.

I wanted to help the other programmers on my team learn how to become better at refactoring, but the book Refactoring by Martin Fowler hadn’t been updated since 1999. So I was considering writing a Python-specific version of the book to modernize it into something my colleagues could read. However, I put that on hold when I learned that Fowler was working on the second edition of Refactoring, which has since been released and uses JavaScript as the implementation language.

Unfortunately, it still falls short of what I need. I would love to see a Python-specific refactoring book, or a set of workable examples, that take advantage of everything the language has to offer.

Ricky: Now for my last few questions. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python and programming?

Brett: We have two very young children at home, so my wife and I spend a lot of time with them. Singing songs and playing music (piano primarily, but sometimes guitar, ukulele, xylophone, and harmonica) have been the best ways to enjoy our time together. I’m terrible at all of these instruments, and not a great singer either, but we still have a lot of nice times.

I like to be outside every day, either walking the hills of San Francisco or going for a run (usually while pushing a stroller). I love to surf when I get a chance. I enjoy reading, especially long-form journalism, but in reality, it’s mostly reddit and Lobsters. I’d like to be a better cook and find the right balance between laziness and deliciousness. I’m also trying to teach myself how to build statistical models using Bayesian methods. If I’ve learned anything over the years, it’s that nothing is for certain.

Thank you, Brett, for joining me this week! You can find Brett on Twitter or GitHub, and learn more about the second edition of Effective Python at effectivepython.com.

If there’s someone you’d like me to interview in the future, then reach out to me in the comments below, or send me a message on Twitter. Happy coding!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Playing and Recording Sound in Python

2020-02-11T14:00:00+00:00

In this course, you’ll learn how to play and record sound in Python using some of the most popular audio libraries. You’ll learn about the most straightforward methods for playing and recording sound first, and then you’ll learn about some libraries that offer some more functionality in exchange for a few extra lines of code.

By the end of this course, you’ll know how to:

Play MP3 and WAV files, as well as a range of other audio formats
Play NumPy and Python arrays containing sound
Record sound using Python
Save your recordings or audio files in a range of different file formats

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Implementing an Interface in Python

2020-02-10T14:00:00+00:00

Interfaces play an important role in software engineering. As an application grows, updates and changes to the code base become more difficult to manage. More often than not, you wind up having classes that look very similar but are unrelated, which can lead to some confusion. In this tutorial, you’ll see how you can use a Python interface to help determine what class you should use to tackle the current problem.

In this tutorial, you’ll be able to:

Understand how interfaces work and the caveats of Python interface creation
Comprehend how useful interfaces are in a dynamic language like Python
Implement an informal Python interface
Use abc.ABCMeta and @abc.abstractmethod to implement a formal Python interface

Interfaces in Python are handled differently than in most other languages, and they can vary in their design complexity. By the end of this tutorial, you’ll have a better understanding of some aspects of Python’s data model, as well as how interfaces in Python compare to those in languages like Java, C++, and Go.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Python Interface Overview

At a high level, an interface acts as a blueprint for designing classes. Like classes, interfaces define methods. Unlike classes, these methods are abstract. An abstract method is one that the interface simply defines. It doesn’t implement the methods. This is done by classes, which then implement the interface and give concrete meaning to the interface’s abstract methods.

Python’s approach to interface design is somewhat different when compared to languages like Java, Go, and C++. These languages all have an interface keyword, while Python does not. Python further deviates from other languages in one other aspect. It doesn’t require the class that’s implementing the interface to define all of the interface’s abstract methods.

Informal Interfaces

In certain circumstances, you may not need the strict rules of a formal Python interface. Python’s dynamic nature allows you to implement an informal interface. An informal Python interface is a class that defines methods that can be overridden, but there’s no strict enforcement.

In the following example, you’ll take the perspective of a data engineer who needs to extract text from various different unstructured file types, like PDFs and emails. You’ll create an informal interface that defines the methods that will be in both the PdfParser and EmlParser concrete classes:

class InformalParserInterface:
    def load_data_source(self, path: str, file_name: str) -> str:
        """Load in the file for extracting text."""
        pass

    def extract_text(self, full_file_name: str) -> dict:
        """Extract text from the currently loaded file."""
        pass

InformalParserInterface defines the two methods .load_data_source() and .extract_text(). These methods are defined but not implemented. The implementation will occur once you create concrete classes that inherit from InformalParserInterface.

As you can see, InformalParserInterface looks identical to a standard Python class. You rely on duck typing to inform users that this is an interface and should be used accordingly.

Note: Haven’t heard of duck typing? This term says that if you have an object that looks like a duck, walks like a duck, and quacks like a duck, then it must be a duck! To learn more, check out Duck Typing.

With duck typing in mind, you define two classes that implement the InformalParserInterface. To use your interface, you must create a concrete class. A concrete class is a subclass of the interface that provides an implementation of the interface’s methods. You’ll create two concrete classes to implement your interface. The first is PdfParser, which you’ll use to parse the text from PDF files:

class PdfParser(InformalParserInterface):
    """Extract text from a PDF"""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides InformalParserInterface.load_data_source()"""
        pass

    def extract_text(self, full_file_path: str) -> dict:
        """Overrides InformalParserInterface.extract_text()"""
        pass

The concrete implementation of InformalParserInterface now allows you to extract text from PDF files.

The second concrete class is EmlParser, which you’ll use to parse the text from emails:

class EmlParser(InformalParserInterface):
    """Extract text from an email"""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides InformalParserInterface.load_data_source()"""
        pass

    def extract_text_from_email(self, full_file_path: str) -> dict:
        """A method defined only in EmlParser.
        Does not override InformalParserInterface.extract_text()
        """
        pass

The concrete implementation of InformalParserInterface now allows you to extract text from email files.

So far, you’ve defined two concrete implementations of the InformalPythonInterface. However, note that EmlParser fails to properly define .extract_text(). If you were to check whether EmlParser implements InformalParserInterface, then you’d get the following result:

>>>

>>> # Check if both PdfParser and EmlParser implement InformalParserInterface
>>> issubclass(PdfParser, InformalParserInterface)
True

>>> issubclass(EmlParser, InformalParserInterface)
True

This would return True, which poses a bit of a problem since it violates the definition of an interface!

Now check the method resolution order (MRO) of PdfParser and EmlParser. This tells you the superclasses of the class in question, as well as the order in which they’re searched for executing a method. You can view a class’s MRO by using the dunder method cls.__mro__:

>>>

>>> PdfParser.__mro__
(__main__.PdfParser, __main__.InformalParserInterface, object)

>>> EmlParser.__mro__
(__main__.EmlParser, __main__.InformalParserInterface, object)

Such informal interfaces are fine for small projects where only a few developers are working on the source code. However, as projects get larger and teams grow, this could lead to developers spending countless hours looking for hard-to-find logic errors in the codebase!

Using Metaclasses

Ideally, you would want issubclass(EmlParser, InformalParserInterface to return False when the implementing class doesn’t define all of the interface’s abstract methods. To do this, you’ll create a metaclass called ParserMeta. You’ll be overriding two dunder methods:

.__instancecheck__()
.__subclasscheck__()

In the code block below, you create a class called UpdatedInformalParserInterface that builds from the ParserMeta metaclass:

class ParserMeta(type):
    """A Parser metaclass that will be used for parser class creation.
    """
    def __instancecheck__(cls, instance):
        return cls.__subclasscheck__(type(instance))

    def __subclasscheck__(cls, subclass):
        return (hasattr(subclass, 'load_data_source') and 
                callable(subclass.load_data_source) and 
                hasattr(subclass, 'extract_text') and 
                callable(subclass.extract_text))

class UpdatedInformalParserInterface(metaclass=ParserMeta):
    """This interface is used for concrete classes to inherit from.
    There is no need to define the ParserMeta methods as any class
    as they are implicitly made available via .__subclasscheck__().
    """
    pass

Now that ParserMeta and UpdatedInformalParserInterface have been created, you can create your concrete implementations.

First, create a new class for parsing PDFs called PdfParserNew:

class PdfParserNew:
    """Extract text from a PDF."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides UpdatedInformalParserInterface.load_data_source()"""
        pass

    def extract_text(self, full_file_path: str) -> dict:
        """Overrides UpdatedInformalParserInterface.extract_text()"""
        pass

Here, PdfParserNew overrides .load_data_source() and .extract_text(), so issubclass(PdfParserNew, UpdatedInformalParserInterface) should return True.

In this next code block, you have a new implementation of the email parser called EmlParserNew:

class EmlParserNew:
    """Extract text from an email."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides UpdatedInformalParserInterface.load_data_source()"""
        pass

    def extract_text_from_email(self, full_file_path: str) -> dict:
        """A method defined only in EmlParser.
        Does not override UpdatedInformalParserInterface.extract_text()
        """
        pass

Here, you have a metaclass that’s used to create UpdatedInformalParserInterface. By using a metaclass, you don’t need to explicitly define the subclasses. Instead, the subclass must define the required methods. If it doesn’t, then issubclass(EmlParserNew, UpdatedInformalParserInterface) will return False.

Running issubclass() on your concrete classes will produce the following:

>>>

>>> issubclass(PdfParserNew, UpdatedInformalParserInterface)
True

>>> issubclass(EmlParserNew, UpdatedInformalParserInterface)
False

As expected, EmlParserNew is not a subclass of UpdatedInformalParserInterface since .extract_text() wasn’t defined in EmlParserNew.

Now, let’s have a look at the MRO:

>>>

>>> PdfParserNew.__mro__
(<class '__main__.PdfParserNew'>, <class 'object'>)

As you can see, UpdatedInformalParserInterface is a superclass of PdfParserNew, but it doesn’t appear in the MRO. This unusual behavior is caused by the fact that UpdatedInformalParserInterface is a virtual base class of PdfParserNew.

Using Virtual Base Classes

In the previous example, issubclass(EmlParserNew, UpdatedInformalParserInterface) returned True, even though UpdatedInformalParserInterface did not appear in the EmlParserNew MRO. That’s because UpdatedInformalParserInterface is a virtual base class of EmlParserNew.

The key difference between these and standard subclasses is that virtual base classes use the .__subclasscheck__() dunder method to implicitly check if a class is a virtual subclass of the superclass. Additionally, virtual base classes don’t appear in the subclass MRO.

Take a look at this code block:

class PersonMeta(type):
    """A person metaclass"""
    def __instancecheck__(cls, instance):
        return cls.__subclasscheck__(type(instance))

    def __subclasscheck__(cls, subclass):
        return (hasattr(subclass, 'name') and 
                callable(subclass.name) and 
                hasattr(subclass, 'age') and 
                callable(subclass.age))

class PersonSuper:
    """A person superclass"""
    def name(self) -> str:
        pass

    def age(self) -> int:
        pass

class Person(metaclass=PersonMeta):
    """Person interface built from PersonMeta metaclass."""
    pass

Here, you have the setup for creating your virtual base classes:

The metaclass PersonMeta
The base class PersonSuper
The Python interface Person

Now that the setup for creating virtual base classes is done you’ll define two concrete classes, Employee and Friend. The Employee class inherits from PersonSuper, while Friend implicitly inherits from Person:

# Inheriting subclasses
class Employee(PersonSuper):
    """Inherits from PersonSuper
    PersonSuper will appear in Employee.__mro__
    """
    pass

class Friend:
    """Built implicitly from Person
    Friend is a virtual subclass of Person since
    both required methods exist.
    Person not in Friend.__mro__
    """
    def name(self):
        pass

    def age(self):
        pass

Although Friend does not explicitly inherit from Person, it implements .name() and .age(), so Person becomes a virtual base class of Friend. When you run issubclass(Friend, Person) it should return True, meaning that Friend is a subclass of Person.

The following UML diagram shows what happens when you call issubclass() on the Friend class:

Taking a look at PersonMeta, you’ll notice that there’s another dunder method called .__instancecheck__(). This method is used to check if instances of Friend are created from the Person interface. Your code will call .__instancecheck__() when you use isinstance(Friend, Person).

Formal Interfaces

Informal interfaces can be useful for projects with a small code base and a limited number of programmers. However, informal interfaces would be the wrong approach for larger applications. In order to create a formal Python interface, you’ll need a few more tools from Python’s abc module.

Using `abc.ABCMeta`

To enforce the subclass instantiation of abstract methods, you’ll utilize Python’s builtin ABCMeta from the abc module. Going back to your UpdatedInformalParserInterface interface, you created your own metaclass, ParserMeta, with the overridden dunder methods .__instancecheck__() and .__subclasscheck__().

Rather than create your own metaclass, you’ll use abc.ABCMeta as the metaclass. Then, you’ll overwrite .__subclasshook__() in place of .__instancecheck__() and .__subclasscheck__(), as it creates a more reliable implementation of these dunder methods.

Using `.subclasshook()`

Here’s the implementation of FormalParserInterface using abc.ABCMeta as your metaclass:

import abc

class FormalParserInterface(metaclass=abc.ABCMeta):
    @classmethod
    def __subclasshook__(cls, subclass):
        return (hasattr(subclass, 'load_data_source') and 
                callable(subclass.load_data_source) and 
                hasattr(subclass, 'extract_text') and 
                callable(subclass.extract_text))

class PdfParserNew:
    """Extract text from a PDF."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text(self, full_file_path: str) -> dict:
        """Overrides FormalParserInterface.extract_text()"""
        pass

class EmlParserNew:
    """Extract text from an email."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text_from_email(self, full_file_path: str) -> dict:
        """A method defined only in EmlParser.
        Does not override FormalParserInterface.extract_text()
        """
        pass

If you run issubclass() on PdfParserNew and EmlParserNew, then issubclass() will return True and False, respectively.

Using `abc` to Register a Virtual Subclass

Once you’ve imported the abc module, you can directly register a virtual subclass by using the .register() metamethod. In the next example, you register the interface Double as a virtual base class of the built-in __float__ class:

class Double(metaclass=abc.ABCMeta):
    """Double precision floating point number."""
    pass

Double.register(float)

You can check out the effect of using .register():

>>>

>>> issubclass(float, Double)
True

>>> isinstance(1.2345, Double)
True

By using the .register() meta method, you’ve successfully registered Double as a virtual subclass of float.

Once you’ve registered Double, you can use it as class decorator to set the decorated class as a virtual subclass:

@Double.register
class Double64:
    """A 64-bit double-precision floating-point number."""
    pass

print(issubclass(Double64, Double))  # True

The decorator register method helps you to create a hierarchy of custom virtual class inheritance.

Using Subclass Detection With Registration

You must be careful when you’re combining .__subclasshook__() with .register(), as .__subclasshook__() takes precedence over virtual subclass registration. To ensure that the registered virtual subclasses are taken into consideration, you must add NotImplemented to the .__subclasshook__() dunder method. The FormalParserInterface would be updated to the following:

class FormalParserInterface(metaclass=abc.ABCMeta):
    @classmethod
    def __subclasshook__(cls, subclass):
        return (hasattr(subclass, 'load_data_source') and 
                callable(subclass.load_data_source) and 
                hasattr(subclass, 'extract_text') and 
                callable(subclass.extract_text) or 
                NotImplemented)

class PdfParserNew:
    """Extract text from a PDF."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text(self, full_file_path: str) -> dict:
        """Overrides FormalParserInterface.extract_text()"""
        pass

@FormalParserInterface.register
class EmlParserNew:
    """Extract text from an email."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text_from_email(self, full_file_path: str) -> dict:
        """A method defined only in EmlParser.
        Does not override FormalParserInterface.extract_text()
        """
        pass

print(issubclass(PdfParserNew, FormalParserInterface))  # True
print(issubclass(EmlParserNew, FormalParserInterface))  # True

Since you’ve used registration, you can see that EmlParserNew is considered a virtual subclass of your FormalParserInterface interface. This is not what you wanted since EmlParserNew doesn’t override .extract_text(). Please use caution with virtual subclass registration!

Using Abstract Method Declaration

An abstract method is a method that’s declared by the Python interface, but it may not have a useful implementation. The abstract method must be overridden by the concrete class that implements the interface in question.

To create abstract methods in Python, you add the @abc.abstractmethod decorator to the interface’s methods. In the next example, you update the FormalParserInterface to include the abstract methods .load_data_source() and .extract_text():

class FormalParserInterface(metaclass=abc.ABCMeta):
    @classmethod
    def __subclasshook__(cls, subclass):
        return (hasattr(subclass, 'load_data_source') and 
                callable(subclass.load_data_source) and 
                hasattr(subclass, 'extract_text') and 
                callable(subclass.extract_text) or 
                NotImplemented)

    @abc.abstractmethod
    def load_data_source(self, path: str, file_name: str):
        """Load in the data set"""
        raise NotImplementedError

    @abc.abstractmethod
    def extract_text(self, full_file_path: str):
        """Extract text from the data set"""
        raise NotImplementedError

class PdfParserNew(FormalParserInterface):
    """Extract text from a PDF."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text(self, full_file_path: str) -> dict:
        """Overrides FormalParserInterface.extract_text()"""
        pass

class EmlParserNew(FormalParserInterface):
    """Extract text from an email."""
    def load_data_source(self, path: str, file_name: str) -> str:
        """Overrides FormalParserInterface.load_data_source()"""
        pass

    def extract_text_from_email(self, full_file_path: str) -> dict:
        """A method defined only in EmlParser.
        Does not override FormalParserInterface.extract_text()
        """
        pass

In the above example, you’ve finally created a formal interface that will raise errors when the abstract methods aren’t overridden. The PdfParserNew instance, pdf_parser, won’t raise any errors, as PdfParserNew is correctly overriding the FormalParserInterface abstract methods. However, EmlParserNew will raise an error:

>>>

>>> pdf_parser = PdfParserNew()
>>> eml_parser = EmlParserNew()
Traceback (most recent call last):
  File "real_python_interfaces.py", line 53, in <module>
    eml_interface = EmlParserNew()
TypeError: Can't instantiate abstract class EmlParserNew with abstract methods extract_text

As you can see, the traceback message tells you that you haven’t overridden all the abstract methods. This is the behavior you expect when building a formal Python interface.

Interfaces in Other Languages

Interfaces appear in many programming languages, and their implementation varies greatly from language to language. In the next few sections, you’ll compare interfaces in Python to Java, C++, and Go.

Java

Unlike Python, Java contains an interface keyword. Keeping with the file parser example, you declare an interface in Java like so:

public interface FileParserInterface {
    // Static fields, and abstract methods go here ...
    public void loadDataSource();
    public void extractText();
}

Now you’ll create two concrete classes, PdfParser and EmlParser, to implement the FileParserInterface. To do so, you must use the implements keyword in the class definition, like so:

public class EmlParser implements FileParserInterface {
     public void loadDataSource() {
            // Code to load the data set
        }

     public void extractText() {
            // Code to extract the text
        }
}

Continuing with your file parsing example, a fully-functional Java interface would look something like this:

import java.util.*;
import java.io.*;

public class FileParser {
    public static void main(String[] args) throws IOException {
        // The main entry point
    }

    public interface FileParserInterface {
        HashMap<String, ArrayList<String>> file_contents = null;

        public void loadDataSource();
        public void extractText();
    }

    public class PdfParser implements FileParserInterface {
        public void loadDataSource() {
            // Code to load the data set
        }

        public void extractText() {
            // Code to extract the text
        }

    }

    public class EmlParser implements FileParserInterface {
        public void loadDataSource() {
            // Code to load the data set
        }

        public void extractText() {
            // Code to extract the text
        }
    }

}

As you can see, a Python interface gives you much more flexibility during creation than a Java interface does.

C++

Like Python, C++ uses abstract base classes to create interfaces. When defining an interface in C++, you use the keyword virtual to describe a method that should be overwritten in the concrete class:

class FileParserInterface {

    public:
        virtual void loadDataSource(std::string path, std::string file_name);
        virtual void extractText(std::string full_file_name);
};

When you want to implement the interface, you’ll give the concrete class name, followed by a colon (:), and then the name of the interface. The following example demonstrates C++ interface implementation:

class PdfParser : FileParserInterface {
    public:
        void loadDataSource(std::string path, std::string file_name);
        void extractText(std::string full_file_name);
};

class EmlParser : FileParserInterface {
    public:
        void loadDataSource(std::string path, std::string file_name);
        void extractText(std::string full_file_name);
};

A Python interface and a C++ interface have some similarities in that they both make use of abstract base classes to simulate interfaces.

Go

Although Go’s syntax is reminiscent of Python, the Go programming language contains an interface keyword, like Java. Let’s create the fileParserInterface in Go:

type fileParserInterface interface {
        loadDataSet(path string, filename string)
        extractText(full_file_path string)
}

A big difference between Python and Go is that Go doesn’t have classes. Rather, Go is similar to C in that it uses the struct keyword to create structures. A structure is similar to a class in that a structure contains data and methods. However, unlike a class, all of the data and methods are publicly accessed. The concrete structs in Go will be used to implement the fileParserInterface.

Here’s an example of how Go uses interfaces:

package main

type fileParserInterface interface {
        loadDataSet(path string, filename string)
        extractText(full_file_path string)
}

type pdfParser struct {
        // Data goes here ...
}

type emlParser struct {
        // Data goes here ...
}

func (p pdfParser) loadDataSet() {
        // Method definition ...
}

func (p pdfParser) extractText() {
        // Method definition ...
}

func (e emlParser) loadDataSet() {
        // Method definition ...
}

func (e emlParser) extractText() {
        // Method definition ...
}

func main() {
        // Main entrypoint
}

Unlike a Python interface, a Go interface is created using structs and the explicit keyword interface.

Conclusion

Python offers great flexibility when you’re creating interfaces. An informal Python interface is useful for small projects where you’re less likely to get confused as to what the return types of the methods are. As a project grows, the need for a formal Python interface becomes more important as it becomes more difficult to infer return types. This ensures that the concrete class, which implements the interface, overwrites the abstract methods.

Now you can:

Understand how interfaces work and the caveats of creating a Python interface
Understand the usefulness of interfaces in a dynamic language like Python
Implement formal and informal interfaces in Python
Compare Python interfaces to those in languages like Java, C++, and Go

Now that you’ve become familiar with how to create a Python interface, add a Python interface to your next project to see its usefulness in action!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Command Line Arguments

2020-02-05T14:00:00+00:00

Adding the capability of processing Python command line arguments provides a user-friendly interface to your text-based command line program. It’s similar to what a graphical user interface is for a visual application that’s manipulated by graphical elements or widgets.

Python exposes a mechanism to capture and extract your Python command line arguments. These values can be used to modify the behavior of a program. For example, if your program processes data read from a file, then you can pass the name of the file to your program, rather than hard-coding the value in your source code.

By the end of this tutorial, you’ll know:

The origins of Python command line arguments
The underlying support for Python command line arguments
The standards guiding the design of a command line interface
The basics to manually customize and handle Python command line arguments
The libraries available in Python to ease the development of a complex command line interface

If you want a user-friendly way to supply Python command line arguments to your program without importing a dedicated library, or if you want to better understand the common basis for the existing libraries that are dedicated to building the Python command line interface, then keep on reading!

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

The Command Line Interface

A command line interface (CLI) provides a way for a user to interact with a program running in a text-based shell interpreter. Some examples of shell interpreters are Bash on Linux or Command Prompt on Windows. A command line interface is enabled by the shell interpreter that exposes a command prompt. It can be characterized by the following elements:

A command or program
Zero or more command line arguments
An output representing the result of the command
Textual documentation referred to as usage or help

Not every command line interface may provide all these elements, but this list isn’t exhaustive, either. The complexity of the command line ranges from the ability to pass a single argument, to numerous arguments and options, much like a Domain Specific Language. For example, some programs may launch web documentation from the command line or start an interactive shell interpreter like Python.

The two following examples with the Python command illustrates the description of a command line interface:

$ python -c "print('Real Python')"
Real Python

In this first example, the Python interpreter takes option -c for command, which says to execute the Python command line arguments following the option -c as a Python program.

Another example shows how to invoke Python with -h to display the help:

$ python -h
usage: python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Options and arguments (and corresponding environment variables):
-b     : issue warnings about str(bytes_instance), str(bytearray_instance)
         and comparing bytes/bytearray with str. (-bb: issue errors)
[ ... complete help text not shown ... ]

Try this out in your terminal to see the complete help documentation.

The C Legacy

Python command line arguments directly inherit from the C programming language. As Guido Van Rossum wrote in An Introduction to Python for Unix/C Programmers in 1993, C had a strong influence on Python. Guido mentions the definitions of literals, identifiers, operators, and statements like break, continue, or return. The use of Python command line arguments is also strongly influenced by the C language.

To illustrate the similarities, consider the following C program:

 1 // main.c
 2 #include <stdio.h>
 3 
 4 int main(int argc, char *argv[]) {
 5     printf("Arguments count: %d\n", argc);
 6     for (int i = 0; i < argc; i++) {
 7         printf("Argument %6d: %s\n", i, argv[i]);
 8     }
 9     return 0;
10 }

Line 4 defines main(), which is the entry point of a C program. Take good note of the parameters:

argc is an integer representing the number of arguments of the program.
argv is an array of pointers to characters containing the name of the program in the first element of the array, followed by the arguments of the program, if any, in the remaining elements of the array.

You can compile the code above on Linux with gcc -o main main.c, then execute with ./main to obtain the following:

$ gcc -o main main.c
$ ./main
Arguments count: 1
Argument      0: ./main

Unless explicitly expressed at the command line with the option -o, a.out is the default name of the executable generated by the gcc compiler. It stands for assembler output and is reminiscent of the executables that were generated on older UNIX systems. Observe that the name of the executable ./main is the sole argument.

Let’s spice up this example by passing a few Python command line arguments to the same program:

$ ./main Python Command Line Arguments
Arguments count: 5
Argument      0: ./main
Argument      1: Python
Argument      2: Command
Argument      3: Line
Argument      4: Arguments

The output shows that the number of arguments is 5, and the list of arguments includes the name of the program, main, followed by each word of the phrase "Python Command Line Arguments", which you passed at the command line.

Note: argc stands for argument count, while argv stands for argument vector. To learn more, check out A Little C Primer/C Command Line Arguments.

The compilation of main.c assumes that you used a Linux or a Mac OS system. On Windows, you can also compile this C program with one of the following options:

Windows Subsystem for Linux (WSL): It’s available in a few Linux distributions, like Ubuntu, OpenSUSE, and Debian, among others. You can install it from the Microsoft Store.
Windows Build Tools: This includes the Windows command line build tools, the Microsoft C/C++ compiler cl.exe, and a compiler front end named clang.exe for C/C++.
Microsoft Visual Studio: This is the main Microsoft integrated development environment (IDE). To learn more about IDEs that can be used for both Python and C on various operating systems, including Windows, check out Python IDEs and Code Editors (Guide).
mingw-64 project: This supports the GCC compiler on Windows.

If you’ve installed Microsoft Visual Studio or the Windows Build Tools, then you can compile main.c as follows:

C:/>cl main.c

You’ll obtain an executable named main.exe that you can start with:

C:/>main
Arguments count: 1
Argument      0: main

You could implement a Python program, main.py, that’s equivalent to the C program, main.c, you saw above:

# main.py
import sys

if __name__ == "__main__":
    print(f"Arguments count: {len(sys.argv)}")
    for i, arg in enumerate(sys.argv):
        print(f"Argument {i:>6}: {arg}")

You don’t see an argc variable like in the C code example. It doesn’t exist in Python because sys.argv is sufficient. You can parse the Python command line arguments in sys.argv without having to know the length of the list, and you can call the built-in len() if the number of arguments is needed by your program.

Also, note that enumerate(), when applied to an iterable, returns an enumerate object that can emit pairs associating the index of an element in sys.arg to its corresponding value. This allows looping through the content of sys.argv without having to maintain a counter for the index in the list.

Execute main.py as follows:

$ python main.py Python Command Line Arguments
Arguments count: 5
Argument      0: main.py
Argument      1: Python
Argument      2: Command
Argument      3: Line
Argument      4: Arguments

sys.argv contains the same information as in the C program:

The name of the program main.py is the first item of the list.
The arguments Python, Command, Line, and Arguments are the remaining elements in the list.

With this short introduction into a few arcane aspects of the C language, you’re now armed with some valuable knowledge to further grasp Python command line arguments.

Two Utilities From the Unix World

To use Python command line arguments in this tutorial, you’ll implement some partial features of two utilities from the Unix ecosystem:

sha1sum
seq

You’ll gain some familiarity with these Unix tools in the following sections.

`sha1sum`

sha1sum calculates SHA-1 hashes, and it’s often used to verify the integrity of files. For a given input, a hash function always returns the same value. Any minor changes in the input will result in a different hash value. Before you use the utility with concrete parameters, you may try to display the help:

$ sha1sum --help
Usage: sha1sum [OPTION]... [FILE]...
Print or check SHA1 (160-bit) checksums.

With no FILE, or when FILE is -, read standard input.

  -b, --binary         read in binary mode
  -c, --check          read SHA1 sums from the FILEs and check them
      --tag            create a BSD-style checksum
  -t, --text           read in text mode (default)
  -z, --zero           end each output line with NUL, not newline,
                       and disable file name escaping
[ ... complete help text not shown ... ]

Displaying the help of a command line program is a common feature exposed in the command line interface.

To calculate the SHA-1 hash value of the content of a file, you proceed as follows:

$ sha1sum main.c
125a0f900ff6f164752600550879cbfabb098bc3  main.c

The result shows the SHA-1 hash value as the first field and the name of the file as the second field. The command can take more than one file as arguments:

$ sha1sum main.c main.py
125a0f900ff6f164752600550879cbfabb098bc3  main.c
d84372fc77a90336b6bb7c5e959bcb1b24c608b4  main.py

Thanks to the wildcards expansion feature of the Unix terminal, it’s also possible to provide Python command line arguments with wildcard characters. One such a character is the asterisk or star (*):

$ sha1sum main.*
3f6d5274d6317d580e2ffc1bf52beee0d94bf078  main.c
f41259ea5835446536d2e71e566075c1c1bfc111  main.py

The shell converts main.* to main.c and main.py, which are the two files matching the pattern main.* in the current directory, and passes them to sha1sum. The program calculates the SHA1 hash of each of the files in the argument list. You’ll see that, on Windows, the behavior is different. Windows has no wildcard expansion, so the program may have to accommodate for that. Your implementation may need to expand wildcards internally.

Without any argument, sha1sum reads from the standard input. You can feed data to the program by typing characters on the keyboard. The input may incorporate any characters, including the carriage return Enter. To terminate the input, you must signal the end of file with Enter, followed by the sequence Ctrl+D:

 1 $ sha1sum
 2 Real
 3 Python
 4 87263a73c98af453d68ee4aab61576b331f8d9d6  -

You first enter the name of the program, sha1sum, followed by Enter, and then Real and Python, each also followed by Enter. To close the input stream, you type Ctrl+D. The result is the value of the SHA1 hash generated for the text Real\nPython\n. The name of the file is -. This is a convention to indicate the standard input. The hash value is the same when you execute the following commands:

$ python -c "print('Real\nPython\n', end='')" | sha1sum
87263a73c98af453d68ee4aab61576b331f8d9d6  -
$ python -c "print('Real\nPython')" | sha1sum
87263a73c98af453d68ee4aab61576b331f8d9d6  -
$ printf "Real\nPython\n" | sha1sum
87263a73c98af453d68ee4aab61576b331f8d9d6  -

Up next, you’ll read a short description of seq.

`seq`

seq generates a sequence of numbers. In its most basic form, like generating the sequence from 1 to 5, you can execute the following:

$ seq 5
1
2
3
4
5

To get an overview of the possibilities exposed by seq, you can display the help at the command line:

$ seq --help
Usage: seq [OPTION]... LAST
  or:  seq [OPTION]... FIRST LAST
  or:  seq [OPTION]... FIRST INCREMENT LAST
Print numbers from FIRST to LAST, in steps of INCREMENT.

Mandatory arguments to long options are mandatory for short options too.
  -f, --format=FORMAT      use printf style floating-point FORMAT
  -s, --separator=STRING   use STRING to separate numbers (default: \n)
  -w, --equal-width        equalize width by padding with leading zeroes
      --help     display this help and exit
      --version  output version information and exit
[ ... complete help text not shown ... ]

For this tutorial, you’ll write a few simplified variants of sha1sum and seq. In each example, you’ll learn a different facet or combination of features about Python command line arguments.

On Mac OS and Linux, sha1sum and seq should come pre-installed, though the features and the help information may sometimes differ slightly between systems or distributions. If you’re using Windows 10, then the most convenient method is to run sha1sum and seq in a Linux environment installed on the WSL. If you don’t have access to a terminal exposing the standard Unix utilities, then you may have access to online terminals:

Create a free account on PythonAnywhere and start a Bash Console.
Create a temporary Bash terminal on repl.it.

These are two examples, and you may find others.

The `sys.argv` Array

Before exploring some accepted conventions and discovering how to handle Python command line arguments, you need to know that the underlying support for all Python command line arguments is provided by sys.argv. The examples in the following sections show you how to handle the Python command line arguments stored in sys.argv and to overcome typical issues that occur when you try to access them. You’ll learn:

How to access the content of sys.argv
How to mitigate the side effects of the global nature of sys.argv
How to process whitespaces in Python command line arguments
How to handle errors while accessing Python command line arguments
How to ingest the original format of the Python command line arguments passed by bytes

Let’s get started!

Displaying Arguments

The sys module exposes an array named argv that includes the following:

argv[0] contains the name of the current Python program.
argv[1:], the rest of the list, contains any and all Python command line arguments passed to the program.

The following example demonstrates the content of sys.argv:

 1 # argv.py
 2 import sys
 3 
 4 print(f"Name of the script      : {sys.argv[0]=}")
 5 print(f"Arguments of the script : {sys.argv[1:]=}")

Here’s how this code works:

Line 2 imports the internal Python module sys.
Line 4 extracts the name of the program by accessing the first element of the list sys.argv.
Line 5 displays the Python command line arguments by fetching all the remaining elements of the list sys.argv.

Note: The f-string syntax used in argv.py leverages the new debugging specifier in Python 3.8. To read more about this new f-string feature and others, check out Cool New Features in Python 3.8.

If your Python version is less than 3.8, then simply remove the equals sign (=) in both f-strings to allow the program to execute successfully. The output will only display the value of the variables, not their names.

Execute the script argv.py above with a list of arbitrary arguments as follows:

$ python argv.py un deux trois quatre
Name of the script      : sys.argv[0]='argv.py'
Arguments of the script : sys.argv[1:]=['un', 'deux', 'trois', 'quatre']

The output confirms that the content of sys.argv[0] is the Python script argv.py, and that the remaining elements of the sys.argv list contains the arguments of the script, ['un', 'deux', 'trois', 'quatre'].

To summarize, sys.argv contains all the argv.py Python command line arguments. When the Python interpreter executes a Python program, it parses the command line and populates sys.argv with the arguments.

Reversing the First Argument

Now that you have enough background on sys.argv, you’re going to operate on arguments passed at the command line. The example reverse.py reverses the first argument passed at the command line:

 1 # reverse.py
 2 
 3 import sys
 4 
 5 arg = sys.argv[1]
 6 print(arg[::-1])

In reverse.py the process to reverse the first argument is performed with the following steps:

Line 5 fetches the first argument of the program stored at index 1 of sys.argv. Remember that the program name is stored at index 0 of sys.argv.
Line 6 prints the reversed string. args[::-1] is a Pythonic way to use a slice operation to reverse a list.

You execute the script as follows:

$ python reverse.py "Real Python"
nohtyP laeR

As expected, reverse.py operates on "Real Python" and reverses the only argument to output "nohtyP laeR". Note that surrounding the multi-word string "Real Python" with quotes ensures that the interpreter handles it as a unique argument, instead of two arguments. You’ll delve into argument separators in a later section.

Mutating `sys.argv`

sys.argv is globally available to your running Python program. All modules imported during the execution of the process have direct access to sys.argv. This global access might be convenient, but sys.argv isn’t immutable. You may want to implement a more reliable mechanism to expose program arguments to different modules in your Python program, especially in a complex program with multiple files.

Observe what happens if you tamper with sys.argv:

# argv_pop.py

import sys

print(sys.argv)
sys.argv.pop()
print(sys.argv)

You invoke .pop() to remove and return the last item in sys.argv.

Execute the script above:

$ python argv_pop.py un deux trois quatre
['argv_pop.py', 'un', 'deux', 'trois', 'quatre']
['argv_pop.py', 'un', 'deux', 'trois']

Notice that the fourth argument is no longer included in sys.argv.

In a short script, you can safely rely on the global access to sys.argv, but in a larger program, you may want to store arguments in a separate variable. The previous example could be modified as follows:

# argv_var_pop.py

import sys

print(sys.argv)
args = sys.argv[1:]
print(args)
sys.argv.pop()
print(sys.argv)
print(args)

This time, although sys.argv lost its last element, args has been safely preserved. args isn’t global, and you can pass it around to parse the arguments per the logic of your program. The Python package manager, pip, uses this approach. Here’s a short excerpt of the pip source code:

def main(args=None):
    if args is None:
        args = sys.argv[1:]

In this snippet of code taken from the pip source code, main() saves into args the slice of sys.argv that contains only the arguments and not the file name. sys.argv remains untouched, and args isn’t impacted by any inadvertent changes to sys.argv.

Escaping Whitespace Characters

In the reverse.py example you saw earlier, the first and only argument is "Real Python", and the result is "nohtyP laeR". The argument includes a whitespace separator between "Real" and "Python", and it needs to be escaped.

On Linux, whitespaces can be escaped by doing one of the following:

Surrounding the arguments with single quotes (')
Surrounding the arguments with double quotes (")
Prefixing each space with a backslash (\)

Without one of the escape solutions, reverse.py stores two arguments, "Real" in sys.argv[1] and "Python" in sys.argv[2]:

$ python reverse.py Real Python
laeR

The output above shows that the script only reverses "Real" and that "Python" is ignored. To ensure both arguments are stored, you’d need to surround the overall string with double quotes (").

You can also use a backslash (\) to escape the whitespace:

$ python reverse.py Real\ Python
nohtyP laeR

With the backslash (\), the command shell exposes a unique argument to Python, and then to reverse.py.

In Unix shells, the internal field separator (IFS) defines characters used as delimiters. The content of the shell variable, IFS, can be displayed by running the following command:

$ printf "%q\n" "$IFS"
$' \t\n'

From the result above, ' \t\n', you identify three delimiters:

Space (' ')
Tab (\t)
Newline (\n)

Prefixing a space with a backslash (\) bypasses the default behavior of the space as a delimiter in the string "Real Python". This results in one block of text as intended, instead of two.

Note that, on Windows, the whitespace interpretation can be managed by using a combination of double quotes. It’s slightly counterintuitive because, in the Windows terminal, a double quote (") is interpreted as a switch to disable and subsequently to enable special characters like space, tab, or pipe (|).

As a result, when you surround more than one string with double quotes, the Windows terminal interprets the first double quote as a command to ignore special characters and the second double quote as one to interpret special characters.

With this information in mind, it’s safe to assume that surrounding more than one string with double quotes will give you the expected behavior, which is to expose the group of strings as a single argument. To confirm this peculiar effect of the double quote on the Windows command line, observe the following two examples:

C:/>python reverse.py "Real Python"
nohtyP laeR

In the example above, you can intuitively deduce that "Real Python" is interpreted as a single argument. However, realize what occurs when you use a single double quote:

C:/>python reverse.py "Real Python
nohtyP laeR

The command prompt passes the whole string "Real Python" as a single argument, in the same manner as if the argument was "Real Python". In reality, the Windows command prompt sees the unique double quote as a switch to disable the behavior of the whitespaces as separators and passes anything following the double quote as a unique argument.

For more information on the effects of double quotes in the Windows terminal, check out A Better Way To Understand Quoting and Escaping of Windows Command Line Arguments.

Handling Errors

Python command line arguments are loose strings. Many things can go wrong, so it’s a good idea to provide the users of your program with some guidance in the event they pass incorrect arguments at the command line. For example, reverse.py expects one argument, and if you omit it, then you get an error:

 1 $ python reverse.py
 2 Traceback (most recent call last):
 3   File "reverse.py", line 5, in <module>
 4     arg = sys.argv[1]
 5 IndexError: list index out of range

The Python exception IndexError is raised, and the corresponding traceback shows that the error is caused by the expression arg = sys.argv[1]. The message of the exception is list index out of range. You didn’t pass an argument at the command line, so there’s nothing in the list sys.argv at index 1.

This is a common pattern that can be addressed in a few different ways. For this initial example, you’ll keep it brief by including the expression arg = sys.argv[1] in a try block. Modify the code as follows:

 1 # reverse_exc.py
 2 
 3 import sys
 4 
 5 try:
 6     arg = sys.argv[1]
 7 except IndexError:
 8     raise SystemExit(f"Usage: {sys.argv[0]} <string_to_reverse>")
 9 print(arg[::-1])

The expression on line 4 is included in a try block. Line 8 raises the built-in exception SystemExit. If no argument is passed to reverse_exc.py, then the process exits with a status code of 1 after printing the usage. Note the integration of sys.argv[0] in the error message. It exposes the name of the program in the usage message. Now, when you execute the same program without any Python command line arguments, you can see the following output:

$ python reverse.py
Usage: reverse.py <string_to_reverse>

$ echo $?
1

reverse.py didn’t have an argument passed at the command line. As a result, the program raises SystemExit with an error message. This causes the program to exit with a status of 1, which displays when you print the special variable $? with echo.

Calculating the `sha1sum`

You’ll write another script to demonstrate that, on Unix-like systems, Python command line arguments are passed by bytes from the OS. This script takes a string as an argument and outputs the hexadecimal SHA-1 hash of the argument:

 1 # sha1sum.py
 2 
 3 import sys
 4 import hashlib
 5 
 6 data = sys.argv[1]
 7 m = hashlib.sha1()
 8 m.update(bytes(data, 'utf-8'))
 9 print(m.hexdigest())

This is loosely inspired by sha1sum, but it intentionally processes a string instead of the contents of a file. In sha1sum.py, the steps to ingest the Python command line arguments and to output the result are the following:

Line 6 stores the content of the first argument in data.
Line 7 instantiates a SHA1 algorithm.
Line 8 updates the SHA1 hash object with the content of the first program argument. Note that hash.update takes a byte array as an argument, so it’s necessary to convert data from a string to a bytes array.
Line 9 prints a hexadecimal representation of the SHA1 hash computed on line 8.

When you run the script with an argument, you get this:

$ python sha1sum.py "Real Python"
0554943d034f044c5998f55dac8ee2c03e387565

For the sake of keeping the example short, the script sha1sum.py doesn’t handle missing Python command line arguments. Error handling could be addressed in this script the same way you did it in reverse_exc.py.

Note: Checkout hashlib for more details about the hash functions available in the Python standard library.

From the sys.argv documentation, you learn that in order to get the original bytes of the Python command line arguments, you can use os.fsencode(). By directly obtaining the bytes from sys.argv[1], you don’t need to perform the string-to-bytes conversion of data:

 1 # sha1sum_bytes.py
 2 
 3 import os
 4 import sys
 5 import hashlib
 6 
 7 data = os.fsencode(sys.argv[1])
 8 m = hashlib.sha1()
 9 m.update(data)
10 print(m.hexdigest())

The main difference between sha1sum.py and sha1sum_bytes.py are highlighted in the following lines:

Line 7 populates data with the original bytes passed to the Python command line arguments.
Line 9 passes data as an argument to m.update(), which receives a bytes-like object.

Execute sha1sum_bytes.py to compare the output:

$ python sha1sum_bytes.py "Real Python"
0554943d034f044c5998f55dac8ee2c03e387565

The hexadecimal value of the SHA1 hash is the same as in the previous sha1sum.py example.

The Anatomy of Python Command Line Arguments

Now that you’ve explored a few aspects of Python command line arguments, most notably sys.argv, you’re going to apply some of the standards that are regularly used by developers while implementing a command line interface.

Python command line arguments are a subset of the command line interface. They can be composed of different types of arguments:

Options modify the behavior of a particular command or program.
Arguments represent the source or destination to be processed.
Subcommands allow a program to define more than one command with the respective set of options and arguments.

Before you go deeper into the different types of arguments, you’ll get an overview of the accepted standards that have been guiding the design of the command line interface and arguments. These have been refined since the advent of the computer terminal in the mid-1960s.

Standards

A few available standards provide some definitions and guidelines to promote consistency for implementing commands and their arguments. These are the main UNIX standards and references:

The standards above define guidelines and nomenclatures for anything related to programs and Python command line arguments. The following points are examples taken from those references:

POSIX:
- A program or utility is followed by options, option-arguments, and operands.
- All options should be preceded with a hyphen or minus (-) delimiter character.
- Option-arguments should not be optional.
GNU:
- All programs should support two standard options, which are --version and --help.
- Long-named options are equivalent to the single-letter Unix-style options. An example is --debug and -d.
docopt:
- Short options can be stacked, meaning that -abc is equivalent to -a -b -c.
- Long options can have arguments specified after a space or the equals sign (=). The long option --input=ARG is equivalent to --input ARG.

These standards define notations that are helpful when you describe a command. A similar notation can be used to display the usage of a particular command when you invoke it with the option -h or --help.

The GNU standards are very similar to the POSIX standards but provide some modifications and extensions. Notably, they add the long option that’s a fully named option prefixed with two hyphens (--). For example, to display the help, the regular option is -h and the long option is --help.

Note: You don’t need to follow those standards rigorously. Instead, follow the conventions that have been used successfully for years since the advent of UNIX. If you write a set of utilities for you or your team, then ensure that you stay consistent across the different utilities.

In the following sections, you’ll learn more about each of the command line components, options, arguments, and sub-commands.

Options

An option, sometimes called a flag or a switch, is intended to modify the behavior of the program. For example, the command ls on Linux lists the content of a given directory. Without any arguments, it lists the files and directories in the current directory:

$ cd /dev
$ ls
autofs
block
bsg
btrfs-control
bus
char
console

Let’s add a few options. You can combine -l and -s into -ls, which changes the information displayed in the terminal:

$ cd /dev
$ ls -ls
total 0
0 crw-r--r--  1 root root       10,   235 Jul 14 08:10 autofs
0 drwxr-xr-x  2 root root             260 Jul 14 08:10 block
0 drwxr-xr-x  2 root root              60 Jul 14 08:10 bsg
0 crw-------  1 root root       10,   234 Jul 14 08:10 btrfs-control
0 drwxr-xr-x  3 root root              60 Jul 14 08:10 bus
0 drwxr-xr-x  2 root root            4380 Jul 14 15:08 char
0 crw-------  1 root root        5,     1 Jul 14 08:10 console

An option can take an argument, which is called an option-argument. See an example in action with od below:

$ od -t x1z -N 16 main
0000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00  >.ELF............<
0000020

od stands for octal dump. This utility displays data in different printable representations, like octal (which is the default), hexadecimal, decimal, and ASCII. In the example above, it takes the binary file main and displays the first 16 bytes of the file in hexadecimal format. The option -t expects a type as an option-argument, and -N expects the number of input bytes.

In the example above, -t is given type x1, which stands for hexadecimal and one byte per integer. This is followed by z to display the printable characters at the end of the input line. -N takes 16 as an option-argument for limiting the number of input bytes to 16.

Arguments

The arguments are also called operands or parameters in the POSIX standards. The arguments represent the source or the destination of the data that the command acts on. For example, the command cp, which is used to copy one or more files to a file or a directory, takes at least one source and one target:

 1 $ ls main
 2 main
 3 
 4 $ cp main main2
 5 
 6 $ ls -lt
 7 main
 8 main2
 9 ...

In line 4, cp takes two arguments:

main: the source file
main2: the target file

It then copies the content of main to a new file named main2. Both main and main2 are arguments, or operands, of the program cp.

Subcommands

The concept of subcommands isn’t documented in the POSIX or GNU standards, but it does appear in docopt. The standard Unix utilities are small tools adhering to the Unix philosophy. Unix programs are intended to be programs that do one thing and do it well. This means no subcommands are necessary.

By contrast, a new generation of programs, including git, go, docker, and gcloud, come with a slightly different paradigm that embraces subcommands. They’re not necessarily part of the Unix landscape as they span several operating systems, and they’re deployed with a full ecosystem that requires several commands.

Take git as an example. It handles several commands, each possibly with their own set of options, option-arguments, and arguments. The following examples apply to the git subcommand branch:

git branch displays the branches of the local git repository.
git branch custom_python creates a local branch custom_python in a local repository.
git branch -d custom_python deletes the local branch custom_python.
git branch --help displays the help for the git branch subcommand.

In the Python ecosystem, pip has the concept of subcommands, too. Some pip subcommands include list, install, freeze, or uninstall.

Windows

On Windows, the conventions regarding Python command line arguments are slightly different, in particular, those regarding command line options. To validate this difference, take tasklist, which is a native Windows executable that displays a list of the currently running processes. It’s similar to ps on Linux or macOS systems. Below is an example of how to execute tasklist in a command prompt on Windows:

C:/>tasklist /FI "IMAGENAME eq notepad.exe"

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
notepad.exe                  13104 Console                    6     13,548 K
notepad.exe                   6584 Console                    6     13,696 K

Note that the separator for an option is a forward slash (/) instead of a hyphen (-) like the conventions for Unix systems. For readability, there’s a space between the program name, taskslist, and the option /FI, but it’s just as correct to type taskslist/FI.

The particular example above executes tasklist with a filter to only show the Notepad processes currently running. You can see that the system has two running instances of the Notepad process. Although it’s not equivalent, this is similar to executing the following command in a terminal on a Unix-like system:

$ ps -ef | grep vi | grep -v grep
andre     2117     4  0 13:33 tty1     00:00:00 vi .gitignore
andre     2163  2134  0 13:34 tty3     00:00:00 vi main.c

The ps command above shows all the current running vi processes. The behavior is consistent with the Unix Philosophy, as the output of ps is transformed by two grep filters. The first grep command selects all the occurrences of vi, and the second grep filters out the occurrence of grep itself.

With the spread of Unix tools making their appearance in the Windows ecosystem, non-Windows-specific conventions are also accepted on Windows.

Visuals

At the start of a Python process, Python command line arguments are split into two categories:

Python options: These influence the execution of the Python interpreter. For example, adding option -O is a means to optimize the execution of a Python program by removing assert and __debug__ statements. There are other Python options available at the command line.
Python program and its arguments: Following the Python options (if there are any), you’ll find the Python program, which is a file name that usually has the extension .py, and its arguments. By convention, those can also be composed of options and arguments.

Take the following command that’s intended to execute the program main.py, which takes options and arguments. Note that, in this example, the Python interpreter also takes some options, which are -B and -v.

$ python -B -v main.py --verbose --debug un deux

In the command line above, the options are Python command line arguments and are organized as follows:

The option -B tells Python not to write .pyc files on the import of source modules. For more details about .pyc files, check out the section What Does a Compiler Do? in Your Guide to the CPython Source Code.
The option -v stands for verbose and tells Python to trace all import statements.
The arguments passed to main.py are fictitious and represent two long options (--verbose and --debug) and two arguments (un and deux).

This example of Python command line arguments can be illustrated graphically as follows:

Within the Python program main.py, you only have access to the Python command line arguments inserted by Python in sys.argv. The Python options may influence the behavior of the program but are not accessible in main.py.

A Few Methods for Parsing Python Command Line Arguments

Now you’re going to explore a few approaches to apprehend options, option-arguments, and operands. This is done by parsing Python command line arguments. In this section, you’ll see some concrete aspects of Python command line arguments and techniques to handle them. First, you’ll see an example that introduces a straight approach relying on list comprehensions to collect and separate options from arguments. Then you will:

Use regular expressions to extract elements of the command line
Learn how to handle files passed at the command line
Apprehend the standard input in a way that’s compatible with the Unix tools
Differentiate the regular output of the program from the errors
Implement a custom parser to read Python command line arguments

This will serve as a preparation for options involving modules in the standard libraries or from external libraries that you’ll learn about later in this tutorial.

For something uncomplicated, the following pattern, which doesn’t enforce ordering and doesn’t handle option-arguments, may be enough:

# cul.py

import sys

opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
args = [arg for arg in sys.argv[1:] if not arg.startswith("-")]

if "-c" in opts:
    print(" ".join(arg.capitalize() for arg in args))
elif "-u" in opts:
    print(" ".join(arg.upper() for arg in args))
elif "-l" in opts:
    print(" ".join(arg.lower() for arg in args))
else:
    raise SystemExit(f"Usage: {sys.argv[0]} (-c | -u | -l) <arguments>...")

The intent of the program above is to modify the case of the Python command line arguments. Three options are available:

-c to capitalize the arguments
-u to convert the arguments to uppercase
-l to convert the argument to lowercase

The code collects and separates the different argument types using list comprehensions:

Line 5 collects all the options by filtering on any Python command line arguments starting with a hyphen (-).
Line 6 assembles the program arguments by filtering out the options.

When you execute the Python program above with a set of options and arguments, you get the following output:

$ python cul.py -c un deux trois
Un Deux Trois

This approach might suffice in many situations, but it would fail in the following cases:

If the order is important, and in particular, if options should appear before the arguments
If support for option-arguments is needed
If some arguments are prefixed with a hyphen (-)

You can leverage other options before you resort to a library like argparse or click.

Regular Expressions

You can use a regular expression to enforce a certain order, specific options and option-arguments, or even the type of arguments. To illustrate the usage of a regular expression to parse Python command line arguments, you’ll implement a Python version of seq, which is a program that prints a sequence of numbers. Following the docopt conventions, a specification for seq.py could be this:

Print integers from <first> to <last>, in steps of <increment>.

Usage:
  python seq.py --help
  python seq.py [-s SEPARATOR] <last>
  python seq.py [-s SEPARATOR] <first> <last>
  python seq.py [-s SEPARATOR] <first> <increment> <last>

Mandatory arguments to long options are mandatory for short options too.
  -s, --separator=STRING use STRING to separate numbers (default: \n)
      --help             display this help and exit

If <first> or <increment> are omitted, they default to 1. When <first> is
larger than <last>, <increment>, if not set, defaults to -1.
The sequence of numbers ends when the sum of the current number and
<increment> reaches the limit imposed by <last>.

First, look at a regular expression that’s intended to capture the requirements above:

 1 args_pattern = re.compile(
 2     r"""
 3     ^
 4     (
 5         (--(?P<HELP>help).*)|
 6         ((?:-s|--separator)\s(?P<SEP>.*?)\s)?
 7         ((?P<OP1>-?\d+))(\s(?P<OP2>-?\d+))?(\s(?P<OP3>-?\d+))?
 8     )
 9     $
10 """,
11     re.VERBOSE,
12 )

To experiment with the regular expression above, you may use the snippet recorded on Regular Expression 101. The regular expression captures and enforces a few aspects of the requirements given for seq. In particular, the command may take:

A help option, in short (-h) or long format (--help), captured as a named group called HELP
A separator option, -s or --separator, taking an optional argument, and captured as named group called SEP
Up to three integer operands, respectively captured as OP1, OP2, and OP3

For clarity, the pattern args_pattern above uses the flag re.VERBOSE on line 11. This allows you to spread the regular expression over a few lines to enhance readability. The pattern validates the following:

Argument order: Options and arguments are expected to be laid out in a given order. For example, options are expected before the arguments.
Option values**: Only --help, -s, or --separator are expected as options.
Argument mutual exclusivity: The option --help isn’t compatible with other options or arguments.
Argument type: Operands are expected to be positive or negative integers.

For the regular expression to be able to handle these things, it needs to see all Python command line arguments in one string. You can collect them using str.join():

arg_line = " ".join(sys.argv[1:])

This makes arg_line a string that includes all arguments, except the program name, separated by a space.

Given the pattern args_pattern above, you can extract the Python command line arguments with the following function:

def parse(arg_line: str) -> Dict[str, str]:
    args: Dict[str, str] = {}
    if match_object := args_pattern.match(arg_line):
        args = {k: v for k, v in match_object.groupdict().items()
                if v is not None}
    return args

The pattern is already handling the order of the arguments, mutual exclusivity between options and arguments, and the type of the arguments. parse() is applying re.match() to the argument line to extract the proper values and store the data in a dictionary.

The dictionary includes the names of each group as keys and their respective values. For example, if the arg_line value is --help, then the dictionary is {'HELP': 'help'}. If arg_line is -s T 10, then the dictionary becomes {'SEP': 'T', 'OP1': '10'}. You can expand the code block below to see an implementation of seq with regular expressions.

The code below implements a limited version of seq with a regular expression to handle the command line parsing and validation:

# seq_regex.py

from typing import List, Dict
import re
import sys

USAGE = (
    f"Usage: {sys.argv[0]} [-s <separator>] [first [increment]] last"
)

args_pattern = re.compile(
    r"""
    ^
    (
        (--(?P<HELP>help).*)|
        ((?:-s|--separator)\s(?P<SEP>.*?)\s)?
        ((?P<OP1>-?\d+))(\s(?P<OP2>-?\d+))?(\s(?P<OP3>-?\d+))?
    )
    $
""",
    re.VERBOSE,
)

def parse(arg_line: str) -> Dict[str, str]:
    args: Dict[str, str] = {}
    if match_object := args_pattern.match(arg_line):
        args = {k: v for k, v in match_object.groupdict().items()
                if v is not None}
    return args

def seq(operands: List[int], sep: str = "\n") -> str:
    first, increment, last = 1, 1, 1
    if len(operands) == 1:
        last = operands[0]
    if len(operands) == 2:
        first, last = operands
        if first > last:
            increment = -1
    if len(operands) == 3:
        first, increment, last = operands
    last = last + 1 if increment > 0 else last - 1
    return sep.join(str(i) for i in range(first, last, increment))

def main() -> None:
    args = parse(" ".join(sys.argv[1:]))
    if not args:
        raise SystemExit(USAGE)
    if args.get("HELP"):
        print(USAGE)
        return
    operands = [int(v) for k, v in args.items() if k.startswith("OP")]
    sep = args.get("SEP", "\n")
    print(seq(operands, sep))

if __name__ == "__main__":
    main()

You can execute the code above by running this command:

$ python seq_regex.py 3

This should output the following:

1
2
3

Try this command with other combinations, including the --help option.

You didn’t see a version option supplied here. This was done intentionally to reduce the length of the example. You may consider adding the version option as an extended exercise. As a hint, you could modify the regular expression by replacing the line (--(?P<HELP>help).*)| with (--(?P<HELP>help).*)|(--(?P<VER>version).*)|. An additional if block would also be needed in main().

At this point, you know a few ways to extract options and arguments from the command line. So far, the Python command line arguments were only strings or integers. Next, you’ll learn how to handle files passed as arguments.

File Handling

It’s time now to experiment with Python command line arguments that are expected to be file names. Modify sha1sum.py to handle one or more files as arguments. You’ll end up with a downgraded version of the original sha1sum utility, which takes one or more files as arguments and displays the hexadecimal SHA1 hash for each file, followed by the name of the file:

# sha1sum_file.py

import hashlib
import sys

def sha1sum(filename: str) -> str:
    hash = hashlib.sha1()
    with open(filename, mode="rb") as f:
        hash.update(f.read())
    return hash.hexdigest()

for arg in sys.argv[1:]:
    print(f"{sha1sum(arg)}  {arg}")

sha1sum() is applied to the data read from each file that you passed at the command line, rather than the string itself. Take note that m.update() takes a bytes-like object as an argument and that the result of invoking read() after opening a file with the mode rb will return a bytes object. For more information about handling file content, check out Reading and Writing Files in Python, and in particular, the section Working With Bytes.

The evolution of sha1sum_file.py from handling strings at the command line to manipulating the content of files is getting you closer to the original implementation of sha1sum:

$ sha1sum main main.c
9a6f82c245f5980082dbf6faac47e5085083c07d  main
125a0f900ff6f164752600550879cbfabb098bc3  main.c

The execution of the Python program with the same Python command line arguments gives this:

$ python sha1sum_file.py main main.c
9a6f82c245f5980082dbf6faac47e5085083c07d  main
125a0f900ff6f164752600550879cbfabb098bc3  main.c

Because you interact with the shell interpreter or the Windows command prompt, you also get the benefit of the wildcard expansion provided by the shell. To prove this, you can reuse main.py, which displays each argument with the argument number and its value:

$ python main.py main.*
Arguments count: 5
Argument      0: main.py
Argument      1: main.c
Argument      2: main.exe
Argument      3: main.obj
Argument      4: main.py

You can see that the shell automatically performs wildcard expansion so that any file with a base name matching main, regardless of the extension, is part of sys.argv.

The wildcard expansion isn’t available on Windows. To obtain the same behavior, you need to implement it in your code. To refactor main.py to work with wildcard expansion, you can use glob. The following example works on Windows and, though it isn’t as concise as the original main.py, the same code behaves similarly across platforms:

 1 # main_win.py
 2 
 3 import sys
 4 import glob
 5 import itertools
 6 from typing import List
 7 
 8 def expand_args(args: List[str]) -> List[str]:
 9     arguments = args[:1]
10     glob_args = [glob.glob(arg) for arg in args[1:]]
11     arguments += itertools.chain.from_iterable(glob_args)
12     return arguments
13 
14 if __name__ == "__main__":
15     args = expand_args(sys.argv)
16     print(f"Arguments count: {len(args)}")
17     for i, arg in enumerate(args):
18         print(f"Argument {i:>6}: {arg}")

In main_win.py, expand_args relies on glob.glob() to process the shell-style wildcards. You can verify the result on Windows and any other operating system:

C:/>python main_win.py main.*
Arguments count: 5
Argument      0: main_win.py
Argument      1: main.c
Argument      2: main.exe
Argument      3: main.obj
Argument      4: main.py

This addresses the problem of handling files using wildcards like the asterisk (*) or question mark (?), but how about stdin?

If you don’t pass any parameter to the original sha1sum utility, then it expects to read data from the standard input. This is the text you enter at the terminal that ends when you type Ctrl+D on Unix-like systems or Ctrl+Z on Windows. These control sequences send an end of file (EOF) to the terminal, which stops reading from stdin and returns the data that was entered.

In the next section, you’ll add to your code the ability to read from the standard input stream.

Standard Input

When you modify the previous Python implementation of sha1sum to handle the standard input using sys.stdin, you’ll get closer to the original sha1sum:

# sha1sum_stdin.py

from typing import List
import hashlib
import pathlib
import sys

def process_file(filename: str) -> bytes:
    return pathlib.Path(filename).read_bytes()

def process_stdin() -> bytes:
    return bytes("".join(sys.stdin), "utf-8")

def sha1sum(data: bytes) -> str:
    sha1_hash = hashlib.sha1()
    sha1_hash.update(data)
    return sha1_hash.hexdigest()

def output_sha1sum(data: bytes, filename: str = "-") -> None:
    print(f"{sha1sum(data)}  {filename}")

def main(args: List[str]) -> None:
    if not args:
        args = ["-"]
    for arg in args:
        if arg == "-":
            output_sha1sum(process_stdin(), "-")
        else:
            output_sha1sum(process_file(arg), arg)

if __name__ == "__main__":
    main(sys.argv[1:])

Two conventions are applied to this new sha1sum version:

Without any arguments, the program expects the data to be provided in the standard input, sys.stdin, which is a readable file object.
When a hyphen (-) is provided as a file argument at the command line, the program interprets it as reading the file from the standard input.

Try this new script without any arguments. Enter the first aphorism of The Zen of Python, then complete the entry with the keyboard shortcut Ctrl+D on Unix-like systems or Ctrl+Z on Windows:

$ python sha1sum_stdin.py
Beautiful is better than ugly.
ae5705a3efd4488dfc2b4b80df85f60c67d998c4  -

You can also include one of the arguments as stdin mixed with the other file arguments like so:

$ python sha1sum_stdin.py main.py - main.c
d84372fc77a90336b6bb7c5e959bcb1b24c608b4  main.py
Beautiful is better than ugly.
ae5705a3efd4488dfc2b4b80df85f60c67d998c4  -
125a0f900ff6f164752600550879cbfabb098bc3  main.c

Another approach on Unix-like systems is to provide /dev/stdin instead of - to handle the standard input:

$ python sha1sum_stdin.py main.py /dev/stdin main.c
d84372fc77a90336b6bb7c5e959bcb1b24c608b4  main.py
Beautiful is better than ugly.
ae5705a3efd4488dfc2b4b80df85f60c67d998c4  /dev/stdin
125a0f900ff6f164752600550879cbfabb098bc3  main.c

On Windows there’s no equivalent to /dev/stdin, so using - as a file argument works as expected.

The script sha1sum_stdin.py isn’t covering all necessary error handling, but you’ll cover some of the missing features later in this tutorial.

Standard Output and Standard Error

Command line processing may have a direct relationship with stdin to respect the conventions detailed in the previous section. The standard output, although not immediately relevant, is still a concern if you want to adhere to the Unix Philosophy. To allow small programs to be combined, you may have to take into account the three standard streams:

stdin
stdout
stderr

The output of a program becomes the input of another one, allowing you to chain small utilities. For example, if you wanted to sort the aphorisms of the Zen of Python, then you could execute the following:

$ python -c "import this" | sort
Although never is often better than *right* now.
Although practicality beats purity.
Although that way may not be obvious at first unless you're Dutch.
...

The output above is truncated for better readability. Now imagine that you have a program that outputs the same data but also prints some debugging information:

# zen_sort_debug.py

print("DEBUG >>> About to print the Zen of Python")
import this
print("DEBUG >>> Done printing the Zen of Python")

Executing the Python script above gives:

$ python zen_sort_debug.py
DEBUG >>> About to print the Zen of Python
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
...
DEBUG >>> Done printing the Zen of Python

The ellipsis (...) indicates that the output was truncated to improve readability.

Now, if you want to sort the list of aphorisms, then execute the command as follows:

$ python zen_sort_debug.py | sort

Although never is often better than *right* now.
Although practicality beats purity.
Although that way may not be obvious at first unless you're Dutch.
Beautiful is better than ugly.
Complex is better than complicated.
DEBUG >>> About to print the Zen of Python
DEBUG >>> Done printing the Zen of Python
Errors should never pass silently.
...

You may realize that you didn’t intend to have the debug output as the input of the sort command. To address this issue, you want to send traces to the standard errors stream, stderr, instead:

# zen_sort_stderr.py
import sys

print("DEBUG >>> About to print the Zen of Python", file=sys.stderr)
import this
print("DEBUG >>> Done printing the Zen of Python", file=sys.stderr)

Execute zen_sort_stderr.py to observe the following:

$ python zen_sort_stderr.py | sort
DEBUG >>> About to print the Zen of Python
DEBUG >>> Done printing the Zen of Python

Although never is often better than *right* now.
Although practicality beats purity.
Although that way may not be obvious at first unless you're Dutch
....

Now, the traces are displayed to the terminal, but they aren’t used as input for the sort command.

Custom Parsers

You can implement seq by relying on a regular expression if the arguments aren’t too complex. Nevertheless, the regex pattern may quickly render the maintenance of the script difficult. Before you try getting help from specific libraries, another approach is to create a custom parser. The parser is a loop that fetches each argument one after another and applies a custom logic based on the semantics of your program.

A possible implementation for processing the arguments of seq_parse.py could be as follows:

 1 def parse(args: List[str]) -> Tuple[str, List[int]]:
 2     arguments = collections.deque(args)
 3     separator = "\n"
 4     operands: List[int] = []
 5     while arguments:
 6         arg = arguments.popleft()
 7         if not operands:
 8             if arg == "--help":
 9                 print(USAGE)
10                 sys.exit(0)
11             if arg in ("-s", "--separator"):
12                 separator = arguments.popleft()
13                 continue
14         try:
15             operands.append(int(arg))
16         except ValueError:
17             raise SystemExit(USAGE)
18         if len(operands) > 3:
19             raise SystemExit(USAGE)
20 
21     return separator, operands

parse() is given the list of arguments without the Python file name and uses collections.deque() to get the benefit of .popleft(), which removes the elements from the left of the collection. As the items of the arguments list unfold, you apply the logic that’s expected for your program. In parse() you can observe the following:

The while loop is at the core of the function, and terminates when there are no more arguments to parse, when the help is invoked, or when an error occurs.
If the separator option is detected, then the next argument is expected to be the separator.
operands stores the integers that are used to calculate the sequence. There should be at least one operand and at most three.

A full version of the code for parse() is available below:

# seq_parse.py

from typing import Dict, List, Tuple
import collections
import re
import sys

USAGE = (f"Usage: {sys.argv[0]} "
         "[--help] | [-s <sep>] [first [incr]] last")

def seq(operands: List[int], sep: str = "\n") -> str:
    first, increment, last = 1, 1, 1
    if len(operands) == 1:
        last = operands[0]
    if len(operands) == 2:
        first, last = operands
        if first > last:
            increment = -1
    if len(operands) == 3:
        first, increment, last = operands
    last = last + 1 if increment > 0 else last - 1
    return sep.join(str(i) for i in range(first, last, increment))

def parse(args: List[str]) -> Tuple[str, List[int]]:
    arguments = collections.deque(args)
    separator = "\n"
    operands: List[int] = []
    while arguments:
        arg = arguments.popleft()
        if not len(operands):
            if arg == "--help":
                print(USAGE)
                sys.exit(0)
            if arg in ("-s", "--separator"):
                separator = arguments.popleft() if arguments else None
                continue
        try:
            operands.append(int(arg))
        except ValueError:
            raise SystemExit(USAGE)
        if len(operands) > 3:
            raise SystemExit(USAGE)

    return separator, operands

def main() -> None:
    sep, operands = parse(sys.argv[1:])
    if not operands:
        raise SystemExit(USAGE)
    print(seq(operands, sep))

if __name__ == "__main__":
    main()

Note that some error handling aspects are kept to a minimum so as to keep the examples relatively short.

This manual approach of parsing the Python command line arguments may be sufficient for a simple set of arguments. However, it becomes quickly error-prone when complexity increases due to the following:

A large number of arguments
Complexity and interdependency between arguments
Validation to perform against the arguments

The custom approach isn’t reusable and requires reinventing the wheel in each program. By the end of this tutorial, you’ll have improved on this hand-crafted solution and learned a few better methods.

A Few Methods for Validating Python Command Line Arguments

You’ve already performed validation for Python command line arguments in a few examples like seq_regex.py and seq_parse.py. In the first example, you used a regular expression, and in the second example, a custom parser.

Both of these examples took the same aspects into account. They considered the expected options as short-form (-s) or long-form (--separator). They considered the order of the arguments so that options would not be placed after operands. Finally, they considered the type, integer for the operands, and the number of arguments, from one to three arguments.

Type Validation With Python Data Classes

The following is a proof of concept that attempts to validate the type of the arguments passed at the command line. In the following example, you validate the number of arguments and their respective type:

# val_type_dc.py

import dataclasses
import sys
from typing import List, Any

USAGE = f"Usage: python {sys.argv[0]} [--help] | firstname lastname age]"

@dataclasses.dataclass
class Arguments:
    firstname: str
    lastname: str
    age: int = 0

def check_type(obj):
    for field in dataclasses.fields(obj):
        value = getattr(obj, field.name)
        print(
            f"Value: {value}, "
            f"Expected type {field.type} for {field.name}, "
            f"got {type(value)}"
        )
        if type(value) != field.type:
            print("Type Error")
        else:
            print("Type Ok")

def validate(args: List[str]):
    # If passed to the command line, need to convert
    # the optional 3rd argument from string to int
    if len(args) > 2 and args[2].isdigit():
        args[2] = int(args[2])
    try:
        arguments = Arguments(*args)
    except TypeError:
        raise SystemExit(USAGE)
    check_type(arguments)

def main() -> None:
    args = sys.argv[1:]
    if not args:
        raise SystemExit(USAGE)

    if args[0] == "--help":
        print(USAGE)
    else:
        validate(args)

if __name__ == "__main__":
    main()

Unless you pass the --help option at the command line, this script expects two or three arguments:

A mandatory string: firstname
A mandatory string: lastname
An optional integer: age

Because all the items in sys.argv are strings, you need to convert the optional third argument to an integer if it’s composed of digits. str.isdigit() validates if all the characters in a string are digits. In addition, by constructing the data class Arguments with the values of the converted arguments, you obtain two validations:

If the number of arguments doesn’t correspond to the number of mandatory fields expected by Arguments, then you get an error. This is a minimum of two and a maximum of three fields.
If the types after conversion aren’t matching the types defined in the Arguments data class definition, then you get an error.

You can see this in action with the following execution:

$ python val_type_dc.py Guido "Van Rossum" 25
Value: Guido, Expected type <class 'str'> for firstname, got <class 'str'>
Type Ok
Value: Van Rossum, Expected type <class 'str'> for lastname, got <class 'str'>
Type Ok
Value: 25, Expected type <class 'int'> for age, got <class 'int'>
Type Ok

In the execution above, the number of arguments is correct and the type of each argument is also correct.

Now, execute the same command but omit the third argument:

$ python val_type_dc.py Guido "Van Rossum"
Value: Guido, Expected type <class 'str'> for firstname, got <class 'str'>
Type Ok
Value: Van Rossum, Expected type <class 'str'> for lastname, got <class 'str'>
Type Ok
Value: 0, Expected type <class 'int'> for age, got <class 'int'>
Type Ok

The result is also successful because the field age is defined with a default value, 0, so the data class Arguments doesn’t require it.

On the contrary, if the third argument isn’t of the proper type—say, a string instead of integer—then you get an error:

python val_type_dc.py Guido Van Rossum
Value: Guido, Expected type <class 'str'> for firstname, got <class 'str'>
Type Ok
Value: Van, Expected type <class 'str'> for lastname, got <class 'str'>
Type Ok
Value: Rossum, Expected type <class 'int'> for age, got <class 'str'>
Type Error

The expected value Van Rossum, isn’t surrounded by quotes, so it’s split. The second word of the last name, Rossum, is a string that’s handled as the age, which is expected to be an int. The validation fails.

Note: For more details about the usage of data classes in Python, check out The Ultimate Guide to Data Classes in Python 3.7.

Similarly, you could also use a NamedTuple to achieve a similar validation. You’d replace the data class with a class deriving from NamedTuple, and check_type() would change as follows:

from typing import NamedTuple

class Arguments(NamedTuple):
    firstname: str
    lastname: str
    age: int = 0

def check_type(obj):
    for attr, value in obj._asdict().items():
        print(
            f"Value: {value}, "
            f"Expected type {obj.__annotations__[attr]} for {attr}, "
            f"got {type(value)}"
        )
        if type(value) != obj.__annotations__[attr]:
            print("Type Error")
        else:
            print("Type Ok")

A NamedTuple exposes functions like _asdict that transform the object into a dictionary that can be used for data lookup. It also exposes attributes like __annotations__, which is a dictionary storing types for each field, and For more on __annotations__, check out Python Type Checking (Guide).

As highlighted in Python Type Checking (Guide), you could also leverage existing packages like Enforce, Pydantic, and Pytypes for advanced validation.

Custom Validation

Not unlike what you’ve already explored earlier, detailed validation may require some custom approaches. For example, if you attempt to execute sha1sum_stdin.py with an incorrect file name as an argument, then you get the following:

$ python sha1sum_stdin.py bad_file.txt
Traceback (most recent call last):
  File "sha1sum_stdin.py", line 32, in <module>
    main(sys.argv[1:])
  File "sha1sum_stdin.py", line 29, in main
    output_sha1sum(process_file(arg), arg)
  File "sha1sum_stdin.py", line 9, in process_file
    return pathlib.Path(filename).read_bytes()
  File "/usr/lib/python3.8/pathlib.py", line 1222, in read_bytes
    with self.open(mode='rb') as f:
  File "/usr/lib/python3.8/pathlib.py", line 1215, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.8/pathlib.py", line 1071, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'bad_file.txt'

bad_file.txt doesn’t exist, but the program attempts to read it.

Revisit main() in sha1sum_stdin.py to handle non-existing files passed at the command line:

 1 def main(args):
 2     if not args:
 3         output_sha1sum(process_stdin())
 4     for arg in args:
 5         if arg == "-":
 6             output_sha1sum(process_stdin(), "-")
 7             continue
 8         try:
 9             output_sha1sum(process_file(arg), arg)
10         except FileNotFoundError as err:
11             print(f"{sys.argv[0]}: {arg}: {err.strerror}", file=sys.stderr)

To see the complete example with this extra validation, expand the code block below:

# sha1sum_val.py

from typing import List
import hashlib
import pathlib
import sys

def process_file(filename: str) -> bytes:
    return pathlib.Path(filename).read_bytes()

def process_stdin() -> bytes:
    return bytes("".join(sys.stdin), "utf-8")

def sha1sum(data: bytes) -> str:
    m = hashlib.sha1()
    m.update(data)
    return m.hexdigest()

def output_sha1sum(data: bytes, filename: str = "-") -> None:
    print(f"{sha1sum(data)}  {filename}")

def main(args: List[str]) -> None:
    if not args:
        output_sha1sum(process_stdin())
    for arg in args:
        if arg == "-":
            output_sha1sum(process_stdin(), "-")
            continue
        try:
            output_sha1sum(process_file(arg), arg)
        except (FileNotFoundError, IsADirectoryError) as err:
            print(f"{sys.argv[0]}: {arg}: {err.strerror}", file=sys.stderr)

if __name__ == "__main__":
    main(sys.argv[1:])

When you execute this modified script, you get this:

$ python sha1sum_val.py bad_file.txt
sha1sum_val.py: bad_file.txt: No such file or directory

Note that the error displayed to the terminal is written to stderr, so it doesn’t interfere with the data expected by a command that would read the output of sha1sum_val.py:

$ python sha1sum_val.py bad_file.txt main.py | cut -d " " -f 1
sha1sum_val.py: bad_file.txt: No such file or directory
d84372fc77a90336b6bb7c5e959bcb1b24c608b4

This command pipes the output of sha1sum_val.py to cut to only include the first field. You can see that cut ignores the error message because it only receives the data sent to stdout.

The Python Standard Library

Despite the different approaches you took to process Python command line arguments, any complex program might be better off leveraging existing libraries to handle the heavy lifting required by sophisticated command line interfaces. As of Python 3.7, there are three command line parsers in the standard library:

The recommended module to use from the standard library is argparse. The standard library also exposes optparse but it’s officially deprecated and only mentioned here for your information. It was superseded by argparse in Python 3.2 and you won’t see it discussed in this tutorial.

`argparse`

You’re going to revisit sha1sum_val.py, the most recent clone of sha1sum, to introduce the benefits of argparse. To this effect, you’ll modify main() and add init_argparse to instantiate argparse.ArgumentParser:

 1 import argparse
 2 
 3 def init_argparse() -> argparse.ArgumentParser:
 4     parser = argparse.ArgumentParser(
 5         usage="%(prog)s [OPTION] [FILE]...",
 6         description="Print or check SHA1 (160-bit) checksums."
 7     )
 8     parser.add_argument(
 9         "-v", "--version", action="version",
10         version = f"{parser.prog} version 1.0.0"
11     )
12     parser.add_argument('files', nargs='*')
13     return parser
14 
15 def main() -> None:
16     parser = init_argparse()
17     args = parser.parse_args()
18     if not args.files:
19         output_sha1sum(process_stdin())
20     for file in args.files:
21         if file == "-":
22             output_sha1sum(process_stdin(), "-")
23             continue
24         try:
25             output_sha1sum(process_file(file), file)
26         except (FileNotFoundError, IsADirectoryError) as err:
27             print(f"{sys.argv[0]}: {file}: {err.strerror}", file=sys.stderr)

For the cost of a few more lines compared to the previous implementation, you get a clean approach to add --help and --version options that didn’t exist before. The expected arguments (the files to be processed) are all available in field files of object argparse.Namespace. This object is populated on line 17 by calling parse_args().

To look at the full script with the modifications described above, expand the code block below:

# sha1sum_argparse.py

import argparse
import hashlib
import pathlib
import sys

def process_file(filename: str) -> bytes:
    return pathlib.Path(filename).read_bytes()

def process_stdin() -> bytes:
    return bytes("".join(sys.stdin), "utf-8")

def sha1sum(data: bytes) -> str:
    sha1_hash = hashlib.sha1()
    sha1_hash.update(data)
    return sha1_hash.hexdigest()

def output_sha1sum(data: bytes, filename: str = "-") -> None:
    print(f"{sha1sum(data)}  {filename}")

def init_argparse() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        usage="%(prog)s [OPTION] [FILE]...",
        description="Print or check SHA1 (160-bit) checksums.",
    )
    parser.add_argument(
        "-v", "--version", action="version",
        version=f"{parser.prog} version 1.0.0"
    )
    parser.add_argument("files", nargs="*")
    return parser

def main() -> None:
    parser = init_argparse()
    args = parser.parse_args()
    if not args.files:
        output_sha1sum(process_stdin())
    for file in args.files:
        if file == "-":
            output_sha1sum(process_stdin(), "-")
            continue
        try:
            output_sha1sum(process_file(file), file)
        except (FileNotFoundError, IsADirectoryError) as err:
            print(f"{parser.prog}: {file}: {err.strerror}", file=sys.stderr)

if __name__ == "__main__":
    main()

To illustrate the immediate benefit you obtain by introducing argparse in this program, execute the following:

$ python sha1sum_argparse.py --help
usage: sha1sum_argparse.py [OPTION] [FILE]...

Print or check SHA1 (160-bit) checksums.

positional arguments:
  files

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  show program's version number and exit

To delve into the details of argparse, check out How to Build Command Line Interfaces in Python With argparse.

`getopt`

getopt finds its origins in the getopt C function. It facilitates parsing the command line and handling options, option arguments, and arguments. Revisit parse from seq_parse.py to use getopt:

def parse():
    options, arguments = getopt.getopt(
        sys.argv[1:],                      # Arguments
        'vhs:',                            # Short option definitions
        ["version", "help", "separator="]) # Long option definitions
    separator = "\n"
    for o, a in options:
        if o in ("-v", "--version"):
            print(VERSION)
            sys.exit()
        if o in ("-h", "--help"):
            print(USAGE)
            sys.exit()
        if o in ("-s", "--separator"):
            separator = a
    if not arguments or len(arguments) > 3:
        raise SystemExit(USAGE)
    try:
        operands = [int(arg) for arg in arguments]
    except ValueError:
        raise SystemExit(USAGE)
    return separator, operands

getopt.getopt() takes the following arguments:

The usual arguments list minus the script name, sys.argv[1:]
A string defining the short options
A list of strings for the long options

Note that a short option followed by a colon (:) expects an option argument, and that a long option trailed with an equals sign (=) expects an option argument.

The remaining code of seq_getopt.py is the same as seq_parse.py and is available in the collapsed code block below:

# seq_getopt.py

from typing import List, Tuple
import getopt
import sys

USAGE = f"Usage: python {sys.argv[0]} [--help] | [-s <sep>] [first [incr]] last"
VERSION = f"{sys.argv[0]} version 1.0.0"

def seq(operands: List[int], sep: str = "\n") -> str:
    first, increment, last = 1, 1, 1
    if len(operands) == 1:
        last = operands[0]
    elif len(operands) == 2:
        first, last = operands
        if first > last:
            increment = -1
    elif len(operands) == 3:
        first, increment, last = operands
    last = last - 1 if first > last else last + 1
    return sep.join(str(i) for i in range(first, last, increment))

def parse(args: List[str]) -> Tuple[str, List[int]]:
    options, arguments = getopt.getopt(
        args,                              # Arguments
        'vhs:',                            # Short option definitions
        ["version", "help", "separator="]) # Long option definitions
    separator = "\n"
    for o, a in options:
        if o in ("-v", "--version"):
            print(VERSION)
            sys.exit()
        if o in ("-h", "--help"):
            print(USAGE)
            sys.exit()
        if o in ("-s", "--separator"):
            separator = a
    if not arguments or len(arguments) > 3:
        raise SystemExit(USAGE)
    try:
        operands = [int(arg) for arg in arguments]
    except:
        raise SystemExit(USAGE)
    return separator, operands

def main() -> None:
    args = sys.argv[1:]
    if not args:
        raise SystemExit(USAGE)
    sep, operands = parse(args)
    print(seq(operands, sep))

if __name__ == "__main__":
    main()

Next, you’ll take a look at some external packages that will help you parse Python command line arguments.

A Few External Python Packages

Building upon the existing conventions you saw in this tutorial, there are a few libraries available on the Python Package Index (PyPI) that take many more steps to facilitate the implementation and maintenance of command line interfaces.

The following sections offer a glance at Click and Python Prompt Toolkit. You’ll only be exposed to very limited capabilities of these packages, as they both would require a full tutorial—if not a whole series—to do them justice!

Click

As of this writing, Click is perhaps the most advanced library to build a sophisticated command line interface for a Python program. It’s used by several Python products, most notably Flask and Black. Before you try the following example, you need to install Click in either a Python virtual environment or your local environment. If you’re not familiar with the concept of virtual environments, then check out Python Virtual Environments: A Primer.

To install Click, proceed as follows:

$ python -m pip install click

So, how could Click help you handle the Python command line arguments? Here’s a variation of the seq program using Click:

# seq_click.py

import click

@click.command(context_settings=dict(ignore_unknown_options=True))
@click.option("--separator", "-s",
              default="\n",
              help="Text used to separate numbers (default: \\n)")
@click.version_option(version="1.0.0")
@click.argument("operands", type=click.INT, nargs=-1)
def seq(operands, separator) -> str:
    first, increment, last = 1, 1, 1
    if len(operands) == 1:
        last = operands[0]
    elif len(operands) == 2:
        first, last = operands
        if first > last:
            increment = -1
    elif len(operands) == 3:
        first, increment, last = operands
    else:
        raise click.BadParameter("Invalid number of arguments")
    last = last - 1 if first > last else last + 1
    print(separator.join(str(i) for i in range(first, last, increment)))

if __name__ == "__main__":
    seq()

Setting ignore_unknown_options to True ensures that Click doesn’t parse negative arguments as options. Negative integers are valid seq arguments.

As you may have observed, you get a lot for free! A few well-carved decorators are sufficient to bury the boilerplate code, allowing you to focus on the main code, which is the content of seq() in this example.

Note: For more about Python decorators, check out Primer on Python Decorators.

The only import remaining is click. The declarative approach of decorating the main command, seq(), eliminates repetitive code that’s otherwise necessary. This could be any of the following:

Defining a help or usage procedure
Handling the version of the program
Capturing and setting up default values for options
Validating arguments, including the type

The new seq implementation barely scratches the surface. Click offers many niceties that will help you craft a very professional command line interface:

Output coloring
Prompt for omitted arguments
Commands and sub-commands
Argument type validation
Callback on options and arguments
File path validation
Progress bar

There are many other features as well. Check out Writing Python Command-Line Tools With Click to see more concrete examples based on Click.

Python Prompt Toolkit

There are other popular Python packages that are handling the command line interface problem, like docopt for Python. So, you may find the choice of the Prompt Toolkit a bit counterintuitive.

The Python Prompt Toolkit provides features that may make your command line application drift away from the Unix philosophy. However, it helps to bridge the gap between an arcane command line interface and a full-fledged graphical user interface. In other words, it may help to make your tools and programs more user-friendly.

You can use this tool in addition to processing Python command line arguments as in the previous examples, but this gives you a path to a UI-like approach without you having to depend on a full Python UI toolkit. To use prompt_toolkit, you need to install it with pip:

$ python -m pip install prompt_toolkit

You may find the next example a bit contrived, but the intent is to spur ideas and move you slightly away from more rigorous aspects of the command line with respect to the conventions you’ve seen in this tutorial.

As you’ve already seen the core logic of this example, the code snippet below only presents the code that significantly deviates from the previous examples:

def error_dlg():
    message_dialog(
        title="Error",
        text="Ensure that you enter a number",
    ).run()

def seq_dlg():
    labels = ["FIRST", "INCREMENT", "LAST"]
    operands = []
    while True:
        n = input_dialog(
            title="Sequence",
            text=f"Enter argument {labels[len(operands)]}:",
        ).run()
        if n is None:
            break
        if n.isdigit():
            operands.append(int(n))
        else:
            error_dlg()
        if len(operands) == 3:
            break

    if operands:
        seq(operands)
    else:
        print("Bye")        

actions = {"SEQUENCE": seq_dlg, "HELP": help, "VERSION": version}

def main():
    result = button_dialog(
        title="Sequence",
        text="Select an action:",
        buttons=[
            ("Sequence", "SEQUENCE"),
            ("Help", "HELP"),
            ("Version", "VERSION"),
        ],
    ).run()
    actions.get(result, lambda: print("Unexpected action"))()

The code above involves ways to interact and possibly guide users to enter the expected input, and to validate the input interactively using three dialog boxes:

button_dialog
message_dialog
input_dialog

The Python Prompt Toolkit exposes many other features intended to improve interaction with users. The call to the handler in main() is triggered by calling a function stored in a dictionary. Check out Emulating switch/case Statements in Python if you’ve never encountered this Python idiom before.

You can see the full example of the program using prompt_toolkit by expanding the code block below:

# seq_prompt.py

import sys
from typing import List
from prompt_toolkit.shortcuts import button_dialog, input_dialog, message_dialog

def version():
    print("Version 1.0.0")

def help():
    print("Print numbers from FIRST to LAST, in steps of INCREMENT.")

def seq(operands: List[int], sep: str = "\n"):
    first, increment, last = 1, 1, 1
    if len(operands) == 1:
        last = operands[0]
    elif len(operands) == 2:
        first, last = operands
        if first > last:
            increment = -1
    elif len(operands) == 3:
        first, increment, last = operands
    last = last - 1 if first > last else last + 1
    print(sep.join(str(i) for i in range(first, last, increment)))

def error_dlg():
    message_dialog(
        title="Error",
        text="Ensure that you enter a number",
    ).run()

def seq_dlg():
    labels = ["FIRST", "INCREMENT", "LAST"]
    operands = []
    while True:
        n = input_dialog(
            title="Sequence",
            text=f"Enter argument {labels[len(operands)]}:",
        ).run()
        if n is None:
            break
        if n.isdigit():
            operands.append(int(n))
        else:
            error_dlg()
        if len(operands) == 3:
            break

    if operands:
        seq(operands)
    else:
        print("Bye")        

actions = {"SEQUENCE": seq_dlg, "HELP": help, "VERSION": version}

def main():
    result = button_dialog(
        title="Sequence",
        text="Select an action:",
        buttons=[
            ("Sequence", "SEQUENCE"),
            ("Help", "HELP"),
            ("Version", "VERSION"),
        ],
    ).run()
    actions.get(result, lambda: print("Unexpected action"))()

if __name__ == "__main__":
    main()

When you execute the code above, you’re greeted with a dialog prompting you for action. Then, if you choose the action Sequence, another dialog box is displayed. After collecting all the necessary data, options, or arguments, the dialog box disappears, and the result is printed at the command line, as in the previous examples:

As the command line evolves and you can see some attempts to interact with users more creatively, other packages like PyInquirer also allow you to capitalize on a very interactive approach.

To further explore the world of the Text-Based User Interface (TUI), check out Building Console User Interfaces and the Third Party section in Your Guide to the Python Print Function.

If you’re interested in researching solutions that rely exclusively on the graphical user interface, then you may consider checking out the following resources:

Conclusion

In this tutorial, you’ve navigated many different aspects of Python command line arguments. You should feel prepared to apply the following skills to your code:

The conventions and pseudo-standards of Python command line arguments
The origins of sys.argv in Python
The usage of sys.argv to provide flexibility in running your Python programs
The Python standard libraries like argparse or getopt that abstract command line processing
The powerful Python packages like click and python_toolkit to further improve the usability of your programs

Whether you’re running a small script or a complex text-based application, when you expose a command line interface you’ll significantly improve the user experience of your Python software. In fact, you’re probably one of those users!

Next time you use your application, you’ll appreciate the documentation you supplied with the --help option or the fact that you can pass options and arguments instead of modifying the source code to supply different data.

Additional Resources

To gain further insights about Python command line arguments and their many facets, you may want to check out the following resources:

You may also want to try other Python libraries that target the same problems while providing you with different solutions:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sets in Python

2020-02-04T14:00:00+00:00

In this course, you’ll learn about sets. They’re a useful data structure that allows you to do some complex operations more easily. They come up everywhere in the real world and are important to understand.

By the end of this course, you’ll know:

What a set is
How to define a set in Python
How to operate on a set
How to modify a set
When to use sets
Why sets are a good choice for checking membership

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Use a Flask Blueprint to Architect Your Applications

2020-02-03T14:00:00+00:00

Flask is a very popular web application framework that leaves almost all design and architecture decisions up to the developer. In this tutorial, you’ll learn how a Flask Blueprint, or Blueprint for short, can help you structure your Flask application by grouping its functionality into reusable components.

In this tutorial, you’ll learn:

What Flask Blueprints are and how they work
How to create and use a Flask Blueprint to organize your code
How to improve code reusability using your own or a third-party Flask Blueprint

This tutorial assumes that you have some experience using Flask and that you’ve built some applications before. If you haven’t used Flask before, then check out Python Web Applications with Flask (Tutorial Series).

Free Bonus: Click here to get access to a free Flask + Python video tutorial that shows you how to build Flask web app, step-by-step.

What a Flask Application Looks Like

Let’s start by reviewing the structure of a small Flask application. You can create a small web application by following the steps in this section. To get started, you need to install the Flask Python package. You can run the following command to install Flask using pip:

$ pip install Flask==1.1.1

The above command installs Flask version 1.1.1. This is the version you’ll use throughout this tutorial, though you can apply what you’ll learn here to other versions, as well.

Note: For more information on how to install Flask in a virtual environment and other pip options, check out Python Virtual Environments: A Primer and What Is Pip? A Guide for New Pythonistas.

After you install Flask, you’re ready to start implementing its functionality. Since Flask doesn’t impose any restrictions on project structure, you can organize your project’s code as you want. For your first application, you can use a very straightforward layout, as shown below. A single file will contain all the application logic:

app/
|
└── app.py

The file app.py will contain the definition of the application and its views.

When you create a Flask application, you start by creating a Flask object that represents your application, and then you associate views to routes. Flask takes care of dispatching incoming requests to the correct view based on the request URL and the routes you’ve defined.

In Flask, views can be any callable (like a function) that receives requests and returns the response for that request. Flask is responsible for sending the response back to the user.

The following code block is your application’s full source code:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "This is an example app"

This code creates the object app, which belongs to the Flask class. The view function index() is linked to the route / using the app.route decorator. To learn more about decorators, check out Primer on Python Decorators and Python Decorators 101.

You can run the application with the following command:

$ flask run

By default, Flask will run the application you defined in app.py on port 5000. While the application is running, go to http://localhost:5000 using your web browser. You’ll see a page showing the message, This is an example app.

The chosen project layout is great for very small applications, but it doesn’t scale well. As your code grows, it can become harder for you to maintain everything in a single file. So, when your application grows in size or complexity, you may want to structure your code in a different way to keep it maintainable and clear to understand. Throughout this tutorial, you’ll learn how to use a Flask Blueprint to achieve this.

What a Flask Blueprint Looks Like

Flask Blueprints encapsulate functionality, such as views, templates, and other resources. To get a taste for how a Flask Blueprint would work, you can refactor the previous application by moving the index view into a Flask Blueprint. To do so, you have to create a Flask Blueprint that contains the index view and then use it in the application.

This is what the file structure looks like for this new application:

app/
|
├── app.py
└── example_blueprint.py

example_blueprint.py will contain the Flask Blueprint implementation. You’ll then modify app.py to use it.

The following code block shows how you can implement this Flask Blueprint in example_blueprint.py. It contains a view at the route / that returns the text This is an example app:

from flask import Blueprint

example_blueprint = Blueprint('example_blueprint', __name__)

@example_blueprint.route('/')
def index():
    return "This is an example app"

In the above code, you can see the steps common to most Flask Blueprint definitions:

Create a Blueprint object called example_blueprint.
Add views to example_blueprint using the route decorator.

The following code block shows how your application imports and uses the Flask Blueprint:

from flask import Flask
from example_blueprint import example_blueprint

app = Flask(__name__)
app.register_blueprint(example_blueprint)

To use any Flask Blueprint, you have to import it and then register it in the application using register_blueprint(). When a Flask Blueprint is registered, the application is extended with its contents.

You can run the application with the following command:

$ flask run

While the application is running, go to http://localhost:5000 using your web browser. You’ll see a page showing the message, This is an example app.

How Flask Blueprints Work

In this section, you’ll learn in detail how a Flask Blueprint is implemented and used. Each Flask Blueprint is an object that works very similarly to a Flask application. They both can have resources, such as static files, templates, and views that are associated with routes.

However, a Flask Blueprint is not actually an application. It needs to be registered in an application before you can run it. When you register a Flask Blueprint in an application, you’re actually extending the application with the contents of the Blueprint.

This is the key concept behind any Flask Blueprint. They record operations to be executed later when you register them on an application. For example, when you associate a view to a route in a Flask Blueprint, it records this association to be made later in the application when the Blueprint is registered.

Making a Flask Blueprint

Let’s revisit the Flask Blueprint definition that you’ve seen previously and review it in detail. The following code shows the Blueprint object creation:

from flask import Blueprint

example_blueprint = Blueprint('example_blueprint', __name__)

Note that in the above code, some arguments are specified when creating the Blueprint object. The first argument, "example_blueprint", is the Blueprint’s name, which is used by Flask’s routing mechanism. The second argument, __name__, is the Blueprint’s import name, which Flask uses to locate the Blueprint’s resources.

There are other optional arguments that you can provide to alter the Blueprint’s behavior:

static_folder: the folder where the Blueprint’s static files can be found
static_url_path: the URL to serve static files from
template_folder: the folder containing the Blueprint’s templates
url_prefix: the path to prepend to all of the Blueprint’s URLs
subdomain: the subdomain that this Blueprint’s routes will match on by default
url_defaults: a dictionary of default values that this Blueprint’s views will receive
root_path: the Blueprint’s root directory path, whose default value is obtained from the Blueprint’s import name

Note that all paths, except root_path, are relative to the Blueprint’s directory.

The Blueprint object example_blueprint has methods and decorators that allow you to record operations to be executed when registering the Flask Blueprint in an application to extend it. One of the most used decorators is route. It allows you to associate a view function to a URL route. The following code block shows how this decorator is used:

@example_blueprint.route('/')
def index():
    return "This is an example app"

You decorate index() using example_blueprint.route and associate the function to the URL /.

Blueprint objects also provide other methods that you may find useful:

.errorhandler() to register an error handler function
.before_request() to execute an action before every request
.after_request() to execute an action after every request
.app_template_filter() to register a template filter at the application level

You can learn more about using Blueprints and the Blueprint class in the Flask Blueprints Documentation.

Registering the Blueprint in Your Application

Recall that a Flask Blueprint is not actually an application. When you register the Flask Blueprint in an application, you extend the application with its contents. The following code shows how you can register the previously-created Flask Blueprint in an application:

from flask import Flask
from example_blueprint import example_blueprint

app = Flask(__name__)
app.register_blueprint(example_blueprint)

When you call .register_blueprint(), you apply all operations recorded in the Flask Blueprint example_blueprint to app. Now, requests to the app for the URL / will be served using .index() from the Flask Blueprint.

You can customize how the Flask Blueprint extends the application by providing some parameters to register_blueprint:

url_prefix is an optional prefix for all the Blueprint’s routes.
subdomain is a subdomain that Blueprint routes will match.
url_defaults is a dictionary with default values for view arguments.

Being able to do some customization at registration time, instead of at creation time, is particularly useful when you’re sharing the same Flask Blueprint in different projects.

In this section, you’ve seen how Flask Blueprints work and how you can create them and use them. In the following sections, you’ll learn how you can leverage a Flask Blueprint to architect your applications, structuring them into independent components. In some cases, it’s also possible for you to reuse these components in different applications to reduce development time!

How to Use Flask Blueprints to Architect Your Application’s Code

In this section, you’re going to see how you can refactor an example application using a Flask Blueprint. The example application is an e-commerce site with the following features:

Visitors can sign up, log in, and recover passwords.
Visitors can search for products and view their details.
Users can add products to their cart and checkout.
An API enables external systems to search and retrieve product information.

You don’t need to care much about the details of the implementation. Instead, you’ll focus mainly on how a Flask Blueprint can be used to improve the application’s architecture.

Understanding Why Project Layout Matters

Remember, Flask does not enforce any particular project layout. It’s completely feasible to organize this application’s code as follows:

ecommerce/
|
├── static/
|   ├── logo.png
|   ├── main.css
|   ├── generic.js
|   └── product_view.js
|
├── templates/
|   ├── login.html
|   ├── forgot_password.html
|   ├── signup.html
|   ├── checkout.html
|   ├── cart_view.html
|   ├── index.html
|   ├── products_list.html
|   └── product_view.html
|
├── app.py
├── config.py
└── models.py

This application’s code is organized using these directories and files:

static/ contains the application’s static files.
templates/ contains the application’s templates.
models.py contains the definition of the application’s models.
app.py contains the application logic.
config.py contains the application configuration parameters.

This is an example of how many applications begin. Although this layout is pretty straightforward, it has several drawbacks that arise as the app complexity increases. For example, it will be hard for you to reuse the application logic in other projects because all the functionality is bundled in app.py. If you split this functionality into modules instead, then you could reuse complete modules across different projects.

Also, if you have just one file for the application logic, then you would end up with a very large app.py that mixes code that’s nearly unrelated. This can make it hard for you to navigate and maintain the script.

What’s more, large code files are a source of conflicts when you’re working in a team, since everybody will be making changes to the same file. These are just a few reasons why the previous layout is only good for very small applications.

Organizing Your Projects

Instead of structuring the application using the previous layout, you can leverage a Flask Blueprint to split the code into different modules. In this section, you’ll see how to architect the previous application to make Blueprints that encapsulate related functionality. In this layout, there are five Flask Blueprints:

API Blueprint to enable external systems to search and retrieve product information
Authentication Blueprint to enable users to log in and recover their password
Cart Blueprint for cart and checkout functionality
General Blueprint for the homepage
Products Blueprint for searching and viewing products

If you use a separate directory for each Flask Blueprint and its resources, then the project layout would look as follows:

ecommerce/
|
├── api/
|   ├── __init__.py
|   └── api.py
|
├── auth/
|   ├── templates/
|   |   └── auth/
|   |       ├── login.html
|   |       ├── forgot_password.html
|   |       └── signup.html
|   |
|   ├── __init__.py
|   └── auth.py
|
├── cart/
|   ├── templates/
|   |   └── cart/
|   |       ├── checkout.html
|   |       └── view.html
|   |
|   ├── __init__.py
|   └── cart.py
|
├── general/
|   ├── templates/
|   |   └── general/
|   |       └── index.html
|   |
|   ├── __init__.py
|   └── general.py
|
├── products/
|   ├── static/
|   |   └── view.js
|   |
|   ├── templates/
|   |   └── products/
|   |       ├── list.html
|   |       └── view.html
|   |
|   ├── __init__.py
|   └── products.py
|
├── static/
|   ├── logo.png
|   ├── main.css
|   └── generic.js
|
├── app.py
├── config.py
└── models.py

To organize code in this way, you move all views from app.py into the corresponding Flask Blueprint. You also moved templates and non-global static files. This structure makes it easier for you to find the code and resources related to a given functionality. For example, if you want to find the application logic about products, then you can go to the Products Blueprint in products/products.py instead of scrolling through app.py.

Let’s see the Products Blueprint implementation in products/products.py:

from flask import Blueprint, render_template
from ecommerce.models import Product

products_bp = Blueprint('products_bp', __name__,
    template_folder='templates',
    static_folder='static', static_url_path='assets')

@products_bp.route('/')
def list():
    products = Product.query.all()
    return render_template('products/list.html', products=products)

@products_bp.route('/view/<int:product_id>')
def view(product_id):
    product = Product.query.get(product_id)
    return render_template('products/view.html', product=product)

This code defines the products_bp Flask Blueprint and contains only the code that’s related to product functionality. Since this Flask Blueprint has its own templates, you need to specify the template_folder relative to the Blueprint’s root in the Blueprint object creation. Since you specify static_folder='static' and static_url_path='assets', files in ecommerce/products/static/ will be served under the /assets/ URL.

Now you can move the rest of your code’s functionality to the corresponding Flask Blueprint. In other words, you can create Blueprints for API, authentication, cart, and general functionality. Once you’ve done so, the only code left in app.py will be code that deals with application initialization and Flask Blueprint registration:

from flask import Flask

from ecommmerce.api.api import api_bp
from ecommmerce.auth.auth import auth_bp
from ecommmerce.cart.cart import cart_bp
from ecommmerce.general.general import general_bp
from ecommmerce.products.products import products_bp

app = Flask(__name__)

app.register_blueprint(api_bp, url_prefix='/api')
app.register_blueprint(auth_bp)
app.register_blueprint(cart_bp, url_prefix='/cart')
app.register_blueprint(general_bp)
app.register_blueprint(products_bp, url_prefix='/products')

Now, app.py simply imports and registers the Blueprints to extend the application. Since you use url_prefix, you can avoid URL collisions between Flask Blueprint routes. For example, the URLs /products/ and /cart/ resolve to different endpoints defined in the products_bp and cart_bp Blueprints for the same route, /.

Including Templates

In Flask, when a view renders a template, the template file is searched in all the directories that were registered in the application’s template search path. By default, this path is ["/templates"], so templates are only searched for in the /templates directory inside the application’s root directory.

If you set the template_folder argument in a Blueprint’s creation, then its templates folder is added to the application’s template search path when the Flask Blueprint is registered. However, if there are duplicated file paths under different directories that are part of the template search path, then one will take precedence, depending on their registration order.

For example, if a view requests the template view.html and there are files with this same name in different directories in the template search path, then one of these will take precedence over the other. Since it may be hard to remember the precedence order, it’s best to avoid having files under the same path in different template directories. That’s why the following structure for the templates in the application makes sense:

ecommerce/
|
└── products/
    └── templates/
        └── products/
            ├── search.html
            └── view.html

At first, it may look redundant to have the Flask Blueprint name appear twice:

As the Blueprint’s root directory
Inside the templates directory

However, know that by doing this, you can avoid possible template name collisions between different Blueprints. Using this directory structure, any views requiring the view.html template for products can use products/view.html as the template file name when calling render_template. This avoids conflicts with the view.html that belongs to the Cart Blueprint.

As a final note, it’s important to know that templates in the application’s template directory have greater precedence than those inside the Blueprint’s template directory. This can be useful to know if you want to override Flask Blueprint templates without actually modifying the template file.

For example, if you wanted to override the template products/view.html in the Products Blueprint, then you can accomplish this by creating a new file products/view.html in the application templates directory:

ecommerce/
|
├── products/
|   └── templates/
|       └── products/
|           ├── search.html
|           └── view.html
|
└── templates/
        └── products/
            └── view.html

When you do this, your program will use templates/products/view.html instead of products/templates/products/view.html whenever a view requires the template products/view.html.

Providing Functionality Other Than Views

So far, you’ve only seen Blueprints that extend applications with views, but Flask Blueprints don’t have to provide just views! They can extend applications with templates, static files, and template filters. For example, you could create a Flask Blueprint to provide a set of icons and use it across your applications. This would be the file structure for such a Blueprint:

app/
|
└── icons/
    ├── static/
    |   ├── add.png
    |   ├── remove.png
    |   └── save.png
    |
    ├── __init__.py
    └── icons.py

The static folder contains the icon files and icons.py is the Flask Blueprint definition.

This is how icons.py might look:

from flask import Blueprint

icons_bp = Blueprint('icons_bp', __name__,
    static_folder='static',
    static_url_path='icons')

This code defines the icons_bp Flask Blueprint that exposes the files in the static directory under the /icons/ URL. Note that this Blueprint does not define any route.

When you can create Blueprints that package views and other types of content, you make your code and assets more reusable across your applications. You’ll learn more about Flask Blueprint reusability in the following section.

How to Use Flask Blueprints to Improve Code Reuse

Besides code organization, there’s another advantage to structuring your Flask application as a collection of independent components. You can reuse these components even across different applications! For example, if you created a Flask Blueprint that provides functionality for a contact form, then you can reuse it in all your applications.

You can also leverage Blueprints created by other developers to accelerate your work. While there’s no centralized repository for existing Flask Blueprints, you can find them using the Python Package Index, GitHub Search, and web search engines. You can learn more about searching PyPI packages in What Is Pip? A Guide for New Pythonistas.

There are various Flask Blueprints and Flask Extensions (which are implemented using Blueprints) that provide functionality that you may find useful:

Authentication
Admin/CRUD generation
CMS functionality
And more!

Instead of coding your application from scratch, you may consider searching for an existing Flask Blueprint or Extension that you can reuse. Leveraging third-party Blueprints and Extensions can help you to reduce development time and keep your focus on your application’s core logic!

Conclusion

In this tutorial, you’ve seen how Flask Blueprints work, how to use them, and how they can help you to organize your application’s code. Flask Blueprints are a great tool for dealing with application complexity as it increases.

You’ve learned:

What Flask Blueprints are and how they work
How you can implement and use a Flask Blueprint
How Flask Blueprints can help you to organize your application’s code
How you can use Flask Blueprints to ease the reusability of your own and third-party components
How using a Flask Blueprint in your project can reduce development time

You can use what you’ve learned in this tutorial to start organizing your applications as a set of blueprints. When you architect your applications this way, you’ll improve code reuse, maintainability, and teamwork!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Modules and Packages: An Introduction

2020-01-28T14:00:00+00:00

In this course, you’ll learn about Python modules and Python packages, two mechanisms that facilitate modular programming.

Modular programming is the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be put together like building blocks to create a larger application.

Learn how to write and import modules so you can optimize the structure of your own programs and make them easier to maintain and grow.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Basic Data Types in Python

2020-01-21T14:00:00+00:00

In this step-by-step course, you’ll dig into the basic data types that are built into Python.

By the end of this course:

You’ll learn about several basic numeric, string, and Boolean types that are built into Python.
You’ll see what objects of these types look like and how you can represent them.
You’ll get an overview of Python’s built-in functions, which are pre-written chunks of code that you can call to do useful things.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Supercharge Your Classes With Python super()

2020-01-14T14:00:00+00:00

While Python isn’t purely an object-oriented language, it’s flexible enough and powerful enough to allow you to build your applications using the object-oriented paradigm. One of the ways in which Python achieves this is by supporting inheritance, which it does with super().

By the end of this course, you’ll be able to:

Compose a class
Use super() to access parent methods
Understand single and multiple inheritance

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python args and kwargs: Demystified

2020-01-07T14:00:00+00:00

Sometimes, when you look at a function definition in Python, you might see that it takes two strange arguments: *args and **kwargs. If you’ve ever wondered what these peculiar variables are, or why your IDE defines them in main(), then this course is for you! You’ll learn how to use args and kwargs in Python to add more flexibility to your functions.

By the end of the course, you’ll know:

What *args and **kwargs actually mean
How to use *args and **kwargs in function definitions
How to use a single asterisk (*) to unpack iterables
How to use two asterisks (**) to unpack dictionaries

This course assumes that you already know how to define Python functions and work with lists and dictionaries.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sorting Data With Python

2019-12-31T14:00:00+00:00

All programmers will have to write code to sort items or data at some point. Sorting can be critical to the user experience in your application, whether it’s ordering a user’s most recent activity by timestamp, or putting a list of email recipients in alphabetical order by last name. Python sorting functionality offers robust features to do basic sorting or customize ordering at a granular level.

In this course, you’ll learn how to sort various types of data in different data structures, customize the order, and work with two different methods of sorting in Python.

By the end of this tutorial, you’ll know how to:

Implement basic Python sorting and ordering on data structures
Differentiate between sorted() and .sort()
Customize a complex sort order in your code based on unique requirements

For this course, you’ll need a basic understanding of lists and tuples as well as sets. Those data structures will be used in this course, and some basic operations will be performed on them. Also, this course uses Python 3, so example output might vary slightly from what you’d see if you were using Python 2.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Alexa Python Development: Build and Deploy an Alexa Skill

Getting Started With Alexa Python Development

Understanding Alexa Skills

Setting Up Your Environment

Understanding the Alexa Skill Model

Viewing a Sample Intent

Creating New Intents

Building the Skill Backend

Creating the JokeIntent Handler

Adding Jokes

Conclusion

How to Implement a Python Stack

Python Bindings: Calling C or C++ From Python

Python Bindings Overview

Marshalling Data Types

Understanding Mutable and Immutable Values

Managing Memory

Setting Up Your Environment

Using the invoke Tool

C or C++ Source

ctypes

How It’s Installed

Calling the Function

Library Loading

Calling Your Function

Strengths and Weaknesses

CFFI

How It’s Installed

Calling the Function

Write the Bindings

Build the Python Bindings

Calling Your Function

Strengths and Weaknesses

PyBind11

How It’s Installed

Calling the Function

Writing the Bindings

Build the Python Bindings

Calling Your Function

Strengths and Weaknesses

Cython

How It’s Installed

Calling the Function

Write the Bindings

Build the Python Bindings

Calling Your Function

Strengths and Weaknesses

Other Solutions

PyBindGen

Boost.Python

SIP

Cppyy

Shiboken

SWIG

Conclusion

The Beginner's Guide to Python Turtle

Getting to Know the Python turtle Library

Getting Started With turtle

Programming With turtle

Moving the Turtle

Drawing a Shape

Drawing Preset Figures

Changing the Screen Color

Changing the Screen Title

Changing the Turtle Size

Changing the Pen Size

Changing the Turtle and Pen Color

Filling in an Image

Changing the Turtle Shape

Changing the Pen Speed

Customizing in One Line

Picking the Pen Up and Down

Undoing Changes

Clearing the Screen

Resetting the Environment

Leaving a Stamp

Cloning Your Turtle

Using Loops and Conditional Statements

for Loops

Using the `invoke` Tool

`ctypes`

`CFFI`

`PyBind11`

`Cython`

`PyBindGen`

`Boost.Python`

`SIP`

`Cppyy`

`Shiboken`

`SWIG`

Getting to Know the Python `turtle` Library

Getting Started With `turtle`

Programming With `turtle`

`for` Loops

`while` Loops

`SELECT`

`JOIN`

`WHERE`

Using Python’s Null Object `None`

Using `None` as a Default Parameter

Using `None` as a Null Value in Python

Deciphering `None` in Tracebacks

The Python String `.format()` Method

The String `.format()` Method: Arguments

The String `.format()` Method: Simple Replacement Fields

The `<name>` Component

The `<conversion>` Component

The `<format_spec>` Component

The `<type>` Subcomponent

The `<fill>` and `<align>` Subcomponents

The `<sign>` Subcomponent

The `#` Subcomponent

The `0` Subcomponent

The `<width>` Subcomponent

The `<group>` Subcomponent

The `.<prec>` Subcomponent

The String `.format()` Method: Nested Replacement Fields