Executing Python Scripts With a Shebang – Real Python

What’s a Shebang, and When Should You Use It?

In short, a shebang is a special kind of comment that you may include in your source code to tell the operating system’s shell where to find the interpreter for the rest of the file:

If you’re using a shebang, it must appear on the first line in your script, and it has to start with a hash sign (#) followed by an exclamation mark (!), colloquially known as the bang, hence the name shebang. The choice of the hash sign to begin this special sequence of characters wasn’t accidental, as many scripting languages use it for inline comments.

You should make sure you don’t put any other comments before the shebang line if you want it to work correctly, or else it won’t be recognized! After the exclamation mark, specify an absolute path to the relevant code interpreter, such as Python. Providing a relative path will have no effect, unfortunately.

It’s not uncommon to combine a shebang with the name-main idiom, which prevents the main block of code from running when someone imports the file from another module:

With this conditional statement, Python will call the print() function only when you run this module directly as a script—for example, by providing its path to the Python interpreter:

As long as the script’s content starts with a correctly defined shebang line and your system user has permission to execute the corresponding file, you can omit the python3 command to run that script:

A shebang is only relevant to runnable scripts that you wish to execute without explicitly specifying the program to run them through. You wouldn’t typically put a shebang in a Python module that only contains function and class definitions meant for importing from other modules. Therefore, use the shebang when you don’t want to prefix the command that runs your Python script with python or python3.

Now that you have a high-level understanding of what a shebang is and when to use it, you’re ready to explore it in more detail. In the next section, you’ll take a closer look at how it works.

How Does a Shebang Work?

Normally, to run a program in the terminal, you must provide the full path to a particular binary executable or the name of a command present in one of the directories listed on the PATH environment variable. One or more command-line arguments may follow this path or command:

Here, you run the Python interpreter in a non-interactive mode against a one-liner program passed through the -c option. In the first case, you provide an absolute path to python3, while in the second case, you rely on the fact that the parent folder, /usr/bin/, is included on the search path by default. Your shell can find the Python executable, even if you don’t provide the full path, by looking through the directories on the PATH variable.

In practice, most of your Python programs will consist of more than one line of code spread across several modules. There will usually be a single runnable entry point to your program: a script, which you can pass on to the Python interpreter for execution:

So far, there’s nothing surprising about this invocation because you’ve seen it before. Notice, though, that you still run a binary executable carrying the machine code for your platform and computer architecture, which in turn interprets the Python code:

On many Linux distributions, python3 is an alias to an executable file that’s been compiled down to the Executable and Linkable Format (ELF), which you can take a peek at using the hexdump command.

However, your shell can also execute scripts or text files that contain source code expressed in a high-level interpreted language like Python, Perl, or JavaScript. Because executing scripts can potentially have harmful side effects, especially if they come from untrusted sources, files aren’t executable by default. When you try running a Python script without making it executable first, you’ll see this error message in the terminal:

In general, you can give permission to execute the specified file to its owner, a user belonging to the user group associated with the file, or everyone else. To let anyone execute your script, you can change its file mode bits using the chmod command:

Unix file permissions follow a symbolic notation where the letter x stands for the permission to execute, and the plus sign (+) turns the associated bit on. On some terminals, this will also change the color used to display your executable scripts so that you can tell them apart at a glance.

While you should be able to run your script now, it’s still not going to work the way you intended:

Unless you include a shebang at the beginning of the file, the shell will assume that your script is written in the corresponding shell language. For example, if you’re on the Bash shell, then the shell will expect to find the Bash commands in your file. So, when it stumbles on a call to Python’s print() function in your script, it doesn’t understand it. Note that the file’s extension, such as .py, is completely irrelevant!

It’s only when you provide the absolute path to your Python interpreter using a shebang in your script that the shell will know where to pass that script:

This is very convenient because you can now make runnable Python scripts. Unfortunately, hard-coding an absolute path in the shebang isn’t super portable across systems, even within the Unix family. What if Python came installed in a different location or the python3 command was replaced with python? What about using a virtual environment or pyenv? Currently, you’ll always run your script through the operating system’s default Python interpreter.

In the next section, you’ll look into addressing these concerns by improving your shebang and exploring some alternatives.

How Can You Define a Portable Shebang?

Having a fixed absolute path in a shebang means that your script may not work on everyone’s system because there might be slight differences.

Remember that you can’t specify a relative path in a shebang, as it always has to be absolute. Because of this limitation, many developers have adopted a work-around by using the /usr/bin/env command, which can figure out the actual path to the Python interpreter:

When invoked without any arguments, the /usr/bin/env command will display the environment variables defined in your shell. Its primary purpose, though, is to run a program in a modified environment, letting you temporarily override certain variables. For example, you can change the language of a given program by setting the LANG variable with it:

The git status command would normally display this message in your default language, but here, you request Spanish. Note that not every program supports multiple languages, and you might need to install an additional language pack on your operating system first for it to take effect.

The nice thing about /usr/bin/env is that you don’t have to change any environment variables whatsoever to run a command:

As a side effect, it’ll find the first occurrence of the specified executable, such as python3, on the PATH variable and run it for you. That’s quite useful when you consider that activating a Python virtual environment modifies the PATH variable in your current terminal session by prepending the parent folder of the Python executable in the active virtual environment:

When there’s no active virtual environment in your shell, the python3 command is short for /usr/bin/python3. But, as soon as you create and activate a new virtual environment, that same command points to the Python executable in a local venv/ folder. You can see why that happens by inspecting the PATH variable, which now starts with this folder, taking precedence over the globally installed Python interpreter.

One shortcoming of /usr/bin/env is that it doesn’t let you pass any arguments to the underlying command out of the box. So, if you wanted to run python3 -i to keep the Python interpreter running in an interactive mode after your script finishes, then it would throw a wrench into the works:

Fortunately, there’s a quick fix for that, which the error message hints at. You can use the /usr/bin/env command’s -S option to split the string that follows into separate arguments passed to the interpreter:

After the script runs, you’ll be dropped into the interactive Python REPL so that you can inspect the state of the variables, which might be helpful in post-mortem debugging.

At the end of the day, the shebang is a relatively straightforward way to make runnable Python scripts, but it requires a certain level of knowledge about the shell, environment variables, and the operating system that you’re working with. On top of that, it isn’t perfect in terms of portability because it primarily works on Unix-like systems.

If you don’t want to set the shebang yourself, then you can rely on tools like setuptools or Poetry that’ll do this job for you. They let you configure convenient entry points to your project through regular functions. Alternatively, you might create a special __main__.py file to turn your Python module into a runnable unit, or you can build an executable ZIP application in Python, avoiding the need for using the shebang.

Those are worthwhile alternatives to the shebang, and they allow you to avoid some of its weaknesses.

What Are Shebang Examples?

Up to this point, you’ve used the shebang to indicate a specific version of the Python interpreter for your scripts. However, in some cases, there might be more than one interpreter capable of understanding and acting on the same code. For example, you can write a script in a way that’s compatible with Python 2 and Python 3 at the same time:

The parentheses around the print statement in Python 2 end up being ignored, so you can safely use the python command without an explicit version in your shebang as long as you stay conservative with your syntax:

Heck, you can even run this code through the Perl interpreter, which also happens to have a print function behaving in a fashion that’s similar to its Python counterpart:

You don’t even need to change the file extension, which the shell doesn’t care about. Just by updating the shebang line, you’ll be able run this script with Perl if you’ve installed its interpreter before:

The result of running this script looks nearly the same, except that Python adds a trailing newline, while Perl doesn’t. This is a bare-bones example of a polyglot program written so that a few programming languages can understand it and produce the same output.

The proper version of the Hello, World! program written in Perl might look something like this:

Using parentheses around the function arguments in Perl is considered good practice but isn’t strictly mandatory. Notice the newline character (\n) in the string literal and the semicolon (;) at the end of the line, which terminates the statement.

If you have Node.js hanging around on your computer, then you can execute JavaScript right from your terminal. Here’s an analogous Hello, World! script written in a language that used to be the domain of web browsers:

Even though the hash sign isn’t valid syntax in JavaScript, the Node.js server recognizes the distinctive shebang sequence and ignores the entire line before executing the rest of the file.

With a little bit of effort, you can even write scripts using Java, which isn’t technically a scripting language. Java requires compiling its high-level code to bytecode for the Java Virtual Machine (JVM), which is kind of like the Python interpreter but for binary opcodes.

To make the shebang possible with Java programs, you must follow these steps:

1. Make sure that the file with your Java source code does not have the traditional .java extension. You can give the file a neutral .j extension, for instance. As explained in this StackOverflow answer, this will ensure that Java ignores the illegal hash sign character.
2. Run your source file through the java command instead of javac.
3. Explicitly set the Java 11 version with the --source switch.

Here’s a complete example of such a Java “script”:

Running this script takes noticeably longer than the analogous Hello, World! programs written in genuine scripting languages. That’s because of the extra compilation step, which takes place on the fly on each invocation.

As long as you can point your shell to the right interpreter, you’ll be able to make runnable scripts for any scripting language, including your own domain-specific language (DSL).

In the next section, you’ll see an example of a basic interpreter written in Python that can execute code in an esoteric programming language, which was chosen for its simplicity. The shebang will become the glue between your interpreter and the DSL scripts.

How Can You Use a Shebang With a Custom Interpreter in Python?

The technical details of building an interpreter for the famous esoteric language are far beyond the scope of this tutorial. Instead, you can find the complete Python source code of the said interpreter in the supporting materials, so go ahead and download them now if you’d like to follow along with the upcoming examples interactively:

Once you’ve downloaded the interpreter, you can install it with pip into a new virtual environment:

This should bring the custom brainf command into your virtual environment, which means you have an entry point to the installed Python package.

You can run this command without any arguments, in which case it’ll expect you to provide the code written in your domain-specific language through the standard input (stdin), almost like a Python REPL. When you’re done typing your code, you must confirm it with Enter and then terminate the program by hitting Ctrl+D to send an end-of-file (EOF) character:

This sample piece of code results in printing the ASCII letter A followed by a newline character on the screen. Adding one extra plus (+) instruction just before the first dot (.) in the code above results in printing the letter B instead. You can experiment with this code a little bit to get a feel for the language.

Alternatively, you may save your source code in a text file and provide its path as an argument to the brainf command:

The brainf command behaves pretty much like a Python interpreter at this point. Because of that, it’s also possible to insert an appropriate shebang at the beginning of your script and make it executable:

Wow! Remember it’s your custom interpreter, which was implemented in pure Python, that’s running this code and turning the cryptic characters into meaningful action.

If that wasn’t exciting enough, then feel free to explore some of the more advanced sample scripts made by Daniel B. Cristofani, who maintains an impressive online archive of his esoteric scripts. For example, take a look at this mind-blowing program, which draws the Sierpiński triangle using just seven distinct instructions:

There are a few interactive scripts as well, which can encode your input text using the ROT-13 cipher or calculate the factorial of the Fibonacci sequence. Perhaps the most impressive example you’ll ever find is an animated solution to the Tower of Hanoi puzzle, which was made by Clifford Wolf:

Unsurprisingly, its source code is quite a bit longer than the other examples, weighing almost 55 kilobytes. It also runs very slowly, so the video above is sped up fifteen times and only shows the first few steps of the algorithm.

Despite being toy examples, these perfectly demonstrate the power of using the shebang in executing any scripting language. Perhaps they’ll inspire you to develop your own domain-specific language and its interpreter so that you can solve bigger problems.

What Are Best Practices for the Shebang?

To sum up, here are a few rules that you should follow to successfully use a shebang in your Python scripts:

• Remember that a shebang is only applicable to runnable scripts on Unix-like operating systems.
• When you don’t specify a shebang, your shell will attempt to interpret the script as if it were written in the corresponding shell language.
• Don’t place a shebang in plain Python modules that are only meant to be imported and not executed.
• Make sure that your script is saved in an executable file.
• Consider combining a shebang with the name-main idiom (if __name__ == "__main__":).
• Begin your script with a shebang line, and don’t place any other comments before it.
• Start the shebang with a #! sequence of characters to distinguish it from a standard comment.
• Use the /usr/bin/env python3 command to avoid hard-coding an absolute path to any specific Python interpreter.
• Avoid using the bare python command unless your script is intentionally backward-compatible with Python 2. Generally, you should use the more explicit python3.
• Enable the -S flag if you need to pass extra arguments to the interpreter—for example, #!/usr/bin/env -S python3 -i.
• Be cautious about the number of characters that you put into your shebang line.

Finally, ask yourself if you need to add the shebang manually or if it could be generated automatically or replaced with some higher-level abstraction by a utility in your toolchain.

Conclusion

You know what a shebang is, how it works, and when you might want to include it in your Python scripts. You saw how to define a portable shebang and pass arguments to it. You also looked at some shebang examples and explored how to use a shebang with a custom interpreter written in Python. Finally, you reviewed some of the best practices for using a shebang in your scripts and learned about its shortcomings and alternatives.

In this tutorial, you’ve learned how to:

• Decide when to include the shebang in Python scripts
• Define the shebang in a portable way across systems
• Pass arguments to the command defined in a shebang
• Know the shebang’s limitations and some of its alternatives
• Execute scripts through a custom interpreter written in Python

Now that you understand how to use a shebang, why not try it out in your own scripts? What other uses can you think of for a shebang? Let your fellow programmers know in the comments below!