Many programming enthusiasts are enthused by the fresh new movement of ‘Rust’, the most loved programming language for the past five years, and Python, the most popular language for the past couple of years. Both languages are suitable for quick scripting and larger projects. What happens when you combine both?
RustPython 😉
The arrival of RustPython brings yet another competitor to the various Python interpreters that are already out there. With the potential to become the default Python interpreter, it is interesting to further explore this Rust implementation of Python.
What is RustPython?
RustPython is an Open Source (MIT) Python 3 interpreter written in Rust, available as both a library and a shell environment. Using Rust to implement the Python interpreter enables Python to be used as a programming language for Rust applications. Moreover, it allows Python to be immediately compiled in the browser using WebAssembly, meaning that anyone could easily run their Python code in the browser. As Python is the top programming language that developers want to learn, being able to run it in the browser would reduce the hassle to learn it and make it more accessible to anyone. Additionally, since RustPython is a Rust-based Python interpreter, the benefits of Rust come as a bonus to this interpreter. These bonuses include a minimal runtime, guaranteed memory and thread safety and better integrated error-handling. On top of that, RustPython, by design, doesn’t implement the quirks that are present in, for example, CPython, making it a ‘cleaner’ interpreter that has fewer compatibility hacks.
The core goal of RustPython is to run Python code along with Rust code. To that end, it aims to be a full implementation of the Python 3 environment. If you are unfamiliar with the Python landscape, let’s first clarify the two:
- Python the language: The language in which code is written, which is then interpreted by any of the given interpreters. Its design is decided and broadcast through PEPs, which are voted on and proposed by the entire community.
- CPython: The default interpreter for Python, which transforms python code to bytecode and runs that.
Why should you use RustPython?
Now that you know a bit about RustPython, why would you want to choose this Python interpreter over all the others that are available? Well, RustPython comes with a number of benefits over native CPython:
- Integration with the browser, thus running Python on the web.
- Embedding within a Rust application.
- Native multi-threading as there is no Global Interpreter Lock (GIL).
- Not bogged down by technical debt.
Let’s go into detail for each of these!
Integration with the browser
One of the major usages for RustPython is within the web, as it has the capability to compile to Web Assembly (WASM), which allows it to run almost as fast in a browser as natively, while completely portable and sandboxed! As more packages are made compatible, the entire environment can run on the web browser.
Embedding
Allowing for easy interop with Rust, a safer and more readable language than C, makes it more appealing and easy to rewrite those performance-critical parts of your code in the minimal runtime environment that Rust is.
Native Multi-threading
CPython chose very early on to avoid thinking of multi-threading and all the deadlocks it can cause, and simply only allow one thread per process. This prevents the sharing of data between pieces of code running in parallel and can cause significant slowdowns even on single processors. In RustPython multi-threading is built-in, which allows you to take advantage of it when it suits your needs.
Technical Debt
Throughout the history of a large project, it will inevitably accumulate technical debt, especially considering backwards compatibility. By starting afresh while still taking into account all the lessons learned from the older initiatives, RustPython provides cleaner and more maintainable prospects for its codebase.
Now that you are hopefully convinced of the potential of RustPython and that RustPython will eminently become the best choice when picking a multifaceted Python interpreter in the future, you might still be wondering how it actually works. Well, that is exactly what the next section will be about. 👀⬇️
How does it work?
RustPython is an interpreter, this is a computer program written in Rust that immediately executes instructions written in Python. An interpreter is not to be confused with a compiler, which generates, in one go, a stand-alone machine code program that the computer will process to perform the actions. Instead, an interpreter performs the actions written in the high-level program (Python in our case) line by line.
Given some Python code, RustPython will follow a strategy similar to that of CPython to interpret the code, as depicted by the figure below. It starts by parsing a line of code into tokens. Once these tokens have been validated for their syntax, an Abstract Syntax Tree (AST) will be created from these tokens. This means that the constructs described by the tokens are represented by a tree that contains the information about the instructions to be performed. And finally, the AST is compiled into bytecode, this is machinelike code. This bytecode can then be executed on a Virtual Machine or converted to actual machine code to be executed on a computer.
Figure: RustPython interpreter design (following the CPython strategy)
Who is it for?
Now that we know how RustPython works, it might also be interesting to investigate who the intended users are. To do so, one often endeavours to look into the relevant and interested parties that are involved, but before this is done, let’s first discuss the intended use of RustPython given the current state of affairs. This is a tricky question as, given its current state, there are only a few circumstances in which the use of RustPython is both advisable and possible. Currently, about 70% of Python is implemented in RustPython (that’s 130 000 of 180 000 tests!). As compatibility with C types is still missing, many packages will not load yet. However, a major step for the project is the support for Pip which was released this January.
One context in which this interpreter can be of use today, is on the web. It is quite lightweight and can run most packages that do not depend on C. For instance, pyflakes is considering moving from PyO3 (a collection of Rust bindings) to RustPython given its increased clarity.
When major blockers are resolved, RustPython will be usable as a drop-in replacement for CPython, making any context where its advantages can be made clear, open to consideration. For instance, security packages like PyCa, whose maintainers tend to be a fan of Rust over C, will be able to switch to supporting RustPython’s platform easily and securely.
Having defined the current state of RustPython and the possible use cases now and in the future, let’s continue by investigating the relevant and interested parties that are involved. The vision of a software product is often specified in terms of the needs of these parties. RustPython is no different: it was created by developers (including the Dutch Software Engineer Windel Bouwman) for developers. As the open-source environment of RustPython allows end-users to affect the development cycle, their needs are represented uniquely. Satisfying their needs (or allowing themselves to do so) attracts more developers, which in turn attracts more developers. To make this open-source environment effective, the available code documentation has to be of sufficient quality. The list of stakeholders also extends to businesses, who ultimately decide whether switching to this type of interpreter is worth the risk. The stakeholders impacted by RustPython can be separated into distinct groups.
Businesses
Or: Customers! These represent the companies that decide whether RustPython is a viable option for a new project. Companies that make use of (expensive) virtual machines running python scripts could prefer RustPython paired with Web Assembly in the future, as it allows for client-side python execution.
End Users
Developers looking for a new interpreter of Python written in Rust, offering several different benefits. They might want to run part of their Python code in Rust, want the security a Rust interpreter offers, or look to pair RustPython up with WebAssembly.
The Future of RustPython
As RustPython is currently still very much under development, the use-cases for it are still pretty limited. However, it will be interesting to uncover its roadmap by discussing the goals of RustPython.
Goals
Since RustPython is a relatively young open source project and not started or backed by some company, it’s not uncommon that there are not many long term goals listed somewhere. However, RustPython does specify two distinct short-term goals:
- “Full Python-3 environment entirely in Rust (not CPython bindings)” and
- “A clean implementation without compatibility hacks”.
The first one refers to Python functionalities which they have not yet implemented in Rust, but decided to, for now, directly call the CPython application. Some reasons for these postponements are: the feature being complex to implement, or they have simply not come around to them. The second goal listed is partially related to the first; CPython has some known ‘quirks’, which this project aims to avoid.
Key Attributes
To ensure RustPython is and stays an effective tool for developers, there are both internal and external attributes the project strives to meet.
- Internally, these attributes are crucial for (new) developers. Trivially, there are two aspects that stand out: documentation and maintainability. Testability is also important for developers, although arguably less important than the first two.
- Externally, the key attributes of RustPython represent those for end users. Compatibility is of high importance here, as indicated by the project itself. Full Python 3 compatibility is a minimal requirement for attracting new users. Usability and ease of integration also play a role in this respect.
Developers & Maintainer
Active contributors play a larger role in the development of RustPython and its goals. Maintainers help merge pull requests to ensure that new code is in line with these goals and up to standard. Most notably these include GitHub users Noah (@coolreader18) and Jeong (@youknowone).
Ethical considerations
Hopefully, you have been enthused to use RustPython in the near future. But maybe you are also considering to contribute to this project, if so, please take the following into consideration.
Firstly, all interpreters and compilers need to take into account that their application will most likely be used by other developers, who will create their own application using the interpreter. This means that, in a way, developers of interpreters have an extra responsibility: not only to their own direct users but also all user of applications using your software.
Secondly, interpreters have an added responsibility regarding the sustainability of their software. The amount of energy consumption and memory usage of the applications built using their interpreters will be dependent on how much the interpreter developers take sustainability into account.
You have made it to the end of this blog! Hopefully, you now know a lot more about RustPython! 🐍🤘