| This article brought to you by LWN subscribers
Subscribers to LWN.net made this article — and everything that |
May 2, 2023
This article was contributed by Koen Vervloesem
Linters are tools that analyze a program’s source code to detect various
problems such as syntax errors, programming mistakes, style violations, and
more. They are important for maintaining code quality and
readability in a project, as well as for catching bugs early in the
development cycle. Last year, a new Python linter appeared: Ruff. It’s fast, written in Rust, and in less than a year it has
been adopted by some high-profile projects, including FastAPI, Pandas, and SciPy.
Linting tools are often part of an integrated development
environment, used in
pre-commit
hooks, or as part of continuous-integration (CI) pipelines. Some popular
linters for
Python include Pylint, Flake8, Pyflakes, and pycodestyle (formerly called
pep8), which are all written in Python as well. Each linter checks whether
the code violates a list of rules. Ruff
reimplements a lot of the rules that are defined by these other popular Python
linters, and combines them into one tool.
Orders of magnitude faster
In August 2022, Charlie Marsh announced
Ruff, which he called “an extremely fast Python linter, written in
Rust”. He showed how Ruff is 150 times faster than Flake8 on macOS when
linting the Python files in the CPython code base, 75 times faster
than pycodestyle, and 50
times faster than Pyflakes and Pylint. While the exact speed gains aren’t
that important (and Ruff has become even faster since then), it’s clear that
it’s orders of magnitudes faster than its competitors, as Marsh
explained:
Even a conservative 25x is the difference between ~real-time
feedback (~300-500ms) and sitting around for 12+ seconds. With
a 150x speed-up, it’s ~300-500ms vs. 75 seconds. If you edit
a single file in CPython and re-run
ruff, it’s 60ms total, increasing the speed-up by another order of
magnitude.
In his example, Marsh touches on Ruff’s
caching. When re-running Ruff on a code base, it only lints the files that
have been changed since the previous run. In contrast, Flake8 re-lints all
of the
files every time, except when running it in a wrapper such as flake8-cached.
The gist of his message is: when linting a code base happens almost
instantaneously, there’s really no reason to not do it. This means that more
developers will add the linter to their pre-commit or CI configuration. So
speed is not just a nice-to-have, but is an essential element of improving code
quality.
One reason why Ruff is faster than the alternatives is that it’s
compiled into machine code instead of running in an interpreter. However, a
second reason for its speed is that it runs all of its checks in a single pass
over the code. Marsh contrasts this with how Flake8 works:
Flake8 is really a wrapper around other tools, like pyflakes and
pycodestyle. When you run Flake8, both pyflakes and pycodestyle are reading
every file from disk, tokenizing the code, and traversing the tree (I might
be wrong on some of the details, but you get the idea). If you then use
autoflake to automatically fix some of your lint violations, you’re running
pycodestyle yet again. How many times, in your pre-commit hooks, do you
read your source code from disk, parse it, and traverse the parse tree?
Ruff uses RustPython‘s
abstract syntax tree (AST) parser. For every file, Ruff generates the AST
exactly once, traverses all the nodes in the tree, applying the linter
rules in a single pass as it goes.
Ruff rules
Ruff has over 500 built-in
rules, many of them inspired by popular tools such as Flake8, isort, and pyupgrade, as well as
including some of
its own rules. There’s a
category for each of these linters, and each category comes with a
collection of rules. By default, Ruff enables all rules from the F category
(Pyflakes) and a subset of the E (Flake8’s errors) category. Ruff also
reimplements some of the functionality in the most popular Flake8 plugins as
well as in other
code-quality tools.
For its configuration, Ruff uses pyproject.toml. The
various categories of rules can be enabled and/or configured in this
file. For example, this snippet of the configuration enables the rules from
four linters, ignores two specific rules, and enforces the Google
style for docstrings:
[tool.ruff]
select = [
"ANN", # flake8-annotations
"D", # pydocstyle
"E", # pycodestyle
"F", # Pyflakes
]
ignore = [
"ANN101", # missing-type-self
"D107", # undocumented-public-init
]
[tool.ruff.pydocstyle]
convention = "google"
For strict linting, select = ["ALL"] enables all built-in
linters. On most code bases, this will result in a lot of errors, but
specific categories or rules can be ignored, as seen in the example
configuration above. Alternatively, individual violations for specific
rule codes can be ignored by adding # noqa: {code} at
the end of a line
in the source file.
Not all of the rules of the original tools are reimplemented in
Ruff. The documentation has a comparison
with Flake8, with Pylint,
and a list of implemented Flake8
plugins. Ruff also supports import sorting comparable
to isort, as well as linting docstrings
based on various conventions.
Contrary to the popular Flake8, Ruff doesn’t support plugins. So users
who want to extend the linter need to get their code accepted into Ruff’s
repository. There’s a GitHub issue about
plugins, with a
comment by Marsh that adding support for plugins is not a top priority
now because he
considers the unification of multiple tools into Ruff as a feature, not a
bug. But this means that developers who want to enforce their own quirky rules,
that won’t be accepted into Ruff because they’re too
specialized, will need to run another tool.
Using Ruff
Despite being written in Rust, Ruff can simply be installed with
pip install ruff, at least if there’s a wheel built for
the
environment. This is based on the Maturin project that allows building and
publishing Rust binaries as Python packages. Most users shouldn’t even
notice that
Ruff isn’t written in Python. Ruff supports any Python version from 3.7
onward.
Running Ruff manually is as easy as executing
ruff check src/
when the Python files are in the src directory. It then shows a
list of files with lines and columns where the linter finds an error,
followed by a code and short description of the rule. For example:
def is_uint16(number: int) -> bool:
"""Check whether a number is a 16-bit unsigned integer."""
return isinstance(number, int) and 0 <= number <= 0xFFFF
With the Pylint refactor rules enabled,
ruff check
gave me
the following warning:
src/bluetooth_numbers/utils.py:192:55: PLR2004 Magic value used in
comparison, consider replacing 65535 with a constant variable
I found that complaint to be overly pedantic, so I silenced it by adding
# noqa: PLR2004
to the end of the
return
statement.
A command like
ruff rule PLR2004 shows some more information about a
specific
rule. For some rules, the information provides a clear
example. Unfortunately, for many others the information is rather sparse,
only telling the developer what is not allowed, but not why or how to solve
the problem. Often there's a reference to the original project the rule is
derived
from, so searching that project's home page or repository can help.
There's also a --watch option that continuously re-runs the
linter when source files change. Ruff designates some errors as fixable: it
can resolve them automatically when running ruff check with the
--fix option. Some examples of these fixable errors are things
like unused
imports or invalid unescaped characters in strings.
Most projects benefit from using a configuration file for Ruff, where
specific linters and/or rules are enabled. For some of my own Python
projects, I was able to migrate from a combination of isort, pyupgrade,
Pylint, and Flake8 (with a lot of plugins) to Ruff with a short
configuration file.
What worked for me to get strict linting—all rules enabled—for
my relatively small Python projects was to look at the violations rule by
rule. Pick a rule that the code violates a lot, restrict the output to only
this rule with:
$ ruff check --select RULECODE src
Then fix the
issues one by one or add a
# noqa: RULECODE
comment where
needed. If none of the violations for a rule seem relevant, add it to the
ignore list. Then run Ruff again without restrictions, pick another rule,
look into that one, and so on, until there
are no violations left.
Apart from running Ruff manually, users can also integrate the linter
directly into their code editor (e.g. Visual Studio Code, Vim, Neovim,
Emacs) or
it can be
used with
the language server protocol for any tool that supports it.
The linter can be used as a pre-commit hook or
as a GitHub
action as well.
Ruff-like developer tools
In his announcement of Ruff, Marsh had already hinted at the
possibilities of other Python developer tools using the same approach as
Ruff:
The question I keep asking myself is: could we take the Ruff model and
apply it to other tooling? You could probably give autoformatters (like
Black and isort) the same treatment. But what about type checkers? I'm not
sure! Mypy is already compiled with mypyc, and so is much faster than pure
Python; and Pyright is written in Node. It's something I'd like to put to
the test.
In mid-April, Marsh announced that
he has started a company, Astral, to continue
building high-performance developer tools for the Python ecosystem:
Some of the things we build will look like natural extensions of Ruff
(e.g., an autoformatter); others will diverge from the static analysis
theme. But our North Star is pretty simple: make the Python ecosystem more
productive by building tools people love to use — tools that feel fast,
robust, intuitive, and integrated.
What won't change, according to Marsh, is the open-source and permissively
licensed nature of Ruff (which has an MIT license) and other tools that
will be created by Astral.
Conclusion
For developers who are now using Flake8 with various plugins, Pylint, isort,
pyupgrade, and many other tools to check their code quality, migrating
to Ruff can greatly simplify their development environment. Not only does this
result in fewer dependencies, it also makes linting the code base faster. Black for code formatting, Ruff for
linting, and mypy for type checking
seems to be the sweet spot for many projects these days. It remains to be seen
whether the Astral team will create a code formatter and type
checker to complement Ruff.
(Log in to post comments)