Move linting to ruff #6966

akx · 2023-02-23T12:09:00Z

This PR switches linting (flake8, isort, ~~bandit~~, yesqa, pygrep-hooks) for Pillow to the ruff linter.

Running a full pre-commit run is much faster (45.15 seconds of CPU time to 8.62 seconds on my machine)
It's easier to enable more rules if need be since the configuration is all in one place
👉 The only rule that may not be yet supported by ruff that had been previously enabled is rst-backticks from pygrep-hooks.
- That being said, this PR still enables more lints than had been in use from pygrep-hooks since it enables the whole PGH family.
👉 Instead of touching expressions like x, y, l, r, w, a, d, f = metrics[ix], I opted to adding specific noqa directives to quiesce Ruff's "ambiguous name" for l (which could be an 1 or an I 😁) No longer an issue via Clarify some local variable names #6971.

hugovk

Running a full pre-commit run is much faster (45.15 seconds of CPU time to 8.62 seconds on my machine)

That's a good speedup! I get 5s -> 2s for pre-commit run --all-files, which is still welcome. On pre-commit.ci, it's 23.5s -> 12.3s.

hugovk · 2023-02-23T13:01:35Z

Makefile

-	python3 -c "import isort" > /dev/null 2>&1 || python3 -m pip install isort
-	python3 -m black --target-version py37 .
-	python3 -m isort .
+	ruff --fix .


Why remove Black?

Please also make sure Ruff is installed.

Oops, oversight there.

That said, should this actually be pre-commit run --all-files black ruff?

.pre-commit-config.yaml

hugovk · 2023-02-23T13:05:22Z

src/PIL/IptcImagePlugin.py

@@ -18,10 +18,9 @@
 import tempfile

 from . import Image, ImageFile
-from ._binary import i8
+from ._binary import i8, o8


What happened here? Does Ruff not support isort's Black profile?

isort is a bit looser than Ruff with these; Ruff wants either everything or nothing to be grouped like this, and isort doesn't always care.

Testing, isort always sorts this like the original, with or without the Black profile.

Anyway, I don't mind too much, as long as it remains relatively stable.

On that topic, maybe we should wait until Ruff is at "v1" or "stable", which looks to be close:

astral-sh/ruff#1992

Ruff has Black profile as the ONLY profile possible.

src/PIL/BdfFontFile.py

src/PIL/Image.py

src/PIL/ImageFile.py

src/PIL/PcfFontFile.py

src/PIL/TiffImagePlugin.py

pyproject.toml

hugovk · 2023-02-23T19:38:36Z

The addition of pyproject.toml has changed the build process from setup.py-based to pyproject.toml-based, leading to failures on MinGW:

https://github.com/python-pillow/Pillow/actions/runs/4253283135/jobs/7404119827

akx · 2023-02-24T07:21:00Z

The addition of pyproject.toml has changed the build process from setup.py-based to pyproject.toml-based, leading to failures on MinGW

Ah, should've known there are still uncivilized OSes like that... 😁

Switched to ruff.toml instead 👍

akx · 2023-02-24T08:56:32Z

@hugovk CI is green now, and #6971 will fix up the local variable names.

Makefile

hugovk · 2023-02-25T19:48:16Z

Switched to ruff.toml instead 👍

Makes sense, we'll probably have to move to pyproject.toml at some point, but I agree it's out of scope for this. Please could you also add check-toml to .pre-commit-config.yaml?

.pre-commit-config.yaml

akx · 2023-02-26T13:09:34Z

@hugovk

Is there an issue at Ruff to limit Bandit by severity? If not, please could you open one?

I don't think there is - Ruff issues don't have the same sort of severity + confidence metrics Bandit's rules have.

If you want, I'm not against "belt and suspenders"ing here and keep Bandit also in the configuration...

aclark4life · 2023-02-26T13:35:32Z

Yeah lots not switch, just add ruff … if possible/practical to do so.

cclauss · 2023-03-17T08:09:24Z

If you do ruff --format=github then you get GitHub Annotations like:

Which some contributors might find quite helpful. The problem is that when you run in pre-commit, you are unsure if $CI is true or false and you probably do not want --format=github unless it is true.

I run a separate GitHub that is superfast (<10 seconds) to quicky add annotations to code changes.

Add a GitHub Action for Ruff internetarchive/openlibrary#7606

ruff.toml

.pre-commit-config.yaml

hugovk

We're using the real Bandit directly in pre-commit because it supports severities, so let's remove entirely from Ruff.

.ruff.toml

hugovk · 2023-03-30T11:38:15Z

Running a full pre-commit run is much faster (45.15 seconds of CPU time to 8.62 seconds on my machine)

I'm curious why the original is so slow, what machine do you have? How do the actual elapsed times compare?

cclauss · 2023-03-30T12:08:17Z

Running a full pre-commit run is much faster

Ruff linting the CPython codebase from scratch 0.29 sec.

akx · 2023-03-30T12:40:46Z

Running a full pre-commit run is much faster (45.15 seconds of CPU time to 8.62 seconds on my machine)

I was measuring CPU time on purpose – wallclock time isn't affected that much, but the old tooling isn't able to utilize all cores on my machine (macOS 12.6.3, 2,4 GHz 8-Core Intel Core i9, etc.).

hugovk · 2023-03-30T12:54:48Z

Ah right, still nice but less of a visible benefit. The old pre-commit speed isn't really a problem locally, and on CI it barely makes a difference compared to the full test matrix.

Ruff linting the CPython codebase from scratch 0.29 sec.

Impressive indeed, although CPython doesn't use Flake8 and this repo isn't CPython :)

hugovk · 2023-03-30T13:03:00Z

👉 The only rule that may not be yet supported by ruff that had been previously enabled is rst-backticks from pygrep-hooks.

That being said, this PR still enables more lints than had been in use from pygrep-hooks since it enables the whole PGH family.

https://beta.ruff.rs/docs/rules/#pygrep-hooks-pgh has four of the 10 hooks from https://github.com/pre-commit/pygrep-hooks:

PGH001 eval No builtin eval() allowed
PGH002 deprecated-log-warn warn is deprecated in favor of warning
PGH003 blanket-type-ignore Use specific rule codes when ignoring type issues
PGH004 blanket-noqa Use specific rule codes when using noqa

rst-backticks is useful, let's add it back via the original pygrep-hooks.

akx · 2023-10-30T18:19:48Z

@hugovk Done! I also have another branch that would stack on top of this to switch from black to the Ruff formatter but that's a different concern so not muddling these waters with that. FWIW, the diff there is tiny, mostly just some manual changes where ruff-format doesn't do # fmt: comments in statements like Black does:

 .pre-commit-config.yaml    |  7 +------
 Makefile                   |  3 +--
 Tests/check_jpeg_leaks.py  |  7 ++-----
 Tests/helper.py            |  4 ++--
 Tests/test_box_blur.py     | 40 ++++++++++++++++++++--------------------
 Tests/test_file_jpeg.py    |  8 ++------
 Tests/test_font_leaks.py   |  5 ++++-
 Tests/test_image_access.py |  4 +---
 Tests/test_image_filter.py | 28 +++++++++++++---------------
 setup.py                   |  9 ++++-----
 10 files changed, 50 insertions(+), 65 deletions(-)

hugovk · 2023-10-30T21:17:04Z

Thanks! Yeah, let's keep this for the linter for now.

How does this look with the UP031 fixes?

akx · 2023-10-30T21:19:59Z

The UP031 fixes would involve rewriting the "something %dx%d" % some_tuple style expressions in another way case by case - I'd leave them for an imminent follow up PR?

akx · 2023-11-01T11:20:57Z

@hugovk @radarhere Could this maybe be good to go now? 😁 I could follow up with a PR to remove the six UP031 noqas next...

radarhere · 2023-11-01T12:25:00Z

Makefile

 	python3 -m black --target-version py38 .
-	python3 -m isort .
+	python3 -c "import ruff" > /dev/null 2>&1 || python3 -m pip install ruff
+	python3 -m ruff --preview --fix .


If stability is a concern, then wouldn't it be better to not use --preview? I have to imagine that any features that were in preview back when this PR was created are no longer in preview.

Sans --preview:

warning: Selection `LOG` has no effect because the `--preview` flag was not included. warning: Selection `LOG` has no effect because the `--preview` flag was not included. warning: Selection `LOG` has no effect because the `--preview` flag was not included.

(LOG was added via #7421.)

I think we should go for stability and avoid --preview. Any idea when LOG will leave preview?

I don't really think there's a big huge stability issue here, since whoever ends up updating the Ruff version in this repo will probably look at what they're doing?

Also, the more happy users a preview rule has, the better its chances are to graduate, I think 😁

Let's remove --preview, I'd prefer to keep things stable and not spend time double checking potentially-unstable changes every month.

Removed --preview, and also the "LOG" family of lints, because it would otherwise warn about it on every --previewless invocation (that's not --quietened):

(Pillow) ~/b/Pillow (ruff) $ ruff . warning: Selection `LOG` has no effect because the `--preview` flag was not included. warning: Selection `LOG` has no effect because the `--preview` flag was not included. warning: Selection `LOG` has no effect because the `--preview` flag was not included. (Pillow) ~/b/Pillow (ruff) $ ruff --quiet . (Pillow) ~/b/Pillow (ruff) $

radarhere · 2023-11-01T12:26:48Z

I don't think I have strong opinions on the style changes seen here, so once @hugovk is ok with this, I'm happy for it to be merged.

hugovk

Please also delete .flake8 (recently added in #7484).

Co-authored-by: Andrew Murray <3112309+radarhere@users.noreply.github.com>

Co-authored-by: Andrew Murray <3112309+radarhere@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

akx · 2023-11-06T10:43:49Z

@hugovk Done, rebased, etc. :)

hugovk · 2023-11-06T11:37:44Z

Thanks, a followup for UP031 would be good. And I'd prefer to keep Black rather than the Ruff formatter.

See python-pillow/Pillow#6966 :)

## Summary See python-pillow/Pillow#6966 :) ## Test Plan Looked at the Markdown preview!

akx force-pushed the ruff branch from 30aa546 to 44b2a94 Compare February 23, 2023 12:20

akx marked this pull request as draft February 23, 2023 12:20

hugovk reviewed Feb 23, 2023

View reviewed changes

akx force-pushed the ruff branch from 44b2a94 to 1860c0b Compare February 23, 2023 13:51

akx commented Feb 23, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

akx marked this pull request as ready for review February 23, 2023 13:53

akx requested a review from hugovk February 23, 2023 13:59

hugovk added the Testing label Feb 23, 2023

akx force-pushed the ruff branch from 1860c0b to e35497f Compare February 24, 2023 07:20

akx force-pushed the ruff branch from e35497f to 126aede Compare February 24, 2023 07:29

akx mentioned this pull request Feb 24, 2023

Clarify some local variable names #6971

Merged

radarhere reviewed Feb 25, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

radarhere mentioned this pull request Feb 26, 2023

Removed TODO akx/Pillow#3

Merged

hugovk reviewed Feb 26, 2023

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

.pre-commit-config.yaml Outdated Show resolved Hide resolved

hugovk reviewed Mar 29, 2023

View reviewed changes

ruff.toml Outdated Show resolved Hide resolved

radarhere reviewed Mar 29, 2023

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

akx force-pushed the ruff branch from 49d1219 to 44ea099 Compare March 30, 2023 11:02

hugovk reviewed Mar 30, 2023

View reviewed changes

.ruff.toml Outdated Show resolved Hide resolved

.ruff.toml Outdated Show resolved Hide resolved

.ruff.toml Outdated Show resolved Hide resolved

akx force-pushed the ruff branch from 1ff5f57 to a471d8b Compare October 30, 2023 18:18

radarhere mentioned this pull request Oct 31, 2023

Removed setup.cfg since it has been moved to pyproject.toml akx/Pillow#5

Closed

akx force-pushed the ruff branch from a471d8b to d2f85ff Compare November 1, 2023 06:21

radarhere reviewed Nov 1, 2023

View reviewed changes

radarhere mentioned this pull request Nov 3, 2023

Removed preview akx/Pillow#6

Closed

akx force-pushed the ruff branch 4 times, most recently from c4c9f53 to ec98fb9 Compare November 3, 2023 22:09

hugovk reviewed Nov 6, 2023

View reviewed changes

akx and others added 4 commits November 6, 2023 12:42

editorconfig: specify 2-space indent for TOML

80406a7

Co-authored-by: Andrew Murray <3112309+radarhere@users.noreply.github.com>

Switch linting to ruff

de6c4b0

Co-authored-by: Andrew Murray <3112309+radarhere@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

Apply ruff autofixes

307d00b

Add noqas for UP031

9e615b6

akx force-pushed the ruff branch from ec98fb9 to 9e615b6 Compare November 6, 2023 10:43

hugovk merged commit 0660886 into python-pillow:main Nov 6, 2023
53 checks passed

akx deleted the ruff branch November 6, 2023 16:20

akx mentioned this pull request Nov 6, 2023

Fix up most noqas #7526

Merged

akx added a commit to akx/ruff that referenced this pull request Nov 6, 2023

Add Pillow to Ruff users

3281de8

See python-pillow/Pillow#6966 :)

akx mentioned this pull request Nov 6, 2023

Add Pillow to Ruff users astral-sh/ruff#8523

Merged

zanieb pushed a commit to astral-sh/ruff that referenced this pull request Nov 6, 2023

Add Pillow to Ruff users (#8523)

c07947b

## Summary See python-pillow/Pillow#6966 :) ## Test Plan Looked at the Markdown preview!

akx mentioned this pull request Nov 7, 2023

Switch formatting to ruff format #7531

Closed

hugovk mentioned this pull request Jan 20, 2024

Sort imports in Tests/ #7736

Merged

Move linting to ruff #6966

Move linting to ruff #6966

Conversation

akx commented Feb 23, 2023 • edited

hugovk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugovk commented Feb 23, 2023

akx commented Feb 24, 2023

akx commented Feb 24, 2023

hugovk commented Feb 25, 2023

akx commented Feb 26, 2023

aclark4life commented Feb 26, 2023

cclauss commented Mar 17, 2023 • edited

hugovk left a comment

Choose a reason for hiding this comment

hugovk commented Mar 30, 2023

cclauss commented Mar 30, 2023

akx commented Mar 30, 2023

hugovk commented Mar 30, 2023

hugovk commented Mar 30, 2023

akx commented Oct 30, 2023 • edited

hugovk commented Oct 30, 2023

akx commented Oct 30, 2023

akx commented Nov 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akx Nov 3, 2023 • edited

Choose a reason for hiding this comment

radarhere commented Nov 1, 2023

hugovk left a comment

Choose a reason for hiding this comment

akx commented Nov 6, 2023

hugovk commented Nov 6, 2023

akx commented Feb 23, 2023 •

edited

cclauss commented Mar 17, 2023 •

edited

akx commented Oct 30, 2023 •

edited

akx Nov 3, 2023 •

edited