Dev/black

[lillekemiker]

What type of PR is this? (check all applicable)

• Refactor
• Feature (dev feature and reformatting)
• Bug Fix
• Optimization
• Documentation Update

Have you discussed this change with the InvokeAI team?

• Yes
• No, because:

Description

Introducing black to the code base as a first step towards this:
https://github.com/invoke-ai/InvokeAI/discussions/3721

Related Tickets & Documents

• Related Issue #
• Closes #

QA Instructions, Screenshots, Recordings

Added/updated tests?

• Yes
• No : Not applicable

[optional] Are there any post deployment tasks we need to perform?

All active branches will be affected by this and will need to be updated.

This PR adds a new github workflow for black as well as config for pre-commit hooks to those who wish to use it

[psychedelicious]

Thanks for working on the actions.

We have some large PRs and branches pending and need to get to a point where we can freeze PRs, merge all the big kahunas, and only then can we do a massive formatting sweep. We've been waiting for months to do this but there was way too much changing to break history with a bit format PR.

I believe we settled on base black config but with line length of 100 or 120.

[lillekemiker]

@psychedelicious No worries, I totally get that. I personally find it harder to expand on existing files right now because I have to turn off my auto formaters, but that’s me. Just say the word when you want to pull the trigger. Git makes it look harder than it really is to make this branch mergable. Basically it’s as easy as copying all the files from main and running black on them. No need to actually work out merge conflicts.

[psychedelicious]

I hear you, I disabled my autoformat on save too 😮‍💨 the concern was more that with a big formatting commit, we create a boundary that makes diffs hard to reason about, and the code was in very high flux. Getting close now!

[lillekemiker]

Is there a reason you want to do 100-120 character lines? I always liked the standard 88 (leaving an 8 character buffer from the classic 80).

[psychedelicious]

Our monitors are generally bigger than punchcards or hardware terminals now (excellent talk btw), though of course it's just a preference.

The frontend code will be getting the 100 or 120 width treatment soon too

[lillekemiker]

Yeah, I totally get the issue with one big commit that reformats the whole code base and creates a diff wall. If you want, I’ll happily break it up into individual files if that helps. Especially if we can identify files that are not being changed right now. I just didn’t want to spam you with PR’s without first discussing.

[psychedelicious]

I believe we are pretty close to enabling the formatting but I'll request @ebr and @brandonrising for their opinions

[lstein]

Yeah, I totally get the issue with one big commit that reformats the whole code base and creates a diff wall. If you want, I’ll happily break it up into individual files if that helps. Especially if we can identify files that are not being changed right now. I just didn’t want to spam you with PR’s without first discussing.

I actually did the "reformat everything with black" thing last winter during the 2.3 release and one of the gotchas was that "git blame" ended up blaming me for every line of code in the repository. After some searching I found a technique that preserves blame across the black boundary, applied it, and it improved the situation. Do you know about it? If not, I'll try to retrace my steps.

[lstein]

Is there a reason you want to do 100-120 character lines? I always liked the standard 88 (leaving an 8 character buffer from the classic 80).

I prefer the longer lines. 100 is pretty comfortable for me.

[lstein]

This is great but I'd like to wait on this until the pace of change slows down. About a week, I'd say.

[lillekemiker]

@lstein there’s something here about how to get around the git blame issue:
https://black.readthedocs.io/en/stable/guides/introducing_black_to_your_project.html

I’m not entirely clear on how it works based on this link alone, but maybe one big squash merge and then add the commit hash to ‘.git-blame-ignore-revs’?
Does that look like what you did?

[mikeage]

@lstein , I think you were looking for this: #2855

(note that if you do squash merges, instead of merge commits, you'll need to do a new PR to add the squashed commit of this PR)

[mikeage]

Since you're using pre-commit, may I recommend the following action instead:

      - uses: pre-commit/action@v3.0.0

[lillekemiker]

This is pretty cool. I did not know this existed. Personally I'm a little torn on using it because there's not much gained by using it and it leaves us dependent on an externally managed action. There's also a slight difference in what is being run in the precommit hooks right now. For the sake of speed, they only run black on what is being committed, while the current GHA is running black on everything.
If others prefer using this action, though, I don't feel super strongly about it.

[mikeage]

I'm not a dev here, so please remember that I cannot speak for the project at all :-). I understand what you're saying about external actions, although it is pretty official. Perhaps at least run pre-commit run -a instead of black directly? It reduces the chance that at some point someone might want to change a parameter or something and will need to remember to sync in both places.

IIRC, I think the action it can also be configured to add comments with places that were missed if you pass a token (but possibly not on a PR from a fork). It also includes a git diff after it runs, so instead of just a binary "failed", you can see what it fixed, which can be helpful for new contributors who aren't aware of the coding conventions.

[zopieux]

My 2 cents:

• I agree that using and enforcing a formatter/linter is great practice, in general but especially so for OSS development accepting external contributors. It is very cumbersome having to deal with each project's individual coding style. This should be taken care of by computers, not busy humans – I couldn't care less whether this repo is using 80, 88 or 120 char lines. A nicely formatted code base feels way more welcoming to contributions. I'm currently refraining from non-trivial contributions to the backend, even though I would like to, because of the current situation. I cannot be bothered to disable code formatters in 2023, they're integral to my workflow. Big 👍 to the intent behind this PR.
• I wasn't aware of the blame-skip trick. This is awesome, but if it's too impractical still, note that an alternative to the massive whole-repo reformat is to use a formatter that is able to only apply formatting to git-modified lines. This way, historical code will slowly converge to being formatted correctly as actual logic changes are contributed. Sadly PSF black does not yet support this.

[lillekemiker]

@zopieux - Actually by running black with the pre-commit hooks only, black would only be applied to the files changed in the given commit.
However, I personally think that doing it a little bit at a time would make things even less transpararent. Consider a PR that makes a 1 line change to a 500 line file, except black changes all 500 lines. That gets confusing quickly. And with blame you can always go back in time more than one change, so if you see blame pointing to a commit called "apply black", you just hit the button one more time.

[psychedelicious]

I'm calling this out on discord to hopefully grab everyone's attention (and any strongly-dissenting opinions).

This will also close #2522.

[hipsterusername]

Can we update this to 120 characters? If we can, I think there is no reason not to get this merged in today @ebr @brandonrising

[lstein]

Do it! Just be careful to use the blame-skip trick

…

On Thu, Jul 27, 2023 at 9:16 AM Kent Keirsey ***@***.***> wrote: ***@***.**** commented on this pull request. Can we update this to 120 characters? If we can, I think there is no reason not to get this merged in today @ebr <https://github.com/ebr> @brandonrising <https://github.com/brandonrising>

[ebr]

@lillekemiker could I ask you to please rebase on current state of main, apply the 120 char line length change, and force-push one more time? I will make sure this is reviewed today and will try to merge ASAP.

[lstein]

After this goes in, the owners of all open PRs will need to manually black-ify and push their branches using the correct line length. They won't have access to the updated `pyproject.toml` so it'll have to be done manually. Aside from line length, are there any other non-default settings we should be aware of? It would be helpful to put the appropriate black config file directives into a comment here for quick reference. Also, what's the most efficient way of resolving the conflicts that have already accumulated? Perhaps we need another PR off of main for the formatting changes, and then apply this one to get the pre-commit hooks and pyproject settings.

[ebr]

black is opinionated (for better or worse), so the line length is the most significant configuration option.

Any in-flight PRs would have to apply black --line-length 120 .

There are also a couple of other options, e.g. --skip-magic-trailing-comma ("Don't use trailing commas as a reason to split lines"), but I'm not sure it's worth our time tweaking that right now.

Re: conflict resolution - @lillekemiker just force-pushed the latest changes, so I will make sure to review it ASAP so we can merge it before any more conflicts are introduced. I will add a commit with the blame-skip trick.

[lillekemiker]

@lillekemiker could I ask you to please rebase on current state of main, apply the 120 char line length change, and force-push one more time? I will make sure this is reviewed today and will try to merge ASAP.

Done!

[ebr]

@lillekemiker: did you use the black --fast option to get baseinvocation.py, commands.py and compel.py to reformat? without --fast, it fails to parse the AST. And/or, which version of black do you have installed?
we also need it added to dev dependencies (I can do that)

Nevermind - not an issue with the latest black

[ebr]

@lillekemiker did you have to pass --include installer/lib/* ? these files are currently being ignored by black due to lib/ present in .gitignore. Seems like we should be able to remove that without consequence, but i'm wondering how you got Black to format these files in the first place.

[lillekemiker]

@lillekemiker did you have to pass --include installer/lib/* ? these files are currently being ignored by black due to lib/ present in .gitignore. Seems like we should be able to remove that without consequence, but i'm wondering how you got Black to format these files in the first place.

I found that if I run black using pre-commit run -a, then it formats everything. I'm guessing either git or pre-commit does the file fetching instead of black when done that way.

[ebr]

LGTM!

🚨 I will add added another commit into .git-blame-ignore-revs before merging. Suggest we merge this as a rebase rather than a squash or merge commit, such that the SHA of the black commit remains unchanged.

@lstein - any current in-flight PRs will need to pip install black followed by black --line-length 120 . (also @Millu please take note). But I don't think we can avoid conflict resolution entirely.

Please approve, and we'll merge if this is acceptable.

[mikeage]

. Suggest we merge this as a rebase rather than a squash or merge commit, such that the SHA of the black commit remains unchanged.
.

Both a rebase and a squash will change the SHA. Only a merge commit, or another PR with the SHA of the squash or rebased commit.

[lstein]

LGTM

[ebr]

. Suggest we merge this as a rebase rather than a squash or merge commit, such that the SHA of the black commit remains unchanged.
.

Both a rebase and a squash will change the SHA. Only a merge commit, or another PR with the SHA of the squash or rebased commit.

right, good point. Let's try a merge commit, though it means the blacked commit won't be on main. Worst case, we'll add another commit with the correct SHA

[zopieux]

Let's go! Can finally hack on that codebase :-)