Implement autoformat capabilities

[charliermarsh]

This issue is the public kickoff for Ruff's autoformatting effort.

A few comments on how I'm thinking about the desired end-state:

• The goal is to enable users to replace Black with Ruff. So, ideally, new projects could come in and replace pyflakes, pycodestyle, isort, and black with a single tool (Ruff).
• As with the rest of Ruff, autoformatting will be entirely optional. You can continue to use Ruff alongside Black. You can also use Ruff just as an autoformatter.
• I'd like to adhere to Black's formatting choices as much as possible, but exact 1:1 parity with Black is not a goal. I'm comfortable deviating when we can make a compelling case, but I'd like that case to be based on arguments that are unrelated to code style itself (e.g., simpler implementation).
• I'd like to enable slightly more configuration than is possible with Black (see: Replacing black and blue? #813), using Prettier as a model. Prettier allows you to specify the line-length, tab width, quote style, trailing comma behavior, and one or two other stylistic features. I think it's reasonable to support at least line length, indentation, and quote style, so I plan to start there. (It's not a goal to be as flexible as, say, Rustfmt.)

With big projects like this, I work best by taking some time to hack on possible implementations, so I'd like to do some free exploration before opening up the issue to other contributors. But I'd love to hear feedback on the above (and any tips you may have) as I start working on this.

[ichard26]

While I'm not sure how to feel about seeing Black being replaced as one of its maintainers (I'm kinda attached TBH!) I'd be happy to answer any questions you might have about Black. Feel free to ask about its code style and its implementation.

I barely know anything about Ruff, but from what I'm hearing, it's probably going to be the future of code quality tooling. There's no way we (Black) can ever compete on speed or being an all-in-one solution. I'm not going to contribute code or whatever, but I do feel like I have a responsibility to our users to build a better formatter even if that means helping arguably a competitor.

Obviously I do have my own Black-influenced opinions on "good code style" so don't expect me to suggest significant deviations from Black's style. I couldn't do that to my project, that's too much of a betrayal :)

[charliermarsh]

Thank you for the thoughtful message, and for the kind offer -- I really appreciate it.

To be honest, I was a little hesitant to post this Issue, since I too am a fan and long-time user of Black, and I don't want to send the wrong message by offering an alternative. But, it's just a very natural extension of what Ruff is already doing, and it's something that comes up constantly when talking to users.

(Also: while I do want Ruff to be a viable all-in-one solution, I also want it to be incrementally useable and incrementally adoptable -- so, e.g., even if we were to complete this, my intention is that it'd always be possible to use Black alongside Ruff. Ruff has the same relationship with isort right now.)

I'll keep your offer in mind as I get the ball rolling here :)

[LefterisJP]

Hey @charliermarsh!

Can I chime in and ask for maybe something different? Black is really good at what it's doing and it's also fast (enough). What I see as a really good way for ruff to go for auto-formatting would be the way of yapf .... but better. So faster and more customizable. Black exists and we don't really need another black.

What python lacks is a customizable auto-formatter that can be customized by each project to come as close to their preferred style as possile. Only thing close to that is yapf.

yapf is an auto-formatter tool for python based on the ideas behind clang format. So the idea is to make it as customizable as possible for projects to apply their rules via autoformat.

The current problems with yapf are:

1. It's not as customizable as it can be.
2. It's super slow for big projects. Takes 4-5 minutes for rotki.
3. I think it's not in a very active maintenance mode

I really believe ruff can shine here and would love to see it become this kind of fast and customizable auto-format tool.

[warsaw]

Great to see your thoughts here, and I'll watch closely with a hope of contributing if I have the bandwidth, and at least testing things out when you have stuff to try.

Some thoughts:

• Quote style is probably the number 1 reason why blue even exists. It's not good enough to just not change quote style; blue deliberately chooses single quotes over double quotes wherever possible, for reasons we think are good, but in keeping with the spirit of this ticket, I won't argue that right now.
• We think blue does a better job of maintaining spacing between code and any right hanging comments, although not a perfect job.
• Blue is really difficult to maintain because it monkeypatches black (to gain from all the really great work that black and its maintainers have done!). Monkeypatching of course is extremely fragile. Without proper APIs in black (and I don't fault black for not supporting them), blue will always have a difficult time keeping up.
• Blue has (had?) better configuration options than black, but even there it's problematic. Having to keep up with the internal implementation details of two external projects now is pretty daunting.
• I personally don't care about the wide features of YAPF. For me and my projects (both open source and for $work), black gets us almost all the way there. I remember sending Łukasz a list of like 10-11 bullet points about black's choices back in the early days, and he refuted every one of them. Which of course is perfectly okay! 😄 - in fact, I've accepted and/or come around on all of them but the few that I still feel blue makes better choices on.

All that to say that IMO, it's worth having some choice on formatting styles, blue tries to make that possible, but maintaining it is difficult, the black developers are doing a fantastic job, and I still would like to see ruff provide a good, fast alternative!

[WhyNotHugo]

There's one thing that black won't do and I think ruff would be able to address: E501. When a line is too long, black will often ignore that. For example:

hello = "This is a really long string that's really over 80 characters wide, and black won't do anything about it"

It would be kinda weird if ruff doesn't autoformat this and complains that it's over 88 characters long. The solution to the above should be:

hello = (
    "This is a really long string that's really over 80 characters wide, and "
    "black won't do anything about it"
)

[charliermarsh]

@WhyNotHugo - Black can fix that if you use black --preview :) Hopefully Ruff can support this too.

[timabbott]

This project is very exciting for Zulip! To explain some of the practical benefit, one of the main practical limitations the size of a Python file is that Black takes about a 200ms per 1K line of code, which becomes a very noticeable lag when running that as an on-save hook in an editor. If that part was at Ruff speeds, I'd expect there to be essentially no autoformatter related lag when saving files in an editor, which would be a visible improvement to our Python development experience.

So being able to replace Black with Ruff, as we have with most of our other Python linters would be amazing. I support the design decision to support slightly more configurability than Black has -- while I don't recall the details, I remember our migration to Black being delayed for a long time because they didn't have the options to avoid some things that we considered style regressions.

[charliermarsh]

@timabbott - That's great to hear and I appreciate you chiming in :)

[charliermarsh]

Brief update: I'm continuing to work on this :)

It's going well! Several tests from the Black suite are passing, more are close-to-passing (I'd like to quantify this soon, but I haven't done string normalization yet, and that causes most tests to fail even if the rest of the formatting is correct).

Still a lot of work and cases left to handle. But my current thinking is that I'll OSS it as soon as I'm confident that it can handle all syntax including comment preservation, even if the style drifts a bit over a few releases as we improve compatibility.

[ddahan]

I have to say I really love the idea of using a single tool for everything related to code quality.
I actually never understood the border between linting and formatting.

On the other side, I think Black is amazing, and I'm not sure Python community needs another new formatter right now, especially if it is very similar to Black.


So I was wondering, how to resolve this contradiction?


This may be a naive idea, but could we include black inside Ruff as a dependency? Like every time a new black version is out, Charlie Marsh magic script can convert this black to "Rust Black" and embed it in the next Ruff.

This way, users only install a single tool, but behind the scene, they are using Black, with the exact same behaviour, and exact same config (but with Rust speed !)

I would love this option, but would totally understand if other users prefer something else.
 Thanks for the amazing work anyway!

[ofek]

but with Rust speed

That won't happen magically without a new implementation in Rust

On the other side, I think Black is amazing, and I'm not sure Python community needs another new formatter right now, especially if it is very similar to Black.

So I was wondering, how to resolve this contradiction?

Whenever this happens I'm assuming it would gradually replace Black in the long term

[charliermarsh]

Current work-in-progress implementation has been merged into main -- now working off main as we iterate and increase test completion rate.

If you're interested in following along, #2883 has some details on how it works and what it supports (and doesn't) thus far.

[smackesey]

Excited for this, and chiming in here on two major benefits I see coming from this as someone working on a large monorepo (Dagster:

• Speed: ruff is way faster than black on our ~2500 python file repo. Very noticeable when running from the command line.
• Config resolution flexibility: there are places where we are forced to duplicate black config because not every package in our codebase uses the same line length settings (we use shorter line length for code snippets that will be rendered to docs). Black doesn't support config inheritance. This also sometimes causes problems for us when running black in pre-commit due to the way black resolves config files.

[ambv]

I don't think you need my opinion here but since I already started typing, here it goes!

First of all, it doesn't ruffle any feathers in Blackland that this effort here is happening. Ruff is a qualitative improvement over previous developer experience tooling for Python, one that we can't really touch with Python in terms of performance, so I'm really curious where this will go.

In fact, I like the requirements spelled out here. In particular, trying to follow the Black style while acknowledging that a 1:1 implementation is probably not feasible. Charlie, if you identify some differences that you think are clear improvements, we can also consider moving toward those in Black. That way migration from Black to Rufffmt (sorry, I had to 😅) would be more seamless.

I'm saying this because my initial urge to create the auto-formatter didn't come from a need to be the king of the Python formatting hill. Rather, I wanted a tool that formats things consistently with a set of rules that I can explain in prose, and recognize in the output. I contributed to YAPF before and tried to adopt it at Facebook where I worked at the time. This failed because the rich configurability and clever "dissatisfaction optimization" approach of YAPF caused its results to be sometimes hard to understand. No configuration file we tried saved us from very suboptimal edge cases, and when changing the weights solved a particular issue, other issues popped up somewhere else. Therefore, I just wanted a tool that does one thing and does it well.

I guess I'm softly advising against creating a fully configurable tool here. I'm not saying you shouldn't let people configure more things than Black. Our tool was created when YAPF was already a well-known and mature piece of software. The goal was to end bikeshedding and to a large extent we did. Clearly, normalizing string prefixes is still dividing people after 5 years of the tool existing so I have to conclude that going there was a mistake. I still stand behind the decision for Black not to indent in any other way than 4 spaces. It's literally the first concrete rule in PEP 8 and I think sticking to it made it easier for everybody to read and copy-paste Python code wherever it comes from.

Anyway, to re-iterate: if the end result of this effort is that the community will migrate to Rufffmt as a tool, I will be a happy man... as long as it doesn't reignite bikeshedding over formatting minutiae. The tools won't (and can't) be 1:1 interchangeable but as long as the broad strokes are same-ish, all is good. You see, not even Black is 1:1 the same between code style editions (please run with --preview if you don't mind some additional churn in exchange for getting better style faster). So I have no qualms about Ruffmt being an alternative "edition" of Black.

To be clear, Charlie, Ruff is your tool and it's ultimately your call to do whatever you want with it. I won't lose sleep over any decision. My comment here is mostly meant as encouragement and to extend the possibility of collaboration on the resulting style. I'm happy to see @ichard26 essentially said as much early on here. He's right. Don't be a stranger, we're happy to talk if you ever need anything!

[zellyn]

I always saw black as inspired by gofmt: the appeal is in not having configurable options, or as few as possible. (Although it sounds like it might be worth adding some configurable options internally, but not exposed in the commandline arguments or config files, for the blue folks to use, so they can stop monkey-patching!)

[silverwind]

A 2-space indent option would be a requirement for me to use it. prettier also has it.

[thejcannon]

I'll throw in my 2c.

Line length, spaces, and quotes are ok to configure so long as the overall style remains the same as black (especially considering blacks style attempts to minimize diffs).

The important thing is that the Python community is slowly converging on one style, meaning code across organizations, repos, and geographical distance all looks the same. That's a huge win for developers, since it means less brain power parsing code stylistically and more power parsing it semantically.

[smackesey]

Adding to the peanut gallery: the entire idea that it's a feature that black lacks config options is misguided. It's also inconsistent with ruff, which has literally hundreds of knobs to turn-- which, from this "no flexibility is a good thing" perspective, is a dangerous bikeshedding minefield! And yet ruff is great and people seem happy with it. Why haven't software projects everywhere slowed to a crawl over arguments about import sorting style or whether to enable lint rule X?

It's because the real time-suck has never been decisions over formatting style. It is enforcing a given format style. However, this problem is independent of how configurable a formatting tool is. As long as the tool runs well in CI and is integrated into dev workflows, no "manual' formatting enforcement need occur.

I think that black is a good tool, but its rise also coincided with widespread adoption of tools for in-editor formatting enforcement and use of a formatter check in CI. Aside from these factors, its success comes from simple messaging, good marketing, and polished implementation. I suspect the success has almost nothing to do with the fact that it has few config options, even though the tool claims this as its philosophy.

Indeed IMO black would be better tool with more config options, so I think that's the direction ruff should lean. By all means ruff should limit flexibility in the name of performance or keeping implementation bug-free and low-maintenance, but the "let's limit options so teams never even have the chance to argue about it" argument is silly.

[ofek]

I enjoy meta-commentary just as much as the next person but please starting right now let's keep this specifically about asking for certain configuration options as well as the maintainers discussing the merits of those options and timelines for implementations.

I would simply unsubscribe but this functionality is incredibly important to me so I want to stay up to date on happenings related to it.

Thanks 🙂

[charliermarsh]

Thanks everyone. I've been reading all of these and I appreciate the input.

I will chime in to confirm that our goal for the initial release is to implement Black-compatible formatting with a limited set of known deviations and a couple of configuration options that don't exist in Black (e.g., indentation style, quote style, line ending). That is, we don't plan to make the formatter any more configurable in the initial release.

But I'll also confirm that whether we expose additional configuration options in future releases is an unsettled question.

Speaking for myself and not for the project (to the degree that that's possible...), my biggest concern around configuration / flexibility is the increased maintenance and testing burden. Implementing a formatter already requires handling an enormous combination of expression, statement, and comment placements. Each additional configuration option adds another degree of freedom that we need to maintain and test in perpetuity. So even if we do decide to make the formatter more configurable, there will likely always be some tradeoffs to make in balancing flexibility with performance, maintenance burden, etc.

(Separately, again speaking for myself -- this time as a user -- I find myself really liking the configuration options exposed by rustfmt :))

[jni]

Thanks for listening @charliermarsh! 😊 Would you suggest we make new issues as feature requests for specific configuration options? Then each config option can be evaluated independently.

[LeonarddeR]

I'm really exited to see support for tabs has landed.
Now when talking about configuration, the only thing that that disappoints me about Black and conflicts with Pep8 is the missing extra level of indentation for function parameters, e.g. Black does:

def foo(
    bar
):

instead of expected by pep8:

def foo(
        bar
):

[ofek]

Black does not force extra indentation like that, I'm not sure what you mean

[jsh9]

I'm really exited to see support for tabs has landed. Now when talking about configuration, the only thing that Black doesn't do that disappoints me and conflicts with Pep8 is the extra level of indentation for function parameters, e.g.:

def foo(
    bar
):

instead of

def foo(
        bar
):

Hey @LeonarddeR, welcome to the club (of wanting this 8-space indentation feature)! That is my main motivation of forking Black. You can check out my fork here: https://github.com/jsh9/cercis

I do sincerely hope that the Ruff Formatter will support this, or at least make it configurable.

[jsh9]

Black does not force extra indentation like that, I'm not sure what you mean

@ofek , I think @LeonarddeR meant the opposite: Black uses 4-space indentation at function definition, rather than 8 spaces. And a bunch of people would much prefer 8-space indentation (only at function definition, not in other places).

[ofek]

Oh I misinterpreted, I thought they were saying that Black erroneously adds extra indentation rather than them saying they wish it did so 😅

I personally very much dislike that, but to each their own.

[akx]

I also dislike the 8-space indent - why is it suddenly 8 there where it's 4 everywhere else? Anyway, yeah, to each their own. 😄

[LeonarddeR]

@ofek I'm sorry for the confusion, I've edited my comment above to clarify what I meant.
@jsh9 thanks for the warm welcome. In fact I don't care much myself, but I'm a contributor of a project that follows PEP8 in this particular case, and I think we'd prefer to stick with that.

[jsh9]

I also dislike the 8-space indent - why is it suddenly 8 there where it's 4 everywhere else? Anyway, yeah, to each their own. 😄

The reason I prefer 8-space indentation at function definition is the following:

If the indentation is only 4 spaces, the argument list aligns completely with the function name, making it hard for me to visually distinguish them. Here's an example:

def my_function(
    my_function_arg_1,
    my_function_arg_2,
    my_function_arg_3,
    my_function_arg_4,
):
    pass

Some people may argue that there are syntax highlighting and function names have a different color. But many times I need to look at plain-text logs without any syntax highlighting.

I do understand other people's desire for consistency (4-space indentation everywhere). That's OK; people can have different preferences and priorities.

The once-and-for-all solution, in my humble opinion, is to just add a configurable nob for this, rather than forcing either camp to the other camp.

[warsaw]

I don't much like the 8 space indents for function definitions either, but there aren't a lot of good options. Maybe use Guido's time machine and change def to define? 😛

One thing python-mode does is support "hanging" arguments like so:

def my_function(arg_1,
                arg_2,
                arg_3,
                ):

but only if the open paren is not the last non-whitespace character on the line.

[ofek]

That particular code example makes me want to not have any configuration at all 😆

[jsh9]

I don't much like the 8 space indents for function definitions either, but there aren't a lot of good options. Maybe use Guido's time machine and change def to define? 😛

Yeah, func or function would be much better.

One thing python-mode does is support "hanging" arguments like so:

def my_function(arg_1,
                arg_2,
                arg_3,
                ):

but only if the open paren is not the last non-whitespace character on the line.

This "hanging "style can lead to maintenanbility problems in the future. For example, if in the future people want to rename my_function into something such as func1 (especially with the auto refactoring provided by IDEs), the code then becomes:

def func1(arg_1,
                arg_2,
                arg_3,
                ):

It looks very ugly.

[Samreay]

I'm not sure how many people here are subscribed like me, with the intention of hearing updates from the developers about the status the implementation work, rather than this much larger conversation about possible formatter customisations.

Is there any chance that sort of conversation could be moved into the dedicated GitHub discussion for formatter feedback instead of the implementation issue?

Obviously not a dev for the project, so unsure what works best for them - I'm only coming from this as a "I can no longer see the messages from the devs because there are ten thousand emails about config in the way"

[charliermarsh]

Thanks @Samreay -- that's a good suggestion, and sorry that I haven't done a better job of steering discussion out of the issue. (I don't mind reading through the comments personally, but it's easy for me to forget that there are others subscribed here for updates :))

Humble request to put any future feedback in the formatter discussion (be it feedback on configuration, current code style, future code style, etc.), and we'll keep this issue focused on announcements and project updates.

[zanieb]

We're excited to announce the formatter is in Beta — you can see more details in the changelog.

[Samreay]

@zanieb amazing! Congrats on the release! Is the pre-commit repo going to be getting a sweet 0.1.2 release as well, I'd love to just stick ruff everywhere now!

[JayOfferdahl]

Switched over my organization last week -- it was a breeze. Thanks for all the work here!

[zanieb]

@Samreay the pre-commit repo is automatically tagged on each release of Ruff, so you don't need to wait for GitHub releases to be published in order to use the latest version. I've added the corresponding GitHub releases anyway though.

[charliermarsh]

Now that the formatter is out in Beta, I'm calling this complete :)

[alexisthual]

@WhyNotHugo - Black can fix that if you use black --preview :) Hopefully Ruff can support this too.

Out of curiosity, is it possible to split long strings with ruff already? 😊
I could not find it in the documentation.

[charliermarsh]

@alexisthual -- Not yet, but we plan to support it.

[giampaolo]

Out of curiosity, is it possible to split long strings with ruff already?

Not yet, but we plan to support it.

For the record, this is the only reason which made me go back from ruff format to black after a while. Use case: my editor (Sublime Text) executes both black and ruff on save ("fix" operation, not "check"). The ability to automatically split long lines on the fly, while I type code, instead of later via CLI, is very very valuable.

I'm not sure if this specific black option (split long lines) has been introduced recently, but I achieve it via this conf:

[tool.black]
preview = true
# https://black.readthedocs.io/en/stable/the_black_code_style/future_style.html
enable-unstable-feature = ["string_processing"]

Black advertises this as an "unstable feature", so I understand it may be premature to add it into ruff at this time. I assume though (but it's just a mere assumption) that black devs' intention is to make this the default once they deem the feature reasonably stable.

[giampaolo]

I assume though (but it's just a mere assumption) that black devs' intention is to make this the default once they deem the feature reasonably stable.

It turns out my assumption was wrong, see related black ticket: psf/black#4208. :)