Add --unstable flag (psf#4096)

CHANGES.md

@@ -34,6 +34,14 @@ changes:
 - Fix incorrect formatting of certain async statements (#3609)
 - Allow combining `# fmt: skip` with other comments (#3959)
 
+There are already a few improvements in the `--preview` style, which are slated for the
+2025 stable style. Try them out and
+[share your feedback](https://github.com/psf/black/issues). In the past, the preview
+style has included some features that we were not able to stabilize. This year, we're
+adding a separate `--unstable` style for features with known problems. Now, the
+`--preview` style only includes features that we actually expect to make it into next
+year's stable style.
+
 ### Stable style
 
 <!-- Changes that affect Black's stable style -->
@@ -53,6 +61,12 @@ release:
 
 <!-- Changes that affect Black's preview style -->
 
+- Add `--unstable` style, covering preview features that have known problems that would
+  block them from going into the stable style. Also add the `--enable-unstable-feature`
+  flag; for example, use
+  `--enable-unstable-feature hug_parens_with_braces_and_square_brackets` to apply this
+  preview style throughout 2024, even if a later Black release downgrades the feature to
+  unstable (#4096)
 - Format module docstrings the same as class and function docstrings (#4095)
 - Fix crash when using a walrus in a dictionary (#4155)
 - Fix unnecessary parentheses when wrapping long dicts (#4135)
@@ -67,6 +81,9 @@ release:
 - Fix symlink handling, properly catch and ignore symlinks that point outside of root
   (#4161)
 - Fix cache mtime logic that resulted in false positive cache hits (#4128)
+- Remove the long-deprecated `--experimental-string-processing` flag. This feature can
+  currently be enabled with `--preview --enable-unstable-feature string_processing`.
+  (#4096)
 
 ### Packaging
 

docs/faq.md

@@ -41,9 +41,10 @@ other tools, such as `# noqa`, may be moved by _Black_. See below for more detai
 Stable. _Black_ aims to enforce one style and one style only, with some room for
 pragmatism. See [The Black Code Style](the_black_code_style/index.md) for more details.
 
-Starting in 2022, the formatting output will be stable for the releases made in the same
-year (other than unintentional bugs). It is possible to opt-in to the latest formatting
-styles, using the `--preview` flag.
+Starting in 2022, the formatting output is stable for the releases made in the same year
+(other than unintentional bugs). At the beginning of every year, the first release will
+make changes to the stable style. It is possible to opt in to the latest formatting
+styles using the `--preview` flag.
 
 ## Why is my file not formatted?
 

docs/the_black_code_style/current_style.md

@@ -449,6 +449,12 @@ file that are not enforced yet but might be in a future version of the formatter
 _Black_ will normalize line endings (`\n` or `\r\n`) based on the first line ending of
 the file.
 
+### Form feed characters
+
+_Black_ will retain form feed characters on otherwise empty lines at the module level.
+Only one form feed is retained for a group of consecutive empty lines. Where there are
+two empty lines in a row, the form feed is placed on the second line.
+
 ## Pragmatism
 
 Early versions of _Black_ used to be absolutist in some respects. They took after its

docs/the_black_code_style/future_style.md

@@ -1,117 +1,44 @@
 # The (future of the) Black code style
 
-```{warning}
-Changes to this document often aren't tied and don't relate to releases of
-_Black_. It's recommended that you read the latest version available.
-```
-
-## Using backslashes for with statements
-
-[Backslashes are bad and should be never be used](labels/why-no-backslashes) however
-there is one exception: `with` statements using multiple context managers. Before Python
-3.9 Python's grammar does not allow organizing parentheses around the series of context
-managers.
-
-We don't want formatting like:
-
-```py3
-with make_context_manager1() as cm1, make_context_manager2() as cm2, make_context_manager3() as cm3, make_context_manager4() as cm4:
-    ...  # nothing to split on - line too long
-```
-
-So _Black_ will, when we implement this, format it like this:
-
-```py3
-with \
-     make_context_manager1() as cm1, \
-     make_context_manager2() as cm2, \
-     make_context_manager3() as cm3, \
-     make_context_manager4() as cm4 \
-:
-    ...  # backslashes and an ugly stranded colon
-```
-
-Although when the target version is Python 3.9 or higher, _Black_ uses parentheses
-instead in `--preview` mode (see below) since they're allowed in Python 3.9 and higher.
-
-An alternative to consider if the backslashes in the above formatting are undesirable is
-to use {external:py:obj}`contextlib.ExitStack` to combine context managers in the
-following way:
-
-```python
-with contextlib.ExitStack() as exit_stack:
-    cm1 = exit_stack.enter_context(make_context_manager1())
-    cm2 = exit_stack.enter_context(make_context_manager2())
-    cm3 = exit_stack.enter_context(make_context_manager3())
-    cm4 = exit_stack.enter_context(make_context_manager4())
-    ...
-```
-
-(labels/preview-style)=
-
 ## Preview style
 
 Experimental, potentially disruptive style changes are gathered under the `--preview`
 CLI flag. At the end of each year, these changes may be adopted into the default style,
 as described in [The Black Code Style](index.md). Because the functionality is
 experimental, feedback and issue reports are highly encouraged!
 
-### Improved string processing
-
-_Black_ will split long string literals and merge short ones. Parentheses are used where
-appropriate. When split, parts of f-strings that don't need formatting are converted to
-plain strings. User-made splits are respected when they do not exceed the line length
-limit. Line continuation backslashes are converted into parenthesized strings.
-Unnecessary parentheses are stripped. The stability and status of this feature is
-tracked in [this issue](https://github.com/psf/black/issues/2188).
-
-### Improved line breaks
-
-For assignment expressions, _Black_ now prefers to split and wrap the right side of the
-assignment instead of left side. For example:
-
-```python
-some_dict[
-    "with_a_long_key"
-] = some_looooooooong_module.some_looooooooooooooong_function_name(
-    first_argument, second_argument, third_argument
-)
-```
-
-will be changed to:
+In the past, the preview style included some features with known bugs, so that we were
+unable to move these features to the stable style. Therefore, such features are now
+moved to the `--unstable` style. All features in the `--preview` style are expected to
+make it to next year's stable style; features in the `--unstable` style will be
+stabilized only if issues with them are fixed. If bugs are discovered in a `--preview`
+feature, it is demoted to the `--unstable` style. To avoid thrash when a feature is
+demoted from the `--preview` to the `--unstable` style, users can use the
+`--enable-unstable-feature` flag to enable specific unstable features.
 
-```python
-some_dict["with_a_long_key"] = (
-    some_looooooooong_module.some_looooooooooooooong_function_name(
-        first_argument, second_argument, third_argument
-    )
-)
-```
+Currently, the following features are included in the preview style:
 
-### Improved parentheses management
+- `hex_codes_in_unicode_sequences`: normalize casing of Unicode escape characters in
+  strings
+- `unify_docstring_detection`: fix inconsistencies in whether certain strings are
+  detected as docstrings
+- `hug_parens_with_braces_and_square_brackets`: more compact formatting of nested
+  brackets ([see below](labels/hug-parens))
+- `no_normalize_fmt_skip_whitespace`: whitespace before `# fmt: skip` comments is no
+  longer normalized
 
-For dict literals with long values, they are now wrapped in parentheses. Unnecessary
-parentheses are now removed. For example:
+(labels/unstable-features)=
 
-```python
-my_dict = {
-    "a key in my dict": a_very_long_variable
-    * and_a_very_long_function_call()
-    / 100000.0,
-    "another key": (short_value),
-}
-```
+The unstable style additionally includes the following features:
 
-will be changed to:
+- `string_processing`: split long string literals and related changes
+  ([see below](labels/string-processing))
+- `wrap_long_dict_values_in_parens`: add parentheses to long values in dictionaries
+  ([see below](labels/wrap-long-dict-values))
+- `multiline_string_handling`: more compact formatting of expressions involving
+  multiline strings ([see below](labels/multiline-string-handling))
 
-```python
-my_dict = {
-    "a key in my dict": (
-        a_very_long_variable * and_a_very_long_function_call() / 100000.0
-    ),
-    "another key": short_value,
-}
-```
+(labels/hug-parens)=
 
 ### Improved multiline dictionary and list indentation for sole function parameter
 
@@ -185,6 +112,46 @@ foo(
 )
 ```
 
+(labels/string-processing)=
+
+### Improved string processing
+
+_Black_ will split long string literals and merge short ones. Parentheses are used where
+appropriate. When split, parts of f-strings that don't need formatting are converted to
+plain strings. User-made splits are respected when they do not exceed the line length
+limit. Line continuation backslashes are converted into parenthesized strings.
+Unnecessary parentheses are stripped. The stability and status of this feature is
+tracked in [this issue](https://github.com/psf/black/issues/2188).
+
+(labels/wrap-long-dict-values)=
+
+### Improved parentheses management in dicts
+
+For dict literals with long values, they are now wrapped in parentheses. Unnecessary
+parentheses are now removed. For example:
+
+```python
+my_dict = {
+    "a key in my dict": a_very_long_variable
+    * and_a_very_long_function_call()
+    / 100000.0,
+    "another key": (short_value),
+}
+```
+
+will be changed to:
+
+```python
+my_dict = {
+    "a key in my dict": (
+        a_very_long_variable * and_a_very_long_function_call() / 100000.0
+    ),
+    "another key": short_value,
+}
+```
+
+(labels/multiline-string-handling)=
+
 ### Improved multiline string handling
 
 _Black_ is smarter when formatting multiline strings, especially in function arguments,
@@ -297,13 +264,51 @@ s = (  # Top comment
 )
 ```
 
-=======
+## Potential future changes
+
+This section lists changes that we may want to make in the future, but that aren't
+implemented yet.
+
+### Using backslashes for with statements
+
+[Backslashes are bad and should be never be used](labels/why-no-backslashes) however
+there is one exception: `with` statements using multiple context managers. Before Python
+3.9 Python's grammar does not allow organizing parentheses around the series of context
+managers.
+
+We don't want formatting like:
+
+```py3
+with make_context_manager1() as cm1, make_context_manager2() as cm2, make_context_manager3() as cm3, make_context_manager4() as cm4:
+    ...  # nothing to split on - line too long
+```
+
+So _Black_ will, when we implement this, format it like this:
+
+```py3
+with \
+     make_context_manager1() as cm1, \
+     make_context_manager2() as cm2, \
+     make_context_manager3() as cm3, \
+     make_context_manager4() as cm4 \
+:
+    ...  # backslashes and an ugly stranded colon
+```
+
+Although when the target version is Python 3.9 or higher, _Black_ uses parentheses
+instead in `--preview` mode (see below) since they're allowed in Python 3.9 and higher.
 
-### Form feed characters
+An alternative to consider if the backslashes in the above formatting are undesirable is
+to use {external:py:obj}`contextlib.ExitStack` to combine context managers in the
+following way:
 
-_Black_ will now retain form feed characters on otherwise empty lines at the module
-level. Only one form feed is retained for a group of consecutive empty lines. Where
-there are two empty lines in a row, the form feed will be placed on the second line.
+```python
+with contextlib.ExitStack() as exit_stack:
+    cm1 = exit_stack.enter_context(make_context_manager1())
+    cm2 = exit_stack.enter_context(make_context_manager2())
+    cm3 = exit_stack.enter_context(make_context_manager3())
+    cm4 = exit_stack.enter_context(make_context_manager4())
+    ...
+```
 
-_Black_ already retained form feed literals inside a comment or inside a string. This
-remains the case.
+(labels/preview-style)=

docs/the_black_code_style/index.md

@@ -42,9 +42,11 @@ _Black_:
   enabled by newer Python language syntax as well as due to improvements in the
   formatting logic.
 
-- The `--preview` flag is exempt from this policy. There are no guarantees around the
-  stability of the output with that flag passed into _Black_. This flag is intended for
-  allowing experimentation with the proposed changes to the _Black_ code style.
+- The `--preview` and `--unstable` flags are exempt from this policy. There are no
+  guarantees around the stability of the output with these flags passed into _Black_.
+  They are intended for allowing experimentation with proposed changes to the _Black_
+  code style. The `--preview` style at the end of a year should closely match the stable
+  style for the next year, but we may always make changes.
 
 Documentation for both the current and future styles can be found:
 

docs/usage_and_configuration/black_as_a_server.md

@@ -62,6 +62,12 @@ The headers controlling how source code is formatted are:
 - `X-Preview`: corresponds to the `--preview` command line flag. If present and its
   value is not an empty string, experimental and potentially disruptive style changes
   will be used.
+- `X-Unstable`: corresponds to the `--unstable` command line flag. If present and its
+  value is not an empty string, experimental style changes that are known to be buggy
+  will be used.
+- `X-Enable-Unstable-Feature`: corresponds to the `--enable-unstable-feature` flag. The
+  contents of the flag must be a comma-separated list of unstable features to be
+  enabled. Example: `X-Enable-Unstable-Feature: feature1, feature2`.
 - `X-Fast-Or-Safe`: if set to `fast`, `blackd` will act as _Black_ does when passed the
   `--fast` command line flag.
 - `X-Python-Variant`: if set to `pyi`, `blackd` will act as _Black_ does when passed the

docs/usage_and_configuration/the_basics.md

@@ -144,9 +144,34 @@ magic trailing comma is ignored.
 
 #### `--preview`
 
-Enable potentially disruptive style changes that may be added to Black's main
-functionality in the next major release. Read more about
-[our preview style](labels/preview-style).
+Enable potentially disruptive style changes that we expect to add to Black's main
+functionality in the next major release. Use this if you want a taste of what next
+year's style will look like.
+
+Read more about [our preview style](labels/preview-style).
+
+There is no guarantee on the code style produced by this flag across releases.
+
+#### `--unstable`
+
+Enable all style changes in `--preview`, plus additional changes that we would like to
+make eventually, but that have known issues that need to be fixed before they can move
+back to the `--preview` style. Use this if you want to experiment with these changes and
+help fix issues with them.
+
+There is no guarantee on the code style produced by this flag across releases.
+
+#### `--enable-unstable-feature`
+
+Enable specific features from the `--unstable` style. See
+[the preview style documentation](labels/unstable-features) for the list of supported
+features. This flag can only be used when `--preview` is enabled. Users are encouraged
+to use this flag if they use `--preview` style and a feature that affects their code is
+moved from the `--preview` to the `--unstable` style, but they want to avoid the thrash
+from undoing this change.
+
+There are no guarantees on the behavior of these features, or even their existence,
+across releases.
 
 (labels/exit-code)=
 

pyproject.toml

@@ -16,10 +16,11 @@ extend-exclude = '''
   | profiling
 )/
 '''
-# We use preview style for formatting Black itself. If you
-# want stable formatting across releases, you should keep
-# this off.
-preview = true
+# We use the unstable style for formatting Black itself. If you
+# want bug-free formatting, you should keep this off. If you want
+# stable formatting across releases, you should also keep `preview = true`
+# (which is implied by this flag) off.
+unstable = true
 
 # Build system information and other project-specific configuration below.
 # NOTE: You don't need this in your own Black configuration.