Nested quotes in f-strings

[ujan-r]

Describe the bug

Python 3.12 introduced some updates to the f-string grammar (see PEP 701), including—among other things—quote reuse: strings nested within an f-string can now use the same quote character as the f-string itself. Black doesn't recognize this, inadvertently changing the semantics of some programs containing f-strings.

To Reproduce

For example, take this code:

# file.py
from urllib.parse import quote


dest = 'unescaped callback'
redirect_uri = f'127.0.0.1:443/path?dest={quote(dest, safe='')}'

print(redirect_uri)

$ python file.py
127.0.0.1:443/path?dest=unescaped%20callback

After formatting with black file.py --skip-string-normalization --target-version py312, we get:

# file.py
from urllib.parse import quote


dest = 'unescaped callback'
redirect_uri = f'127.0.0.1:443/path?dest={quote(dest, safe=' ')}'

print(redirect_uri)

$ python file.py
127.0.0.1:443/path?dest=unescaped callback

Black has silently modified the code! It interprets the f-string as two separate strings, inserting a space between them.

We only get an error when the preview flag is enabled (black file.py --preview --skip-string-normalization --target-version py312). The resulting error is:

INTERNAL ERROR: Black produced invalid code: expected argument value expression (<unknown>, line 6).)

Here is the generated log file:

  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/black/__init__.py", line 1444, in assert_equivalent
    dst_ast = parse_ast(dst)
              ^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/black/parsing.py", line 140, in parse_ast
    raise SyntaxError(first_error)
# file.py
from urllib.parse import quote


dest = 'unescaped callback'
redirect_uri = f'127.0.0.1:443/path?dest={quote(dest, safe=)}}'

print(redirect_uri)

Expected behavior

The original code (file.py) shouldn't be modified.

Environment

$ # macOS 13.5
$ black --version
black, 23.11.0 (compiled: no)
Python (CPython) 3.12.0

[hauntsaninja]

Thanks, duplicate of #3746