[syntax-errors] Star expression in index before Python 3.11 #16544
Conversation
Summary -- This PR detects tuple unpacking expressions in index/subscript expressions before Python 3.11. Test Plan -- New inline tests
ntBre
added
parser
|
Summary -- This is closely related to (and stacked on) #16544 and detects star annotations in function definitions. I initially called the variant `StarExpressionInAnnotation` to mirror `StarExpressionInIndex`, but I realized it's not really a "star expression" in this position and renamed it. `StarAnnotation` seems in line with the PEP. Test Plan -- Two new inline tests. It looked like there was pretty good existing coverage of this syntax, so I just added simple examples to test the version cutoff.
| @@ -636,6 +664,9 @@ impl Display for UnsupportedSyntaxError { | |||
| UnsupportedSyntaxErrorKind::TypeParamDefault => { | |||
| "Cannot set default type for a type parameter" | |||
| } | |||
| UnsupportedSyntaxErrorKind::StarExpressionInIndex => { | |||
| "Cannot use star expression in index" | |||
There was a problem hiding this comment.
Nit: Maybe in index operations to align with the terminology used in the PEP
| 2 | lst[*index] # simple index | ||
| | ^^^^^^ Syntax Error: Cannot use star expression in index on Python 3.10 (syntax was added in Python 3.11) |
There was a problem hiding this comment.
I'm not sure this is correct. I haven't checked if it is a parse or semantic error but Python 3.12 gives me:
>>> [][*list]
<python-input-1>:1: SyntaxWarning: list indices must be integers or slices, not tuple; perhaps you missed a comma?
[][*list]
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
[][*list]
~~^^^^^^^
TypeError: list indices must be integers or slices, not tuple
Which is also what I read from the grammar change: It only allows star expressions for slices except the first
slices:
| slice !','
| ','.(slice | starred_expression)+ [',']
There was a problem hiding this comment.
Yeah, I think the check needs to be moved up in the if branch which is a pure tuple case.
This is a TypeError and is not relevant to the parser:
$ bat _.py
File: _.py
1 ~ [][*data]
$ uv run --python=3.12 --no-project python -m ast _.py
Module(
body=[
Expr(
value=Subscript(
value=List(elts=[], ctx=Load()),
slice=Tuple(
elts=[
Starred(
value=Name(id='data', ctx=Load()),
ctx=Load())],
ctx=Load()),
ctx=Load()))],
type_ignores=[])There was a problem hiding this comment.
I may have made this test case a bit misleading by naming the container lst, but as @dhruvmanila points out with the valid AST, this is syntactically valid, and it's not a type error for a container like a numpy array or another container that can be validly indexed by a tuple:
>>> import numpy as np
>>> a = np.array([[1, 2, 3], [4, 5, 6]])
>>> t = [1, 2]
>>> a[*t]
np.int64(6)It's not a parser or compiler error, it's only a runtime TypeError if the container doesn't support tuple indexing.
I believe the grammar in the reference also supports this, albeit with different terminology from the PEP:
slicing ::= primary "[" slice_list "]"
slice_list ::= slice_item ("," slice_item)* [","]
slice_item ::= expression | proper_slice
proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]
lower_bound ::= expression
upper_bound ::= expression
stride ::= expression
slice_item is either an expression or a proper_slice with multiple parts.
There was a problem hiding this comment.
Sorry, I think I mis-understood the change here. I think what Brent has here is correct. The example and I and Micha have pointed out is a runtime error and should throw a syntax error for Python version older than 3.11.
| 2 | lst[*index] # simple index | ||
| | ^^^^^^ Syntax Error: Cannot use star expression in index on Python 3.10 (syntax was added in Python 3.11) |
There was a problem hiding this comment.
Yeah, I think the check needs to be moved up in the if branch which is a pure tuple case.
This is a TypeError and is not relevant to the parser:
$ bat _.py
File: _.py
1 ~ [][*data]
$ uv run --python=3.12 --no-project python -m ast _.py
Module(
body=[
Expr(
value=Subscript(
value=List(elts=[], ctx=Load()),
slice=Tuple(
elts=[
Starred(
value=Name(id='data', ctx=Load()),
ctx=Load())],
ctx=Load()),
ctx=Load()))],
type_ignores=[])Summary -- This is closely related to (and stacked on) #16544 and detects star annotations in function definitions. I initially called the variant `StarExpressionInAnnotation` to mirror `StarExpressionInIndex`, but I realized it's not really a "star expression" in this position and renamed it. `StarAnnotation` seems in line with the PEP. Test Plan -- Two new inline tests. It looked like there was pretty good existing coverage of this syntax, so I just added simple examples to test the version cutoff.
* main: [red-knot] Use `try_call_dunder` for augmented assignment (#16717) [red-knot] Document current state of attribute assignment diagnostics (#16746) [red-knot] Case sensitive module resolver (#16521) [red-knot] Very minor simplification of the render tests (#16759) [syntax-errors] Unparenthesized assignment expressions in sets and indexes (#16404) ruff_db: add a new diagnostic renderer ruff_db: add `context` configuration red_knot: plumb through `DiagnosticFormat` to the CLI ruff_db: add concise diagnostic mode [syntax-errors] Star annotations before Python 3.11 (#16545) [syntax-errors] Star expression in index before Python 3.11 (#16544) Ruff 0.11.0 (#16723) [red-knot] Preliminary tests for typing.Final (#15917) [red-knot] fix: improve type inference for binary ops on tuples (#16725) [red-knot] Assignments to attributes (#16705) [`pygrep-hooks`]: Detect file-level suppressions comments without rul… (#16720) Fallback to requires-python in certain cases when target-version is not found (#16721)
Summary
This PR detects tuple unpacking expressions in index/subscript expressions before Python 3.11.
Test Plan
New inline tests