Use Tokens from parsed type annotation or parsed source
#11740
Merged
+159
−31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
MichaReiser
approved these changes
Jun 5, 2024
dhruvmanila
force-pushed
the
dhruv/parsed-tokens
branch
from
1d129f0 to
f8260cb
Compare
|
I've added test cases for all affected rules as well. Even though the type annotations themselves aren't valid, Ruff shouldn't panic. |
dhruvmanila
added a commit
that referenced
this pull request
Jun 5, 2024
## Summary This PR is a follow-up to #11740 to restrict access to the `Parsed` output by replacing the `parsed` API function with a more specific one. Currently, that is `comment_ranges` but the linked PR exposes a `tokens` method. The main motivation is so that there's no way to get an incorrect information from the checker. And, it also encapsulates the source of the comment ranges and the tokens itself. This way it would become easier to just update the checker if the source for these information changes in the future. ## Test Plan `cargo insta test`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a bug where the checker would require the tokens for an invalid offset w.r.t. the source code.
Taking the source code from the linked issue as an example:
Now, this isn't really a valid type annotation but that's what this PR is fixing. Regardless of whether it's valid or not, Ruff shouldn't panic.
The checker would visit the parsed type annotation (
0.0is 64) and try to detect any violations. Certain rule logic requests the tokens for the same but it would fail because the lexer would only have theStringtoken considering original source code. This worked before because the lexer was invoked again for each rule logic.The solution is to store the parsed type annotation on the checker if it's in a typing context and use the tokens from that instead if it's available. This is enforced by creating a new API on the checker to get the tokens.
But, this means that there are two ways to get the tokens via the checker API. I want to restrict this in a follow-up PR (#11741) to only expose
tokensandcomment_rangesas methods and restrict access to the parsed source code.fixes: #11736
Test Plan
F632rule and update the snapshot