Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document formatter error shrinking #5915

Merged

Conversation

konstin
Copy link
Member

@konstin konstin commented Jul 20, 2023

Summary

Don't minimize files that don't match in the first place This adds a sanity check to the minimizer script that the
input matches the condition (e.g. unstable formatting). Otherwise we run
through all checks with the whole file, which is extremely slow. It's
more reasonable for downstream usage to write an empty string to the
output file instead.

@konstin
Copy link
Member Author

konstin commented Jul 20, 2023

Current dependencies on/for this PR:

This comment was auto-generated by Graphite.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 20, 2023

PR Check Results

Benchmark

Linux

group                                      main                                   pr
-----                                      ----                                   --
formatter/large/dataset.py                 1.00      9.8±0.02ms     4.2 MB/sec    1.00      9.8±0.02ms     4.2 MB/sec
formatter/numpy/ctypeslib.py               1.00   1905.9±5.15µs     8.7 MB/sec    1.00   1903.8±5.67µs     8.7 MB/sec
formatter/numpy/globals.py                 1.00    207.6±0.83µs    14.2 MB/sec    1.00    208.1±0.28µs    14.2 MB/sec
formatter/pydantic/types.py                1.00      4.2±0.01ms     6.1 MB/sec    1.00      4.2±0.00ms     6.0 MB/sec
linter/all-rules/large/dataset.py          1.00     13.5±0.06ms     3.0 MB/sec    1.00     13.5±0.04ms     3.0 MB/sec
linter/all-rules/numpy/ctypeslib.py        1.01      3.4±0.00ms     4.8 MB/sec    1.00      3.4±0.01ms     4.9 MB/sec
linter/all-rules/numpy/globals.py          1.01    370.9±0.81µs     8.0 MB/sec    1.00    368.7±1.00µs     8.0 MB/sec
linter/all-rules/pydantic/types.py         1.00      6.1±0.01ms     4.2 MB/sec    1.00      6.1±0.01ms     4.2 MB/sec
linter/default-rules/large/dataset.py      1.00      7.0±0.01ms     5.8 MB/sec    1.00      7.0±0.01ms     5.8 MB/sec
linter/default-rules/numpy/ctypeslib.py    1.00   1428.5±3.26µs    11.7 MB/sec    1.00   1422.1±3.82µs    11.7 MB/sec
linter/default-rules/numpy/globals.py      1.00    149.5±0.20µs    19.7 MB/sec    1.00    149.4±0.24µs    19.7 MB/sec
linter/default-rules/pydantic/types.py     1.00      3.1±0.00ms     8.2 MB/sec    1.00      3.1±0.01ms     8.2 MB/sec

Windows

group                                      main                                   pr
-----                                      ----                                   --
formatter/large/dataset.py                 1.00      9.0±0.07ms     4.5 MB/sec    1.05      9.4±0.11ms     4.3 MB/sec
formatter/numpy/ctypeslib.py               1.00  1724.6±13.52µs     9.7 MB/sec    1.03  1777.1±10.01µs     9.4 MB/sec
formatter/numpy/globals.py                 1.00    179.4±2.87µs    16.4 MB/sec    1.04    186.4±4.49µs    15.8 MB/sec
formatter/pydantic/types.py                1.00      3.8±0.03ms     6.7 MB/sec    1.04      4.0±0.04ms     6.4 MB/sec
linter/all-rules/large/dataset.py          1.00     12.4±0.09ms     3.3 MB/sec    1.00     12.3±0.07ms     3.3 MB/sec
linter/all-rules/numpy/ctypeslib.py        1.00      3.2±0.01ms     5.2 MB/sec    1.01      3.2±0.01ms     5.2 MB/sec
linter/all-rules/numpy/globals.py          1.00    338.1±3.74µs     8.7 MB/sec    1.01    342.1±2.32µs     8.6 MB/sec
linter/all-rules/pydantic/types.py         1.00      5.5±0.03ms     4.6 MB/sec    1.00      5.5±0.03ms     4.6 MB/sec
linter/default-rules/large/dataset.py      1.00      6.6±0.04ms     6.2 MB/sec    1.01      6.7±0.03ms     6.1 MB/sec
linter/default-rules/numpy/ctypeslib.py    1.00   1303.7±7.40µs    12.8 MB/sec    1.02   1323.3±7.28µs    12.6 MB/sec
linter/default-rules/numpy/globals.py      1.00    138.8±1.04µs    21.3 MB/sec    1.02    142.1±1.60µs    20.8 MB/sec
linter/default-rules/pydantic/types.py     1.00      2.9±0.01ms     8.9 MB/sec    1.01      2.9±0.02ms     8.8 MB/sec

@konstin konstin force-pushed the Don_t_minimize_files_that_don_t_match_in_the_first_place branch from b0c9d41 to de466b9 Compare July 20, 2023 13:02
@konstin konstin requested a review from MichaReiser July 20, 2023 15:21
@konstin konstin force-pushed the Don_t_minimize_files_that_don_t_match_in_the_first_place branch from 6d4ca8e to 963a352 Compare July 20, 2023 15:49
@konstin konstin added the formatter Related to the formatter label Jul 20, 2023
Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation looks good to me.

I recommend splitting the other changes into their own PR, it otherwise reading the Git history. Who would suspect that a PR Documenting... contains code changes.

Splitting should be straightforward as each change is its own commit. You can split the PR by using gt branch split

@konstin
Copy link
Member Author

konstin commented Jul 21, 2023

I recommend splitting the other changes into their own PR, it otherwise reading the Git history. Who would suspect that a PR Documenting... contains code changes.

good point, i didn't consider how this interacts with squash merge

**Summary** This adds a sanity check to the minimizer script that the
input matches the condition (e.g. unstable formatting). Otherwise we run
through all checks with the whole file, which is extremely slow. It's
more reasonable for downstream usage to write an empty string to the
output file instead. It also prints the minimized code in the end
@konstin konstin force-pushed the Don_t_minimize_files_that_don_t_match_in_the_first_place branch from 963a352 to 547a176 Compare July 21, 2023 08:24
@konstin konstin merged commit b56e8ad into main Jul 21, 2023
17 checks passed
@konstin konstin deleted the Don_t_minimize_files_that_don_t_match_in_the_first_place branch July 21, 2023 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
formatter Related to the formatter
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants