Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-119169: Speed up os.[f]walk() #119186

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

barneygale
Copy link
Contributor

@barneygale barneygale commented May 19, 2024

Handle errors from os.scandir() and ScandirIterator similarly, which lets us loop over directory entries with for. In top-down mode, call os.path.join() at most once per iteration.

$ ./python -m timeit -s "import os" "list(os.walk('.', topdown=False))"
10 loops, best of 5: 30.1 msec per loop  # before
10 loops, best of 5: 28.7 msec per loop  # after
# --> 1.05x faster

$ ./python -m timeit -s "import os" "list(os.walk('.', topdown=True))"
10 loops, best of 5: 36.6 msec per loop  # before
10 loops, best of 5: 34.6 msec per loop  # after 
# --> 1.06x faster

Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`. In top-down mode, call
`os.path.join()` at most once per iteration.
@vstinner
Copy link
Member

cc @serhiy-storchaka

@barneygale barneygale marked this pull request as draft May 23, 2024 08:46
@barneygale
Copy link
Contributor Author

#119473 fixes the test failure.

@barneygale barneygale changed the title GH-119169: Speed up os.walk() GH-119169: Speed up os.[f]walk() May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance or resource usage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants