Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

anyio.Path.iterdir blocks in Python 3.13 #873

Closed
2 tasks done
cbornet opened this issue Feb 20, 2025 · 1 comment · Fixed by #874
Closed
2 tasks done

anyio.Path.iterdir blocks in Python 3.13 #873

cbornet opened this issue Feb 20, 2025 · 1 comment · Fixed by #874
Labels
bug Something isn't working

Comments

@cbornet
Copy link
Contributor

cbornet commented Feb 20, 2025

Things to check first

  • I have searched the existing issues and didn't find my bug already reported there

  • I have checked that my bug is still present in the latest release

AnyIO version

HEAD

Python version

3.13

What happened?

Python 3.13 introduced a breaking change on how pathlib.Path.iterdir behaves.
iterdir now materializes the list of files eagerly. See python/cpython#129871 .
The reason for this change was to raise OS errors directly and not at the first iteration : python/cpython#78722

anyio.Path.iterdir is calling pathlib.Path.iterdir directly (not in a thread)

gen = self._path.iterdir()

and probably users would expect it not to block.

Some possibilities to fix would be:

  • Reimplement anyio.Path.iterdir to not use pathlib.Path.iterdir and use os.scandir directly without materializing the iterator. In this case, there will be a small difference with pathlib.Path.iterdir that OS errors will only happen when we start iterating.
  • Make anyio.Path.iterdir async which is a breaking change.

LMK what you think. I can do a PR.

How can we reproduce the bug?

N/A

@cbornet cbornet added the bug Something isn't working label Feb 20, 2025
@agronholm
Copy link
Owner

I would prefer not to introduce any breaking changes, but instead reimplement the async iterator in a way that defers any I/O until the first iteration, like how it used to work.

cbornet added a commit to cbornet/anyio that referenced this issue Feb 20, 2025

Verified

This commit was signed with the committer’s verified signature.
apostasie Apostasie
cbornet added a commit to cbornet/anyio that referenced this issue Feb 20, 2025
cbornet added a commit to cbornet/anyio that referenced this issue Feb 20, 2025
cbornet added a commit to cbornet/anyio that referenced this issue Feb 20, 2025
agronholm added a commit that referenced this issue Feb 21, 2025
Fixes #873.

---------

Co-authored-by: Alex Grönholm <alex.gronholm@nextday.fi>
github-merge-queue bot pushed a commit to DataDog/orchestrion that referenced this issue Mar 25, 2025
…ns/codecov-cli with 2 updates (#581)

Bumps the python-dependencies group in /.github/actions/codecov-cli with
2 updates: [codecov-cli](https://github.com/codecov/codecov-cli) and
[anyio](https://github.com/agronholm/anyio).

Updates `codecov-cli` from 10.2.0 to 10.3.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/codecov/codecov-cli/releases">codecov-cli's
releases</a>.</em></p>
<blockquote>
<h2>Release v10.3.0</h2>
<p>Autogenerated for v10.3.0. Created for <a
href="https://redirect.github.com/codecov/codecov-cli/pull/666">codecov/codecov-cli#666</a></p>
<h2>What's Changed</h2>
<ul>
<li>Release 10.2.0 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/652">codecov/codecov-cli#652</a></li>
<li>Add create commit to <code>empty-upload</code> command by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/654">codecov/codecov-cli#654</a></li>
<li>Release 10.2.1 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/655">codecov/codecov-cli#655</a></li>
<li>Pin requirements.txt to use test-results-parser==0.5.4 by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/656">codecov/codecov-cli#656</a></li>
<li>Revert &quot;Release 10.2.1&quot; by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/657">codecov/codecov-cli#657</a></li>
<li>Release 10.2.1 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/658">codecov/codecov-cli#658</a></li>
<li>Hide and mark ATS-related commands as deprecated by <a
href="https://github.com/Swatinem"><code>@​Swatinem</code></a> in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/662">codecov/codecov-cli#662</a></li>
<li>fix: add list_relevant_files to NoVersioningSystem by <a
href="https://github.com/thomasrockhu-codecov"><code>@​thomasrockhu-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/665">codecov/codecov-cli#665</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/codecov/codecov-cli/compare/v10.2.0...v10.3.0">https://github.com/codecov/codecov-cli/compare/v10.2.0...v10.3.0</a></p>
<h2>Release v10.2.1</h2>
<p>Autogenerated for v10.2.1. Created for <a
href="https://redirect.github.com/codecov/codecov-cli/pull/658">codecov/codecov-cli#658</a></p>
<h2>What's Changed</h2>
<ul>
<li>Release 10.1.1 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/645">codecov/codecov-cli#645</a></li>
<li>add --toc-recurse-submodules argument by <a
href="https://github.com/matt-codecov"><code>@​matt-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/650">codecov/codecov-cli#650</a></li>
<li>Release 10.2.0 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/652">codecov/codecov-cli#652</a></li>
<li>Add create commit to <code>empty-upload</code> command by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/654">codecov/codecov-cli#654</a></li>
<li>Release 10.2.1 by <a
href="https://github.com/codecov-releaser"><code>@​codecov-releaser</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/655">codecov/codecov-cli#655</a></li>
<li>Pin requirements.txt to use test-results-parser==0.5.4 by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/656">codecov/codecov-cli#656</a></li>
<li>Revert &quot;Release 10.2.1&quot; by <a
href="https://github.com/michelletran-codecov"><code>@​michelletran-codecov</code></a>
in <a
href="https://redirect.github.com/codecov/codecov-cli/pull/657">codecov/codecov-cli#657</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/codecov/codecov-cli/compare/v10.1.1...v10.2.1">https://github.com/codecov/codecov-cli/compare/v10.1.1...v10.2.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/codecov/codecov-cli/commit/97e90fa9fff4ab9d4f39217b320f5375bf754aba"><code>97e90fa</code></a>
Prepare release 10.3.0</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/f2b75b3183b4bd1e203f415e5e16b82cc5e62bb9"><code>f2b75b3</code></a>
fix: add list_relevant_files to NoVersioningSystem (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/665">#665</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/6533a8c9d8b9f336aaaef8919459bdf5013ed852"><code>6533a8c</code></a>
Hide and mark ATS-related commands as deprecated (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/662">#662</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/8364881cde4cd111bc2caca137a97e57f3ef1543"><code>8364881</code></a>
Prepare release 10.2.1 (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/658">#658</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/29a7964f9b47541a79e6ec5dee97eb81b73c7cf7"><code>29a7964</code></a>
Revert &quot;Prepare release 10.2.1 (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/655">#655</a>)&quot;
(<a
href="https://redirect.github.com/codecov/codecov-cli/issues/657">#657</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/852ad85e9ef0f2f613ac9bdf2a5825e79786af4a"><code>852ad85</code></a>
Pin requirements.txt to use test-results-parser==0.5.4 (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/656">#656</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/aed3a0473e7bc9b5b62d1753e59de0cab0a9c9ec"><code>aed3a04</code></a>
Prepare release 10.2.1 (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/655">#655</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/30cba3fd3b047bb4c1f2295201568ca458a74cc4"><code>30cba3f</code></a>
Add create commit to <code>empty-upload</code> command (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/654">#654</a>)</li>
<li><a
href="https://github.com/codecov/codecov-cli/commit/079f01ba314c9ed0fdf7f5ef0699fe17be9c250f"><code>079f01b</code></a>
Prepare release 10.2.0 (<a
href="https://redirect.github.com/codecov/codecov-cli/issues/652">#652</a>)</li>
<li>See full diff in <a
href="https://github.com/codecov/codecov-cli/compare/v10.2.0...v10.3.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `anyio` from 4.8.0 to 4.9.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/agronholm/anyio/releases">anyio's
releases</a>.</em></p>
<blockquote>
<h2>4.9.0</h2>
<ul>
<li>Added async support for temporary file handling (<a
href="https://redirect.github.com/agronholm/anyio/issues/344">#344</a>;
PR by <a href="https://github.com/11kkw"><code>@​11kkw</code></a>)</li>
<li>Added 4 new fixtures for the AnyIO <code>pytest</code> plugin:
<ul>
<li><code>free_tcp_port_factory</code>: session scoped fixture returning
a callable that generates unused TCP port numbers</li>
<li><code>free_udp_port_factory</code>: session scoped fixture returning
a callable that generates unused UDP port numbers</li>
<li><code>free_tcp_port</code>: function scoped fixture that invokes the
<code>free_tcp_port_factory</code> fixture to generate a free TCP port
number</li>
<li><code>free_udp_port</code>: function scoped fixture that invokes the
<code>free_udp_port_factory</code> fixture to generate a free UDP port
number</li>
</ul>
</li>
<li>Added <code>stdin</code> argument to
<code>anyio.run_process()</code> akin to what
<code>anyio.open_process()</code>,
<code>asyncio.create_subprocess()</code>,
<code>trio.run_process()</code>, and <code>subprocess.run()</code>
already accept (PR by <a
href="https://github.com/jmehnle"><code>@​jmehnle</code></a>)</li>
<li>Added the <code>info</code> property to <code>anyio.Path</code> on
Python 3.14</li>
<li>Changed <code>anyio.getaddrinfo()</code> to ignore (invalid) IPv6
name resolution results when IPv6 support is disabled in Python</li>
<li>Changed <code>EndOfStream</code> raised from
<code>MemoryObjectReceiveStream.receive()</code> to leave out the
<code>AttributeError</code> from the exception chain which was merely an
implementation detail and caused some confusion</li>
<li>Fixed traceback formatting growing quadratically with level of
<code>TaskGroup</code> nesting on asyncio due to exception chaining when
raising <code>ExceptionGroups</code> in <code>TaskGroup.__aexit__</code>
(<a
href="https://redirect.github.com/agronholm/anyio/issues/863">#863</a>;
PR by <a
href="https://github.com/tapetersen"><code>@​tapetersen</code></a>)</li>
<li>Fixed <code>anyio.Path.iterdir()</code> making a blocking call in
Python 3.13 (<a
href="https://redirect.github.com/agronholm/anyio/issues/873">#873</a>;
PR by <a href="https://github.com/cbornet"><code>@​cbornet</code></a>
and <a
href="https://github.com/agronholm"><code>@​agronholm</code></a>)</li>
<li>Fixed <code>connect_tcp()</code> producing cyclic references in
tracebacks when raising exceptions (<a
href="https://redirect.github.com/agronholm/anyio/pull/809">#809</a>; PR
by <a
href="https://github.com/graingert"><code>@​graingert</code></a>)</li>
<li>Fixed <code>anyio.to_thread.run_sync()</code> needlessly holding on
to references of the context, function, arguments and others until the
next work item on asyncio (PR by <a
href="https://github.com/Wankupi"><code>@​Wankupi</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/agronholm/anyio/blob/master/docs/versionhistory.rst">anyio's
changelog</a>.</em></p>
<blockquote>
<h1>Version history</h1>
<p>This library adheres to <code>Semantic Versioning 2.0
&lt;http://semver.org/&gt;</code>_.</p>
<p><strong>4.9.0</strong></p>
<ul>
<li>
<p>Added async support for temporary file handling
(<code>[#344](agronholm/anyio#344)
&lt;https://github.com/agronholm/anyio/issues/344&gt;</code>_; PR by <a
href="https://github.com/11kkw"><code>@​11kkw</code></a>)</p>
</li>
<li>
<p>Added 4 new fixtures for the AnyIO <code>pytest</code> plugin:</p>
<ul>
<li><code>free_tcp_port_factory</code>: session scoped fixture returning
a callable that
generates unused TCP port numbers</li>
<li><code>free_udp_port_factory</code>: session scoped fixture returning
a callable that
generates unused UDP port numbers</li>
<li><code>free_tcp_port</code>: function scoped fixture that invokes the
<code>free_tcp_port_factory</code> fixture to generate a free TCP port
number</li>
<li><code>free_udp_port</code>: function scoped fixture that invokes the
<code>free_udp_port_factory</code> fixture to generate a free UDP port
number</li>
</ul>
</li>
<li>
<p>Added <code>stdin</code> argument to <code>anyio.run_process()</code>
akin to what
<code>anyio.open_process()</code>,
<code>asyncio.create_subprocess_…()</code>,
<code>trio.run_process()</code>,
and <code>subprocess.run()</code> already accept (PR by <a
href="https://github.com/jmehnle"><code>@​jmehnle</code></a>)</p>
</li>
<li>
<p>Added the <code>info</code> property to <code>anyio.Path</code> on
Python 3.14</p>
</li>
<li>
<p>Changed <code>anyio.getaddrinfo()</code> to ignore (invalid) IPv6
name resolution results when
IPv6 support is disabled in Python</p>
</li>
<li>
<p>Changed <code>EndOfStream</code> raised from
<code>MemoryObjectReceiveStream.receive()</code> to leave
out the <code>AttributeError</code> from the exception chain which was
merely an implementation
detail and caused some confusion</p>
</li>
<li>
<p>Fixed traceback formatting growing quadratically with level of
<code>TaskGroup</code>
nesting on asyncio due to exception chaining when raising
<code>ExceptionGroups</code>
in <code>TaskGroup.__aexit__</code>
(<code>[#863](agronholm/anyio#863)
&lt;https://github.com/agronholm/anyio/issues/863&gt;</code>_; PR by <a
href="https://github.com/tapetersen"><code>@​tapetersen</code></a>)</p>
</li>
<li>
<p>Fixed <code>anyio.Path.iterdir()</code> making a blocking call in
Python 3.13
(<code>[#873](agronholm/anyio#873)
&lt;https://github.com/agronholm/anyio/issues/873&gt;</code>_; PR by <a
href="https://github.com/cbornet"><code>@​cbornet</code></a> and
<a href="https://github.com/agronholm"><code>@​agronholm</code></a>)</p>
</li>
<li>
<p>Fixed <code>connect_tcp()</code> producing cyclic references in
tracebacks when raising
exceptions (<code>[#809](agronholm/anyio#809)
&lt;https://github.com/agronholm/anyio/pull/809&gt;</code>_; PR by <a
href="https://github.com/graingert"><code>@​graingert</code></a>)</p>
</li>
<li>
<p>Fixed <code>anyio.to_thread.run_sync()</code> needlessly holding on
to references of the
context, function, arguments and others until the next work item on
asyncio
(PR by <a
href="https://github.com/Wankupi"><code>@​Wankupi</code></a>)</p>
</li>
</ul>
<p><strong>4.8.0</strong></p>
<ul>
<li>Added <strong>experimental</strong> support for running functions in
subinterpreters on Python
3.13 and later</li>
<li>Added support for the <code>copy()</code>, <code>copy_into()</code>,
<code>move()</code> and <code>move_into()</code>
methods in <code>anyio.Path</code>, available in Python 3.14</li>
<li>Changed <code>TaskGroup</code> on asyncio to always spawn tasks
non-eagerly, even if using a
task factory created via
<code>asyncio.create_eager_task_factory()</code>, to preserve expected
Trio-like task scheduling semantics (PR by <a
href="https://github.com/agronholm"><code>@​agronholm</code></a> and <a
href="https://github.com/graingert"><code>@​graingert</code></a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/agronholm/anyio/commit/a6e9ebb75f0e1492ab5d0740b2ca76d0137f95e3"><code>a6e9ebb</code></a>
Bumped up the version</li>
<li><a
href="https://github.com/agronholm/anyio/commit/9b9520d9ab3b00c0438073278830eddb64ff7875"><code>9b9520d</code></a>
Fixed cyclic references in to_thread.run_sync() on asyncio (<a
href="https://redirect.github.com/agronholm/anyio/issues/887">#887</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/1f04d6b1582f21bc199417ffcd46f78defbc0ec9"><code>1f04d6b</code></a>
Added a note about asyncio cancellation semantics</li>
<li><a
href="https://github.com/agronholm/anyio/commit/e14b17208982aaa90158038a66eb0696fe7bd548"><code>e14b172</code></a>
Upgraded to a newer Sphinx version</li>
<li><a
href="https://github.com/agronholm/anyio/commit/7e13c49b34e908980a1198e9b05f43677dab833a"><code>7e13c49</code></a>
Avoid refcycles in tracebacks from happy eyeballs exceptions (<a
href="https://redirect.github.com/agronholm/anyio/issues/809">#809</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/d134da712b884efc19894966421894da4dab1c86"><code>d134da7</code></a>
Raise EndOfStream from None in MemoryObjectReceiveStream.receive() (<a
href="https://redirect.github.com/agronholm/anyio/issues/889">#889</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/2840e06384d22f35935874043a753d04b0fad70c"><code>2840e06</code></a>
Added cheaper implementation for no_other_refs() on Python 3.14 (<a
href="https://redirect.github.com/agronholm/anyio/issues/886">#886</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/31ce0a5d7532b9148570194c2a934741bbed2cad"><code>31ce0a5</code></a>
Added fixtures for generating bindable TCP/UDP ports (<a
href="https://redirect.github.com/agronholm/anyio/issues/856">#856</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/cd85e47338a32e6eb813998c4b1c8dbf0b9f722b"><code>cd85e47</code></a>
Test on PyPy 3.11 on CI (<a
href="https://redirect.github.com/agronholm/anyio/issues/876">#876</a>)</li>
<li><a
href="https://github.com/agronholm/anyio/commit/d228020e6ea3f3b0bb8eab3a75eb1df31389eddf"><code>d228020</code></a>
Detect blocking calls in coroutines using BlockBuster (<a
href="https://redirect.github.com/agronholm/anyio/issues/875">#875</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/agronholm/anyio/compare/4.8.0...4.9.0">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants