Received "Response payload is not completed" when reading response #4581
Comments
|
CC: @socketpair @asvetlov as the TODO was added in 5c4cb82 |
|
I'm curious if it's somehow related to #4549. |
Change the |
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. This should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have CIBA disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # scripts/run-test-plan.py
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # scripts/run-test-plan.py
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
|
Hi there, I've received this error too. My use case is a reverse-proxy in front of gunicorn. The other reports of this instance on this repo appear to be poorly behaved HTTP servers, which I think is a separate issue. My code streams the proxy body back to upstream, using code roughly like: response = web.StreamingResponse(headers=response_headers, status=status)
async for chunk in gunicorn_response.content.iter_chunked(64 * 1_024):
await response.write(chunk)
await response.write_eof()
return responseWhat seems to be happening is that gunicorn returns a Connection: close header and then there is a race condition between aiohttp reading the body and the connection being closed. If I get the data out of aiohttp in time, it works, but sometimes this stack trace is triggered. Investigation:
|
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # .gitlab-ci.yml # .gitlab-ci/run-tests.sh # scripts/run-test-plan.py # test/Dockerfile
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # .gitlab-ci.yml # .gitlab-ci/run-tests.sh # scripts/run-test-plan.py # test/Dockerfile
|
we have this happening between an aiohttp 3.8.1 client + server in production, trying to reproduce |
|
I've just experienced the same issue via s3fs, downloading a few hundred files from S3. Python: 3.7.12 Traceback (most recent call last):
File "dataset_io.py", line 336, in download
filesystem.get(self._location, self._download_location, recursive=True)
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
raise return_result
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 529, in _get
coros, batch_size=batch_size, callback=callback
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 249, in _run_coros_in_chunks
await asyncio.gather(*chunk, return_exceptions=return_exceptions),
File "/root/.pyenv/versions/3.7.12/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
return await fut
File "/.venv/lib/python3.7/site-packages/s3fs/core.py", line 1002, in _get_file
chunk = await body.read(2 ** 16)
File "/.venv/lib/python3.7/site-packages/aiobotocore/response.py", line 53, in read
chunk = await self.__wrapped__.read(amt if amt is not None else -1)
File "/.venv/lib/python3.7/site-packages/aiohttp/streams.py", line 385, in read
await self._wait("read")
File "/.venv/lib/python3.7/site-packages/aiohttp/streams.py", line 304, in _wait
await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completedSorry I don't have much more information here, other than the fact that this is occurring on Python 3.7, whereas #4549 has been described as occurring on Python 3.8 only. |
|
I got this error on Python 3.9 with aiohttp 3.8.1 on Windows. I haven't been able to reproduce. My server failed around the same time as the error. The |
|
I have the same issue with Python 3.10 |
This comment was marked as duplicate.
This comment was marked as duplicate.
I am also getting that error using iter_chunked when downloading files from S3 |
Instead, pass through the upstream Connection header, like we do with Upgrade: already. This helps to avoid aio-libs/aiohttp#4581
This is conceptually broken -- the browser should decide when it wants to upgrade a connection to websocket, and we should just pass that through (like we do with `Upgrade:` already). This is also what the real 3scale does, so more faithfully reproduces its behaviour. This also helps to avoid aio-libs/aiohttp#4581
This is conceptually broken -- the browser should decide when it wants to upgrade a connection to websocket, and we should just pass that through (like we do with `Upgrade:` already). This is also what the real 3scale does, so more faithfully reproduces its behaviour. This also helps to avoid aio-libs/aiohttp#4581
|
Python 3.7 itself is not supported and may have known security issues. We won't put any effort into supporting a version of Python which is not itself supported in any way, so you'll need to upgrade to a more recent version to test anything or get any support from us. |
No, those are very old pre-releases for old versions of the |
|
@Dreamsorcerer @webknjaz |
I'm not sure if this is relevant here, but maybe the thing to bear in mind is that, when calling stuff like |
|
I have a reproducer: + Exception Group Traceback (most recent call last):
| File "/var/src/aiohttp/test.py", line 42, in <module>
| asyncio.run(main())
| File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/asyncio/base_events.py", line 685, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/test.py", line 37, in main
| async with asyncio.TaskGroup() as g:
| File "/usr/lib64/python3.12/asyncio/taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/var/src/aiohttp/aiohttp/client_proto.py", line 83, in connection_lost
| uncompleted = self._parser.feed_eof()
| ^^^^^^^^^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/aiohttp/http_parser.py", line 277, in feed_eof
| self._payload_parser.feed_eof()
| File "/var/src/aiohttp/aiohttp/http_parser.py", line 788, in feed_eof
| raise TransferEncodingError(
| aiohttp.http_exceptions.TransferEncodingError: 400, message:
Not enough data for satisfy transfer length header.
|
| The above exception was the direct cause of the following exception:
|
| Traceback (most recent call last):
| File "/var/src/aiohttp/test.py", line 15, in do_stuff
| async for chunk in resp.content.iter_chunked(8192):
| File "/var/src/aiohttp/aiohttp/streams.py", line 44, in __anext__
| rv = await self.read_func()
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/aiohttp/streams.py", line 347, in read
| raise self._exception
| aiohttp.client_exceptions.ClientPayloadError: Response payload is not completedUnfortunately i accidentally deleted my fugazi server, give me a moment while i rewrite it. |
|
https://gist.github.com/gtedesco-r7/0276fc7a00ebe568afc7132ce64b011c It explodes for me after a couple of minutes: <<< 8327 bytes
+ Exception Group Traceback (most recent call last):
| File "/var/src/aiohttp/test.py", line 42, in <module>
| asyncio.run(main())
| File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/asyncio/base_events.py", line 685, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/test.py", line 37, in main
| async with asyncio.TaskGroup() as g:
| File "/usr/lib64/python3.12/asyncio/taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/var/src/aiohttp/aiohttp/client_proto.py", line 83, in connection_lost
| uncompleted = self._parser.feed_eof()
| ^^^^^^^^^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/aiohttp/http_parser.py", line 277, in feed_eof
| self._payload_parser.feed_eof()
| File "/var/src/aiohttp/aiohttp/http_parser.py", line 788, in feed_eof
| raise TransferEncodingError(
| aiohttp.http_exceptions.TransferEncodingError: 400, message:
Not enough data for satisfy transfer length header.
|
| The above exception was the direct cause of the following exception:
|
| Traceback (most recent call last):
| File "/var/src/aiohttp/test.py", line 15, in do_stuff
| async for chunk in resp.content.iter_chunked(8192):
| File "/var/src/aiohttp/aiohttp/streams.py", line 44, in __anext__
| rv = await self.read_func()
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/var/src/aiohttp/aiohttp/streams.py", line 347, in read
| raise self._exception
| aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
+------------------------------------
real 1m50.561s
user 0m48.431s
sys 0m7.836s |
Yeah, Weird thing is, even my little bit of debugging seems to turn it into a |
|
I don't remember the details, if you think you can dig into the code and figure out exactly what's going wrong, that'd be great. |
|
@gtedesco-r7 is that with version 3.9.4rc0? Pure-python or Cython/C-extension? I still haven't gotten any instances of this exception in my deployments since Feb 16. |
@gtedesco-r7 this line number correlates with code placement in an older aiohttp 3.9.3. Could you check your repro against 3.9.4rc0? |
|
@gtedesco-r7 your gist doesn't reproduce the problem for me under Python 3.11. Neither aiohttp 3.9.3, nor 3.9.4rc0 breaks like that. Though, it appears you might be using a pure-Python implementation of the parser and not the Cython-based one that's default when available. |
|
@gtedesco-r7 based on the above observation, your specific case should be reproducible with |
|
UPD: it took 6m 13s on my machine and the reproducer in the gist produced a similar exception with Python 3.11, aiohttp 3.9.3 and |
|
UPD: it took 1m 21s of running the repro with Python 3.11, aiohttp 3.9.4rc0 and + Exception Group Traceback (most recent call last):
| File "~/src/github/aio-libs/aiohttp/some-venv/aiohttp-4581-repro/aiohttp-test.py", line 42, in <module>
| asyncio.run(main())
| File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/runners.py", line 190, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "~/src/github/aio-libs/aiohttp/some-venv/aiohttp-4581-repro/aiohttp-test.py", line 37, in main
| async with asyncio.TaskGroup() as g:
| File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/taskgroups.py", line 147, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "~/src/github/aio-libs/aiohttp/some-venv/lib/python3.11/site-packages/aiohttp/client_proto.py", line 94, in connection_lost
| uncompleted = self._parser.feed_eof()
| ^^^^^^^^^^^^^^^^^^^^^^^
| File "~/src/github/aio-libs/aiohttp/some-venv/lib/python3.11/site-packages/aiohttp/http_parser.py", line 279, in feed_eof
| self._payload_parser.feed_eof()
| File "~/src/github/aio-libs/aiohttp/some-venv/lib/python3.11/site-packages/aiohttp/http_parser.py", line 793, in feed_eof
| raise TransferEncodingError(
| aiohttp.http_exceptions.TransferEncodingError: 400, message:
Not enough data for satisfy transfer length header.
|
| The above exception was the direct cause of the following exception:
|
| Traceback (most recent call last):
| File "~/src/github/aio-libs/aiohttp/some-venv/aiohttp-4581-repro/aiohttp-test.py", line 15, in do_stuff
| async for chunk in resp.content.iter_chunked(8192):
| File "~/src/github/aio-libs/aiohttp/some-venv/lib/python3.11/site-packages/aiohttp/streams.py", line 50, in __anext__
| rv = await self.read_func()
| ^^^^^^^^^^^^^^^^^^^^^^
| File "~/src/github/aio-libs/aiohttp/some-venv/lib/python3.11/site-packages/aiohttp/streams.py", line 357, in read
| raise self._exception
| aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
+------------------------------------And on the server side: ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-19080' coro=<fugazi_web() done, defined at ~/src/github/aio-libs/aiohttp/some-venv/aiohttp-4581-repro/fake-server.py:8> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "~/src/github/aio-libs/aiohttp/some-venv/aiohttp-4581-repro/fake-server.py", line 13, in fugazi_web
req = await rd.read(8192)
^^^^^^^^^^^^^^^^^^^
File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/streams.py", line 694, in read
await self._wait_for_data('read')
File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/streams.py", line 526, in _wait_for_data
await self._waiter
File "~/.pyenv/versions/3.11.5/lib/python3.11/asyncio/selector_events.py", line 995, in _read_ready__data_received
data = self._sock.recv(self.max_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer |
|
The Cython-based implementation still doesn't crash with that snippet. So it's either fast enough for the race not to happen, or the implementation doesn't have the same bug as pure-python. |
|
Looking into the source, it's clear that Back to So this means that if by the time @Dreamsorcerer @bdraco thoughts? ideas? |
|
I couldn't reproduce it after 30 mins. I'd probably need to inspect the raw data going through the parser to understand it properly. |
|
@Dreamsorcerer sometimes, it was non-reproducible at all for me too. But given enough time, it's reproducible with the pure-python implementation. Make sure to have that env var set, just in case. I was also adding a |
|
It took about 45 minutes to happen on my system |
|
@bdraco it was inconsistent for me too. I'm mostly curious what you think of what I wrote in #4581 (comment). What I suggested should work in theory but I'm worried it's only addressing the consequence and not the root cause. |
|
I'm poking at it a bit to see if I can find the root cause but given the long test cycle it may take a while and I'm time limited for the next few days |
|
Looks like sometimes I get |
|
Interesting.. I never got that timeout. But it makes sense given that there seems to be a race between something unsetting |
…r retrying (#14545) The treatment of `ClientPayloadError` as a sometimes transient error was originally made in response to [an existing issue](aio-libs/aiohttp#4581) in aiohttp that can cause transient errors on the client that are difficult to distinguish from a real broken server. What's in `main` matched exactly on the error message, but that error message has [since changed](aio-libs/aiohttp@dc38630) to include more information, breaking our transient error handling. This change relaxes the requirement of the error response string to fix transient error handling for our current version of `aiohttp`. I wish I had a better approach. `ClientPayloadError` can also be thrown in the case of malformed data, so I am reticent to treat it as always transient, but we could perhaps make it a `limited_retries_error` and avoid inspecting the error message.
|
Sorry - was there an official fix for this? I am able to reproduce when utilizing Basically: Anyone have any suggestions or thoughts? I'm using |
|
I think I have a reliably reproducible example that is derived from @gtedesco-r7's example Python 3.12 A |
and others
🐞 Describe the bug
A
ClientPayloadError:Response payload is not completedexception is raised when reading the response of a GET request from the GitHub REST API (this should not be especially relevant). The response seems correct and usingcurlsucceeds all the time.💡 To Reproduce
Do lots of requests against an endpoint of the GitHub API (it happens from time to time).
💡 Expected behavior
The response is correctly parsed.
📋 Logs/tracebacks
Some contextual information at the time of the exception at
aiohttp/streams.py", line 358, in read:As you can see the blocks contain the whole payload (the full JSON) but
aiohttpstill complains about the payload not being completed.📋 Your version of the Python
📋 Your version of the aiohttp/yarl/multidict distributions
📋 Additional context
I suspect this has to be with this comment:
aiohttp/aiohttp/streams.py
Lines 352 to 356 in 1740c99
as this is where is failing and the situation seems that is the one described in the
TODO.The text was updated successfully, but these errors were encountered: