Support setting response header size limits #6559

jorop · 2022-01-25T09:19:05Z

What do these changes do?

Allow the user to set the response header limits. The properties max_line_size, max_headers and max_field_size can be set on ClientSession creation. Default values are: max_line_size=8190 max_headers=32768 and max_field_size=8190.
Thus the user can set and - when needed - receive bigger response headers.

Are there changes in behavior for the user?

No

Related issue number

#2304

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES folder
- name it <issue_id>.<type> for example (588.bugfix)
- if you don't have an issue_id change it to the pr id after creating the pr
- ensure type is one of the following:
  - .feature: Signifying a new feature.
  - .bugfix: Signifying a bug fix.
  - .doc: Signifying a documentation improvement.
  - .removal: Signifying a deprecation or removal of public API.
  - .misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

Dreamsorcerer

Will need some tests for the new behaviour to have any chance of being merged, which would also have highlighted the below issue.

aiohttp/client.py

tests/test_client_functional.py

manuel2295 · 2022-03-04T12:48:14Z

When will this change be approved? This issue is very important to the use of the library.

Dreamsorcerer · 2022-03-04T16:49:18Z

There is a test failure, which I've not looked at closely yet. It's not one I've seen before though.

smonist · 2022-03-14T14:58:04Z

This is the test that failed: https://github.com/aio-libs/aiohttp/runs/5072020477?check_suite_focus=true#step:10:3041
It looks like the plausible timeout for this test is set too narrow. Can we run the test again and look at the results? Seems not related to the code changes here.

dmitry-brazhenko · 2023-04-25T09:54:02Z

Hello @Dreamsorcerer!

Is it possible to merge this PR ? :)

Thanks

CHANGES/2304.feature

aiohttp/client.py

aiohttp/client_proto.py

tests/test_client_functional.py

codecov · 2023-04-25T17:33:56Z

Codecov Report

Merging #6559 (e8064f9) into master (315ae90) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #6559   +/-   ##
=======================================
  Coverage   97.31%   97.31%           
=======================================
  Files         107      107           
  Lines       31414    31484   +70     
  Branches     3927     3937   +10     
=======================================
+ Hits        30569    30639   +70     
  Misses        641      641           
  Partials      204      204

Flag	Coverage Δ
CI-GHA	`97.22% <100.00%> (+<0.01%)`	⬆️
OS-Linux	`96.88% <100.00%> (+<0.01%)`	⬆️
OS-Windows	`95.24% <100.00%> (-0.05%)`	⬇️
OS-macOS	`96.46% <100.00%> (+<0.01%)`	⬆️
Py-3.10.11	`96.67% <100.00%> (-0.31%)`	⬇️
Py-3.11.0	`96.41% <100.00%> (+<0.01%)`	⬆️
Py-3.7.16	`96.70% <100.00%> (+<0.01%)`	⬆️
Py-3.7.9	`95.17% <100.00%> (+0.01%)`	⬆️
Py-3.8.10	`95.09% <100.00%> (+0.01%)`	⬆️
Py-3.8.16	`96.61% <100.00%> (+<0.01%)`	⬆️
Py-3.9.13	`?`
Py-3.9.16	`96.63% <100.00%> (+<0.01%)`	⬆️
Py-pypy7.3.11	`94.12% <100.00%> (+<0.01%)`	⬆️
VM-macos	`96.46% <100.00%> (+<0.01%)`	⬆️
VM-ubuntu	`96.88% <100.00%> (+<0.01%)`	⬆️
VM-windows	`95.24% <100.00%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiohttp/client_proto.py	`96.73% <ø> (ø)`
aiohttp/client.py	`94.84% <100.00%> (+0.06%)`	⬆️
tests/test_client_functional.py	`98.51% <100.00%> (+0.04%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

patchback · 2023-04-25T17:55:29Z

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply ecd9c72 on top of patchback/backports/3.9/ecd9c72697ca1529e7cae4693775612987a2a6d2/pr-6559

Backporting merged PR #6559 into master

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/aio-libs/aiohttp.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.9/ecd9c72697ca1529e7cae4693775612987a2a6d2/pr-6559 upstream/3.9

Now, cherry-pick PR Support setting response header size limits #6559 contents into that branch:
```
$ git cherry-pick -x ecd9c72697ca1529e7cae4693775612987a2a6d2
```
If it'll yell at you with something like fatal: Commit ecd9c72697ca1529e7cae4693775612987a2a6d2 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x ecd9c72697ca1529e7cae4693775612987a2a6d2
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Support setting response header size limits #6559 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.9/ecd9c72697ca1529e7cae4693775612987a2a6d2/pr-6559

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Dreamsorcerer · 2023-04-25T17:55:55Z

If someone could handle the backport to 3.9 as above, that'd be great.

rohansahai · 2023-04-28T19:29:24Z

Awesome to see this change coming 😃 . Is there any ETA of the 3.9 release?

Dreamsorcerer · 2023-04-28T21:28:07Z

Not yet, but still waiting on the backport to 3.9, if you'd like to help out:
#6559 (comment)

Allow the user to set the response header limits. The properties max_line_size, max_headers and max_field_size can be set on ClientSession creation. Default values are: `max_line_size=8190` `max_headers=32768` and `max_field_size=8190`. Thus the user can set and - when needed - receive bigger response headers. No - [x] I think the code is well written - [x] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files." --------- Co-authored-by: Josef Prenner <josef@cloudomation.com> Co-authored-by: Sam Bull <aa6bs0@sambull.org> (cherry picked from commit ecd9c72)

…559 (#7264)  ## What do these changes do? This an attempt at a backport following these guidelines - #6559 (comment) . Note, haven't done a backport so no clue if I'm completely missing something or it was as simple as resolving the merge conflicts when cherry-picking that work in. See #6559 (comment) --------- Co-authored-by: jorop <jorop@users.noreply.github.com> Co-authored-by: Sam Bull <aa6bs0@sambull.org>

Josef Prenner added 3 commits January 24, 2022 14:32

max_line_size, max_headers and max_field_size as params (aio-libs#2304)

d882ed3

check optional params (aio-libs#2304)

bcf43c1

added changelog (aio-libs#2304)

fc75c1b

jorop requested a review from asvetlov as a code owner January 25, 2022 09:19

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Jan 25, 2022

Dreamsorcerer requested changes Jan 25, 2022

View reviewed changes

aiohttp/client.py Outdated Show resolved Hide resolved

Josef Prenner added 2 commits January 27, 2022 20:04

moved Optional typehint (aio-libs#2304)

5114470

unittests (aio-libs#2304)

ea982ef

jorop requested a review from Dreamsorcerer February 1, 2022 06:07

Dreamsorcerer approved these changes Feb 1, 2022

View reviewed changes

tests/test_client_functional.py Outdated Show resolved Hide resolved

changed tests (aio-libs#2304)

4b0d566

Merge branch 'master' into master

4d2a746

Dreamsorcerer reviewed Apr 25, 2023

View reviewed changes

Remove no longer existing max_headers

e8064f9

Dreamsorcerer added the backport-3.9 Trigger automatic backporting to the 3.9 release branch by Patchback robot label Apr 25, 2023

Dreamsorcerer merged commit ecd9c72 into aio-libs:master Apr 25, 2023
33 of 35 checks passed

rohansahai mentioned this pull request Apr 29, 2023

Patchback/backports/3.9/ecd9c72697ca1529e7cae4693775612987a2a6d2/pr 6559 #7264

Merged

NotoriousRebel mentioned this pull request Sep 24, 2023

[Bug] Brave error ? laramies/theHarvester#1541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support setting response header size limits #6559

Support setting response header size limits #6559

jorop commented Jan 25, 2022 •

edited

Dreamsorcerer left a comment •

edited

manuel2295 commented Mar 4, 2022

Dreamsorcerer commented Mar 4, 2022

smonist commented Mar 14, 2022

dmitry-brazhenko commented Apr 25, 2023

codecov bot commented Apr 25, 2023 •

edited

patchback bot commented Apr 25, 2023 •

edited

Dreamsorcerer commented Apr 25, 2023

rohansahai commented Apr 28, 2023

Dreamsorcerer commented Apr 28, 2023

Support setting response header size limits #6559

Support setting response header size limits #6559

Conversation

jorop commented Jan 25, 2022 • edited

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

Dreamsorcerer left a comment • edited

Choose a reason for hiding this comment

manuel2295 commented Mar 4, 2022

Dreamsorcerer commented Mar 4, 2022

smonist commented Mar 14, 2022

dmitry-brazhenko commented Apr 25, 2023

codecov bot commented Apr 25, 2023 • edited

Codecov Report

patchback bot commented Apr 25, 2023 • edited

Backport to 3.9: 💔 cherry-picking failed — conflicts found

Backporting merged PR #6559 into master

Dreamsorcerer commented Apr 25, 2023

rohansahai commented Apr 28, 2023

Dreamsorcerer commented Apr 28, 2023

jorop commented Jan 25, 2022 •

edited

Dreamsorcerer left a comment •

edited

codecov bot commented Apr 25, 2023 •

edited

patchback bot commented Apr 25, 2023 •

edited