Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

urllib3 v2 incompatibility #3113

Closed
zachmullen opened this issue Apr 26, 2023 · 40 comments · Fixed by microsoft/Olive#239
Closed

urllib3 v2 incompatibility #3113

zachmullen opened this issue Apr 26, 2023 · 40 comments · Fixed by microsoft/Olive#239

Comments

@zachmullen
Copy link

Minimal repro:

Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
/opt/worker_venv/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (2.0.0) or chardet (None)/charset_normalizer (3.1.0) doesn't match a supported version!
  warnings.warn(
>>> client = docker.from_env(version='auto')
Traceback (most recent call last):
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/worker_venv/lib/python3.8/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/opt/worker_venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/worker_venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'

urllib3 2.0.0 just released today. A quick fix would be to pin to urllib3<2.

@VonRehberg
Copy link

I think also the request version needs to be pinned, as there was a change?
Works with requests 2.28.1, but fails with 2.29.x

/tmp/ansible_docker_swarm_payload_315b1r1d/ansible_docker_swarm_payload.zip/ansible_collections/community/docker/plugins/module_utils/common.py", line 222, in _init_
    super(AnsibleDockerClientBase, self)._init_(**self._connect_params)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in _init_
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(...
....
},
    "msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"
}

@sebrhex
Copy link

sebrhex commented Apr 27, 2023

I can confirm just pinning requests<2.2.29 is solving this for me

@zachmullen
Copy link
Author

That's because older versions of requests vendored in their own urllib3. Modern requests uses the distributed version.

@felixfontein
Copy link
Contributor

I think urllib3 2.0.0 (or vendored urllib3 verions in requests) is a red herring; the problem is requests 2.29.0 itself because of a new feature: https://github.com/psf/requests/pull/6226/files

Docker SDK for Python uses requests with its own HTTP adapters for Unix connections by default (if you don't talk to the Docker daemon over TCP connections). For that, the code in Docker SDK for Python that provides a Unix connection HTTP adapter needs to be adjusted to support the same chunking support that urllib3 (already < 2.0.0) provides, and that requests 2.29.0 uses. (The other HTTP adapters might have the same problems.)

samdbmg added a commit to samdbmg/ansible-traefik-auth-proxy that referenced this issue May 1, 2023
@laurnts
Copy link

laurnts commented May 2, 2023

Wow this post is really helpful as I've just encounter an issue with Ansible managing Docker.

@felixfontein
Copy link
Contributor

I created a PR for making Docker SDK for Python compatible with requests 2.29.0: #3116. It fixes the problems for me (as long as I stick to urllib3 < 2.0, what request has been requiring for a long time now). It seems to fix the problems I had with requests 2.29.0 and urllib3 < 2.0; would be nice to have more real life testing (especially for all the transports I don't use).

@pquentin
Copy link

pquentin commented May 2, 2023

Hello, urllib3 2.0 maintainer here 👋

First, let me mention that right now, if you use pip install docker with a recent enough pip that can do dependency resolution you will end up with requests 2.29.0 and urllib3 1.26.15. With those versions, the docker.from_env reproducer above works. I'm surprised that @felixfontein sees issues with those versions, can you please explain how to reproduce them?

I can confirm however that docker-py is not compatible with urllib3 2.0. Since requests will soon allow it, docker-py should modify setup.py to use urllib3 >= 1.26.0, < 2.0.0 or fix the bug.


So what is the issue? Here is how docker-py supports Unix domain sockets:

class UnixHTTPConnection(httplib.HTTPConnection):
def __init__(self, base_url, unix_socket, timeout=60):
super().__init__(
'localhost', timeout=timeout
)
self.base_url = base_url
self.unix_socket = unix_socket
self.timeout = timeout
def connect(self):
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.settimeout(self.timeout)
sock.connect(self.unix_socket)
self.sock = sock
def putheader(self, header, *values):
super().putheader(header, *values)
def response_class(self, sock, *args, **kwargs):
return httplib.HTTPResponse(sock, *args, **kwargs)

It assumes that urllib3.connectionpool.HTTPConnectionPool will call conn.request in a way that is compatible with http.client, but this has changed in urllib3 2.0, and HTTPConnectionPool has its own request() function with its own signature.

I've opened #3117 for a fix.

@felixfontein
Copy link
Contributor

felixfontein commented May 2, 2023

@pquentin as you mention installing docker with a recent pip will give you requests 2.29.0 and urllib3 < 2.0, so urllib3 2.0 shouldn't be a problem in that case. But the stack trace printed above isn't the only thing that can go wrong with requests 2.29.0, I found some problems in CI with requests 2.29.0 and urllib3 1.26.15 as well. These happen because requests 2.29.0 includes https://github.com/psf/requests/pull/6226/files, which ends up calling UnixHTTPConnection.urlopen() with some parameters that httplib.HTTPConnection's version does not provide. In my case this happened with APIClient.put_archive() when streaming data in. (I'm not using Docker SDK for Python directly, but some code vendored from it - the first stack trace of ansible-collections/community.docker#611 (comment) happens with requests 2.29.0 and urllib3 1.26.15.)

BTW, your fix only fixes one of the transports; my similar PR (#3116) catches some more.

@otaronashvili
Copy link

otaronashvili commented May 2, 2023

Docker is installed over pip with the dependencies:

       "Requirement already up-to-date: docker in /usr/local/lib/python3.8/dist-packages (6.0.1)",
        "Requirement already satisfied, skipping upgrade: requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.29.0)",
        "Requirement already satisfied, skipping upgrade: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)",
        "Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (1.5.1)",
        "Requirement already satisfied, skipping upgrade: urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.1)",
        "Requirement already satisfied, skipping upgrade: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)",
        "Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)",
        "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)"

And we have the same problem:
"msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"

The only solution for us was:

pip3 remove requests
pip3 install requests=2.28.1

@slonopotamus
Copy link

slonopotamus commented May 3, 2023

Hello from downstream consumer of docker-py from ue4-docker! So, what is the current workaround? Lock requests to <2.29, or urllib3 to <2, or both?

And what do you mean by "recent enough" pip? We hit this issue on Ubuntu 20.04 with pip-22.0.4.

@pquentin
Copy link

pquentin commented May 3, 2023

And what do you mean by "recent enough" pip? We hit this issue on Ubuntu 20.04 with pip-22.0.4.

I meant at least pip 20.3, and ideally 21.3 or above, to get the new resolver and all its fixes. Unfortunately the recent enough pip is no longer enough as requests 2.30.0 was just released with urllib3 2.0 support.

Until docker-py pins urllib3 or merges #3116, you’ll have to pin to urllib3<2.0 yourself. As far as I can tell this has nothing to do with requests, so you can use the latest here.

@felixfontein
Copy link
Contributor

You can run the following with requests==0.29.0 and urllib3==1.26.13 to trigger an error:

from docker import APIClient
from docker.utils import kwargs_from_env

kwargs = kwargs_from_env()
client = APIClient(**kwargs)

def generate_data():
    yield b'123'

container = client.create_container(image='debian:bullseye', command=['/bin/sh', '-c', 'ls -la /'], stdin_open=True, detach=True, name='test')
client.put_archive(container['Id'], '/', generate_data())

Stacktrace:

Traceback (most recent call last):
  File "/path/to/my-script.py", line 11, in <module>
    client.put_archive(container['Id'], '/', generate_data())
  File "/path/to/docker-py/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/path/to/docker-py/docker/api/container.py", line 980, in put_archive
    res = self._put(url, params=params, data=data)
  File "/path/to/docker-py/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/path/to/docker-py/docker/api/client.py", line 241, in _put
    return self.put(url, **self._set_request_timeout(kwargs))
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 647, in put
    return self.request("PUT", url, data=data, **kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 396, in _make_request
    conn.request_chunked(method, url, **httplib_request_kw)
AttributeError: 'UnixHTTPConnection' object has no attribute 'request_chunked'

So yes, you need requests < 2.29.0 as well with the currently released Docker SDK for Python.

CheckmkCI pushed a commit to Checkmk/checkmk that referenced this issue Dec 5, 2023
... otherwise "docker load" fails with UnixHTTPConnection see
docker/docker-py#3113

CMK-14530

Change-Id: Id0e9d78d46038c092d7cdbd7140766b524450f23
@alex-kalanis
Copy link

Errors:

  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'preload_content'
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'decode_content'
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 522, in _make_request
    if not conn.is_closed:
AttributeError: 'UnixHTTPConnection' object has no attribute 'is_closed'
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 556, in _make_request
    response.length_remaining,  # type: ignore[attr-defined]
AttributeError: 'UnixHTTPResponse' object has no attribute 'length_remaining'

Why: It returns UnixHTTPResponse and get appropriace request object instead of objects from urllib. And that objects did not have these properties/methods. The worst thing: It's mentioned in comments inside the code (for me line 493).

Fast fix: Comment appropriate lines and blocks in "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py"

Correct fix: Detect existence of these params/methods and use them only if they are available.

@pquentin
Copy link

@alex-kalanis What version of urllib3, requests and docker-py are you using? docker-py is compatible with urllib3 2.x since version 6.1.0, released in May 2023.

@9600
Copy link

9600 commented Mar 7, 2024

Installed this module for use with Ansible, via the geerlingguy.docker and geerlingguy.pip roles. I now have on the target system:

  • docker 7.0.0
  • requests 2.31.0
  • urllib3 2.2.1

If I try to use Ansible to manage Docker configuration this fails with:

"msg": "Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'"

@pquentin
Copy link

pquentin commented Mar 8, 2024

Can you please share the full stack traceback? It's unlikely to come from the versions you mentioned.

@9600
Copy link

9600 commented Mar 8, 2024

The full traceback is:
  File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/common_api.py", line 118, in __init__
    super(AnsibleDockerClientBase, self).__init__(**self._connect_params)
  File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 188, in __init__
    self._version = self._retrieve_server_version()
  File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 212, in _retrieve_server_version
    raise DockerException(
fatal: [XX.XXX.com]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "appends": false,
            "attachable": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "connected": [],
            "debug": false,
            "docker_host": "unix://var/run/docker.sock",
            "driver": "bridge",
            "driver_options": {},
            "enable_ipv6": null,
            "force": false,
            "internal": null,
            "ipam_config": null,
            "ipam_driver": null,
            "ipam_driver_options": null,
            "labels": {},
            "name": "reverse-proxy",
            "scope": null,
            "ssl_version": null,
            "state": "present",
            "timeout": 60,
            "tls": false,
            "tls_hostname": null,
            "use_ssh_client": false,
            "validate_certs": false
        }
    },
    "msg": "Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'"
}

@pquentin
Copy link

pquentin commented Mar 8, 2024

This was fixed in community.docker 3.4.5, are you using an older version? Please report any further issues you have to https://github.com/ansible-collections/community.docker instead.

@9600
Copy link

9600 commented Mar 8, 2024

I was, yes. I'd also tried switching between the O/S (Ubuntu 22.04) package and PIP installs of docker-py on both host and target, and getting nowhere, but hadn't through to update the Ansible collection, which has resolved the issue!

@bergerkiller
Copy link

In case someone lands on this issue like I did trying to use ansible to manage Docker on a Ubuntu 24 LTS server, a quick solution. Make sure to use the new "community.docker.docker_compose_v2" task instead of the v1 "community.docker.docker_compose" task. You get this exact same error with v1.

@edwinm
Copy link

edwinm commented May 19, 2024

In case someone lands on this issue like I did trying to use ansible to manage Docker on a Ubuntu 24 LTS server, a quick solution. Make sure to use the new "community.docker.docker_compose_v2" task instead of the v1 "community.docker.docker_compose" task. You get this exact same error with v1.

Thanks. I spend hours lookinng for a solution for installing Octoprint docker and your comment was the only good suggestion.

BbolroC added a commit to BbolroC/cc-operator that referenced this issue May 23, 2024
The issue docker/docker-py#3113 seems to be resolved.
The pinned version of requests (2.28.1) is no longer compatible with the current
docker pip, resulting in errors like:

"Error connecting: Error while fetching server API version: Not supported URL scheme http+docker"

This commit allows pip to install requests according to its dependency tree.

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
BbolroC added a commit to BbolroC/cc-operator that referenced this issue May 23, 2024
The issue docker/docker-py#3113 seems to be resolved.
The pinned version of requests (2.28.1) is no longer compatible with the current
docker pip, resulting in errors like:

"Error connecting: Error while fetching server API version: Not supported URL scheme http+docker"

This commit allows pip to install requests according to its dependency tree.

Fixes: confidential-containers#381

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
BbolroC added a commit to BbolroC/cc-operator that referenced this issue May 23, 2024
The issue docker/docker-py#3113 seems to be resolved.
The pinned version of requests (2.28.1) is no longer compatible with the current
docker pip, resulting in errors like:

"Error connecting: Error while fetching server API version: Not supported URL scheme http+docker"

This commit allows pip to install requests according to its dependency tree.

Fixes: confidential-containers#381

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
BbolroC added a commit to BbolroC/cc-operator that referenced this issue May 23, 2024
The issue docker/docker-py#3113 seems to be resolved.
The pinned version of requests (2.28.1) is no longer compatible with the current
docker pip, resulting in errors like:

"Error connecting: Error while fetching server API version: Not supported URL scheme http+docker"

Firstly, we try to allow pip to install requests according to its dependency tree.
But, the issue still occurs on Ubuntu 22.04 where docker/docker-py#3256
looks relevant. The best solution so far is to pin the version of the package less than 2.32.

Initially, we attempted to allow pip to install requests according to its dependency tree.
However, the issue persists on Ubuntu 22.04, and it appears to be related to
docker/docker-py#3256.
The best solution so far is to pin the version of the package to less than 2.32.

Fixes: confidential-containers#381

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
BbolroC added a commit to BbolroC/cc-operator that referenced this issue May 23, 2024
The issue docker/docker-py#3113 seems to be resolved.
The pinned version of requests (2.28.1) is no longer compatible with the current
docker pip, resulting in errors like:

"Error connecting: Error while fetching server API version: Not supported URL scheme http+docker"

Initially, we attempted to allow pip to install requests according to its dependency tree.
However, the issue persists on Ubuntu 22.04, and it appears to be related to
docker/docker-py#3256.
The best solution so far is to pin the version of the package to less than 2.32.

Fixes: confidential-containers#381

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.