Docker 25.03 - Updating containers results in IP Error #47441
Assignees
Labels
area/networking
kind/bug
Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
version/25.0
Comments
edrikk
added
kind/bug
|
Hi there, It happened again, so let me update with state. Over night, 2 containers were updated by WatchTower. Portainer updated "CONTAINER_1" first, and it was "successful" (in the sense that it came back up). This container has been defined with stack/compose: However, although "successful", it is given IPv4 Address as below when I look within Portainer's "Networks" section: The 2nd container updated "CONTAINER_2" failed. Below is the WatchTower log: Once the image is created for the container (which has failed and is assigned 252 address), stopping and restarting it does not change the address to the static IP that it is supposed to have. I have to delete the image, and rebuild, at which point the static IP IP address is correctly assigned, and then "CONTAINER_2" can also be repulled etc. |
|
Looks like the same issue that I’m reporting |
Hi @edrikk - yes, it does sound like the same issue, also discussed at #47257 (comment) ... I was unable to reproduce the problem, for me Watchtower managed to re-create the container with its configured address. So, as described in that comment - if you're able to enable Docker's debug logging, it would be useful to see the "form data" log lines corresponding to the create/connect POST requests Watchtower is using to re-create the container. |
|
Thanks @robmry for this. I have captured a log overnight with the failure. Below snippet is the container that got the .252 address. All updates after this one failed with the no ip address error. I have obfuscated some info (e.g. container names, etc) but if you wish for a fuller log or one that isn't edited, just let me know how to provide privately and I can do so. The relevant part of compose/stack: The error log: The updates that followed (and failed) show: |
|
The moby/libnetwork/ipam/allocator.go Line 349 in 2c25ca9 Looking from that code, the |
|
Perhaps these check should check for moby/libnetwork/ipam/allocator.go Lines 354 to 358 in 2c25ca9 |
|
Makes sense. I will note that given that the container should have a static IP (which per the log docker "released" successfully at start of update) even that it's going through the dynamic IP path is odd (to me). Not knowing the code, I would think that either it shouldn't be even calling for an IP assignment, or upon seeing it has an invalid IP it should assign the static IP, or the empty/null IP should be replaced with the static IP before it even makes it to this check. But (stating the obvious here 😃 ), assigning 192.168.42.252 is not the correct end result. 😃 |
|
Yes, indeed - it shouldn't have got as far as the allocator, because an address was supplied, the one Watchtower found in the old container is what's needed. Once the allocator's involved, it's already doomed. ( I'm not sure why that's happened, but will take a look... - thank you for the logs! |
|
Just to update on this ... In the In docker 25.x, we deprecated a container-wide setting for the container's MAC address in the latest version of the API. (Watchtower is using an old version of the API.) For compatibility with old versions of the API, we migrate the container-wide MAC address to endpoint settings - where the updated code expects to find it. In doing that, when the So, we end up two sets of endpoint config for the same network and later lose the config supplied under the network's name, including the preferred IP address lives. The backward-compatibility config migration happens early in the request processing, before it's possible to normalise the network name/id/short-id. So, a fix looks a little tricky, I'm still looking in to it. But - this issue will affect Watchtower updates, when the container has both a configured IP address and a configured MAC address. At the moment, I don't think there's a workaround for that case - it'll need a change in either Watchtower or Docker/moby. |
|
Thank you. Really appreciate the loop back update, and especially one so well explained. |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I had incorrectly opened this issue in the CLI tracker. So closing that, and recreating here.
Copy-paste from that ticket.
Hi,
I had been running docker 24.0.X successfully with current setup without issue, however docker 25 started the below. I saw that fixes had gone into version including the 25.0.3 item to resolve MAC address issues.
I have also followed the instructions around recreating all the dockers once on 25.0.3 but my issue remains.
My setup is as follows:
Since upgrading from 24.0.X (and currently on 25.0.3), it appears that the first(? or one of the first) upgrades will be given the .252 address (even thouigh it should get a hardcoded/static IP as defined in the compose/stack). Since this is the only IP Address allowed in my range, the next updated dockers will fail to load with the error message:
At this point, I have to manually delete the docker container and delete the docker image of both the docker taking up the "wrongly assigned" IP address, as well as the containers that failed to install, then go into the stack/compose and redeploy them. They will then be given the correct static IP addresses, and all is well.
Until the next set of updates.
Reproduce
Per above.
If watchtower updates a given container after the number of updates passes the number of dynamic IPs that docker is allowed to allocated, this error seems to be hit.
In my case, I have set docker to allocate 1 single IP, but all my containers have static IP addresses defined for them - so the dynamic IP should never be used / cause this issue.
Expected behavior
Dockers should be given the \static MAC and IP address as defined in their docker compose/stack. This was the correct behavior in 24.0.X which has regressed in 25.0.X
docker version
Client: Docker Engine - Community Version: 25.0.3 API version: 1.44 Go version: go1.21.6 Git commit: 4debf41 Built: Tue Feb 6 21:14:25 2024 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 25.0.3 API version: 1.44 (minimum version 1.24) Go version: go1.21.6 Git commit: f417435 Built: Tue Feb 6 21:14:25 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.28 GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0docker info
Additional Info
No response
The text was updated successfully, but these errors were encountered: