502 Bad Gateway after updating Ubuntu packages

[SimonUzL]

Steps to Reproduce

1. Setup WSL
2. apt update
3. apt dist-upgrade
4. Install docker: https://docs.docker.com/engine/install/ubuntu/
5. git clone https://github.com/overleaf/toolkit.git ./overleaf-toolkit
6. bin/init
7. bin/up

Expected Behaviour

Overleaf should be available under http://localhost:80

Observed Behaviour

502 Bad Gateway

Context

After upgrading the packages on my Ubuntu Server and rebooting, I get 502 Bad Gateway.

Technical Info

• URL:
• Browser Name and version:
• Operating System and version (desktop or mobile):
• Signed in as:
• Project and/or file:

Analysis

I am using docker 5:26.0.0-1~ubuntu.22.04~jammy

but if I use: 5:23.0.0-1~ubuntu.22.04~jammy, then everything works as expected. Also 5.25 works.

[ykorzikowski]

I get the same error, but just running the container with native linux. Its the same image I build two weeks ago. Did not find the error, yet.

Edit 1: Found this in the logs. There are no more error logs.

{"name":"filestore","hostname":"01b21db226d3","pid":141,"level":40,"err":{"message":"We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy","name":"NOTE","stack":"NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n    at emitWarning (/overleaf/node_modules/aws-sdk/lib/maintenance_mode_message.js:37:13)\n    at Timeout._onTimeout (/overleaf/node_modules/aws-sdk/lib/maintenance_mode_message.js:45:5)\n    at listOnTimeout (node:internal/timers:569:17)\n    at process.processTimers (node:internal/timers:512:7)","info":{}},"msg":"Warning details","time":"2024-03-22T10:25:04.121Z","v":0}

Edit 2: After further investigation:

root@01b21db226d3:/var/log# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:44207        0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      135/nginx: master p
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      135/nginx: master p
tcp6       0      0 :::3100                 :::*                    LISTEN      -
tcp6       0      0 ::1:4000                :::*                    LISTEN      -
tcp6       0      0 ::1:3042                :::*                    LISTEN      -
tcp6       0      0 ::1:3054                :::*                    LISTEN      -
tcp6       0      0 ::1:3048                :::*                    LISTEN      -
tcp6       0      0 ::1:3049                :::*                    LISTEN      -
tcp6       0      0 ::1:3013                :::*                    LISTEN      -
tcp6       0      0 ::1:3010                :::*                    LISTEN      -
tcp6       0      0 ::1:3009                :::*                    LISTEN      -
tcp6       0      0 ::1:3016                :::*                    LISTEN      -
tcp6       0      0 ::1:3026                :::*                    LISTEN      -
tcp6       0      0 ::1:3036                :::*                    LISTEN      -
tcp6       0      0 ::1:3005                :::*                    LISTEN      -
tcp6       0      0 ::1:3003                :::*                    LISTEN      -
tcp6       0      0 :::10843                :::*                    LISTEN      -
udp        0      0 127.0.0.11:52238        0.0.0.0:*                           -

root@01b21db226d3:/var/log# curl http://127.0.0.1:4000
curl: (7) Failed to connect to 127.0.0.1 port 4000: Connection refused

root@01b21db226d3:/var/log# curl -vvv http://[::1]:4000
*   Trying ::1:4000...
* TCP_NODELAY set
* Connected to ::1 (::1) port 4000 (#0)
> GET / HTTP/1.1
> Host: [::1]:4000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
[...]

Checking the nginx config

root@01b21db226d3:/var/log# cat /etc/nginx/sites-enabled/sharelatex.conf
server {
	listen         80;
	server_name    _; # Catch all, see http://nginx.org/en/docs/http/server_names.html

	root /overleaf/services/web/public/;

	# block external access to prometheus /metrics
	location /metrics {
		internal;
	}

	location / {
		proxy_pass http://127.0.0.1:4000;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_read_timeout 10m;
		proxy_send_timeout 10m;
	}

	location /socket.io {
		proxy_pass http://127.0.0.1:3026;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_read_timeout 10m;
		proxy_send_timeout 10m;
	}
[...]

Seems like nodejs is only listening on net6 address, but nginx tries to proxy to net4 address.

Edit 3:

After changing the nginx-conf to proxy to [::1], I get at least the mainentance site.

But after this, I have ... questions: Why is there something updating without me updating the docker container? I am not using the official one, as I pre-download the textLive Environment. So the container is the same for about two weeks.
Why it's running updates on its own. This is not how docker is supposed to work. Container Images should be static and only be updated by building a new container.

Edit 4:
OK Pardon me, it's seems its the updated docker engine that creates trouble.

I just read the Release Notes of 26 Release (https://docs.docker.com/engine/release-notes/26.0/)

Always attempt to enable IPv6 on a container's loopback interface, and only include IPv6 in /etc/hosts if successful. moby/moby#47062
Note
By default, IPv6 will remain enabled on a container's loopback interface when the container is not connected to an IPv6-enabled network. For example, containers that are only connected to an IPv4-only network now have the ::1 address on their loopback interface.
To disable IPv6 in a container, use option --sysctl net.ipv6.conf.all.disable_ipv6=1 in the create or run command, or the equivalent sysctls option in the service configuration section of a Compose file.
If IPv6 is not available in a container because it has been explicitly disabled for the container, or the host's networking stack does not have IPv6 enabled (or for any other reason) the container's /etc/hosts file will not include IPv6 entries.

[johannesjahn]

Downgrading docker to 5.25 or 5.23 didn't work for me. Any way to quickfix for now?

[KKraljic]

The issue referenced here seems to be similar:
#1146

The workaround mentioned there helped me to get at least to the login page and the project overview:

docker exec -it <overleaf_container_name> /bin/bash
sed -i 's/127.0.0.1/localhost/g' /etc/nginx/sites-enabled/sharelatex.conf
/etc/init.d/nginx reload

However, loading a project does not work.

[ykorzikowski]

I solved this by adding this to my docker-compose.yml. And make sure you are using the docker-compose v2 API. (docker compose upinstead of old way docker-compose up)

services:
[...]
  overleaf:
    [...]
    sysctls:
        - net.ipv6.conf.all.disable_ipv6=1

This will disable ipv6 for this container at all.

[KKraljic]

Yep, I can confirm that. The same workaround is described here: #1168 and works also greatly on my system.

[ThexXTURBOXx]

This can be fixed through #1146 (comment)

[mlevans0]

This should be fixed in the latest release, 5.0.1, would you mind giving it a try?

Check the release notes, as there are several breaking changes: https://github.com/overleaf/overleaf/wiki/Release-Notes-5.x.x

[ThexXTURBOXx]

@mlevans0 At least on my machine 5.0.1 works fine without any changes - I upgraded yesterday already... Don't know about OP, though!

[damian-anlauf]

This should be fixed in the latest release, 5.0.1, would you mind giving it a try?

Check the release notes, as there are several breaking changes: https://github.com/overleaf/overleaf/wiki/Release-Notes-5.x.x

Can also confirm. 5.0.1 fixed the issue for me.
FYI, I'm using a regular compose.yaml, not the Overleaf Toolkit.

[mlevans0]

Changes were made inside the container, so it won't matter how your instance is deployed.

[SimonUzL]

I can confirm, that the solution by @ykorzikowski works. Thank you very much for the hint. I was then able to upgrade docker to the newest version. I have not yet tested 5.0.1, but it seems to be a bad idea, according to the documentation, to do it now. Should I close this issue?

[mlevans0]

Thanks for confirming. 5.0.2 has been released (https://github.com/overleaf/overleaf/wiki/Release-Notes-5.x.x) so I'll close this issue.