Docker version upgrade in 6.8.0 messes with voluming behavious

[eathan-langford-myob]

Describe the Bug
After upgrading Buildkite's elastic-ci stack from a version prior to 6 to 6.10, which included an upgrade of Docker from version 20.10.25 to 24.0.5.
We are now encountering permission issues within our Docker containers.
Despite aligning user and group IDs between the host (buildkite-agent with 2000:2000) and the container (ci-user also with 2000:2000), files in the mounted volumes are showing up with incorrect ownership (root:nobody).
This issue is impacting our ability to execute scripts and commands within the Docker container that require specific file permissions.

Steps to Reproduce

• Set up a Buildkite pipeline using elastic-ci stack version 6.10 (with Docker 24.0.5).
• Use a Dockerfile that sets up a non-root user (ci-user) with specific UID and GID.

run adduser --gecos "" --disabled-password -u 2000 ci-user
run chpasswd <<"ci-user:ci-user"

• Run a Docker container from this image, mounting a host directory to a container directory:

COPY --chown=ci-user:ci-user . /src
WORKDIR /src

docker run --rm -it -e CI=true -v $(pwd):/src -w /src bash -c npm i

• Inside the container, observe that files in the mounted directory (/src) have ownership set to root:nobody instead of the expected ci-user ownership.

ls -la

Expected Behavior
Files in the mounted volume should inherit the correct user and group IDs as aligned between the host and container (in this case, 2000:2000).
This would allow scripts and commands run within the container to access and modify these files without permission issues.

Actual Behaviour
Files in the mounted volume within the Docker container are owned by root:nobody, leading to permission denied errors when attempting to access or modify these files as ci-user especially with npm install.

Stack Parameters
AWS Region: [ap-southeast-2]
Version: 6.10 (with Docker 24.0.5)

Additional Context
This issue was not present in earlier versions of the Buildkite elastic-ci stack (version <6 with Docker 20.10.25).
Attempts to resolve the issue include aligning UID/GID between the host and container, ensuring correct permissions on the host directory, and modifying Docker run commands to accommodate permissions.
The problem seems to be specifically related to changes in how Docker handles file permissions in mounted volumes in version 24.0.5 compared to 20.10.25.

[DrJosh9000]

Hi @eathan-langford-myob, have you by any chance disabled/enabled EnableDockerUserNamespaceRemap?