docker load fails with lsetxattr com.apple.provenance /manifest.json: operation not supported on Ventura & later

[Strum355]

Description

Starting with macOS Ventura 13, macOS adds additional xattr metadata to downloaded files. The following article explains this in more detail: https://eclecticlight.co/2023/05/10/how-macos-now-tracks-the-provenance-of-apps/

We've observed on some systems, when building OCI images (via Bazel), we would get the error in the title when attempting to docker load --input <file> the tarball. To workaround this issue, the affected users have had to untar the image, and retar using tar --no-xattr.

Reproduce

If your system doesn't set these attributes already, you can recreate it with any existing image tarball:

$ tar xf <image_tarball_file>
$ xattr -w com.apple.provenance 'sample text' manifest.json
$ tar cf output.tar *
$ docker load --input output.tar
lsetxattr com.apple.provenance /manifest.json: operation not supported

Link to an affected image tarball for convenience (GH won't allow me to attach a tar file): https://cdn.discordapp.com/attachments/312648998328729603/1214914140821258320/tarball.tar. Built from bazel build //cmd/batcheshelper:image_tarball in https://github.com/sourcegraph/sourcegraph

Expected behavior

docker load ignores unsupported xattrs, similarly to #47483

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:26 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.2 (137060)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/noah/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /Users/noah/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /Users/noah/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/noah/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/noah/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/noah/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/noah/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/noah/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /Users/noah/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 21
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 5.79GiB
 Name: docker-desktop
 ID: b63f794a-cd4c-467e-b52e-5e02df615db9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[nise-wg2]

Similar issue w latest Docker desktop on OSX M3.

/bin/bash -c '/Users/xxx/tmp/bazel/base/execroot/main/bazel-out/darwin_arm64-fastbuild/bin/test/integration/intenv/intenv_/intenv /Users/xxx/tmp/bazel/base/execroot/main/bazel-out/darwin_arm64-fastbuild-ST-4a519fd6d3e4/bin/cdr/cdr-events-in/container_image.executable '
lsetxattr com.apple.provenance /f52af47487c7689b94bd0d30141a31953d1e9ff251a758202fc41a45a14a1f9f.tar: operation not supported
tar: Write error

To fix the issue, rolling back to

✗ docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

And it works again.

[thaJeztah]

Thanks for reporting. I think this is related to a bug-fix where previously failures on applying extended-attributes were silently ignored, which in some cases could result to an image that was silently broken (due to those attributes getting lost in the process). The expectation for docker load from a tarball is to restore the filesystem from the tarball, including attributes set on the file. In this case, the generated tarbal contains extended-attributes that are specific to macOS, and that cannot be applied to Linux filesystems (causing an error to be produced). Admitted, handling of extended attributes is not well-defined in the OCI image specification (there's some discussion ongoing in this area to have a better definition).

There's also a difference between Linux and macOS (bsd) tar in handling extended attributes. On Linux, creating a tar by default omits all extended attributes, whereas on macOS the reverse is true, and extended-attributes are included by default. If the files you're adding to the tar are not expected to have extended attributes set that must be preserved (i.e., if they only contain extended attributes added by macOS), then I would recommend using the --no-xattrs options when creating the archive. Also see #47137 (comment)

Looking at your report though, it looks like the failure is happening on the manifest.json, which is not part of the the image rootfs, so perhaps there's something to address here to exclude such files.

cc @vvoland @neersighted @corhere

[corhere]

In contrast to #47483, which had to do with authoring of images where the source inputs were tarballs containing xattrs; this issue has to do with the importing of already-authored images. Images are an interchange format authored by specialized tooling so it would not necessarily be unreasonable for docker load to be stricter than a Dockerfile ADD instruction about what is acceptable. Furthermore, manifest.json is the "Docker-compatible manifest" which is part of the Docker image repository archive format, not the OCI Image specification. Correct me if I'm wrong, but I suspect dockerd is the only consumer of manifest.json files in image repository archives. I could not find a formal specification for the Docker image repository archive format (only the contents of a distribution manifest) so assuming one does not exist, the specification is de facto "what docker save produces and what docker load accepts." If all my preceding speculation is true, we would be entirely within our rights to say that Moby is working as intended and the bug is entirely within rules_oci for authoring a Docker-compatible image repository archive which is incompatible with Docker. We would also be entirely within our rights to unilaterally decide to be more liberal in what we accept and change docker load to ignore file xattrs on manifest.json. Barring new information, neither option is more technically correct than the other so we will have to decide on a course of action some other way.

My read on this issue is that it mostly only impacts direct users of rules_oci -- image authors, by definition. End-users pulling images from registries will not be affected due to how image manifest lists are returned directly in HTTP responses when pulling images rather than being packed into tarballs. Given the users impacted are limited to direct users of the image authoring tooling and the fact that the root cause is being addressed in the authoring tooling, I am inclined to consider this a bug in rules_oci which Moby should not work around.

[vvoland]

I could not find a formal specification for the Docker image repository archive format (only the contents of a distribution manifest) so assuming one does not exist, the specification is de facto "what docker save produces and what docker load accepts."

We have
https://github.com/moby/docker-image-spec/blob/main/spec.md#combined-image-json--filesystem-changeset-format - but it doesn't cover this specific subject.

Also, this behavior is strictly implementation-specific due to how archive package used by graphdriver works. The manifest.json isn't expected to be persisted, but is still extracted into a temporary daemon directory using the same tar unpack code which also attempts to restore the xattrs.

It wouldn't happen with the containerd image store backend enabled, because it reads the manifest.json directly from the tar and only cares about its content.