Image list with containerd store is very slow

[cpuguy83]

Description

docker images is very slow when backed by the containerd store.

Upon inspect it looks like every image is being processed sequentially, where "process" involves a number of API requests to containerd.

Listing 68 images takes roughly a full second.
As part of that there 2891 spans from containerd.
Some of that even involves calculating disk usage from the snapshotters.

This is after I've pruned some images.
Before pruning I had about 325 images and that took approximately 5s.

Reproduce

docker pull a few images and read the trace data

Expected behavior

Image listing should scale better

docker version

docker version
Client:
 Version:           24.0.0-beta.1-450-gfa5a0e8139.m
 API version:       1.44
 Go version:        go1.21.1
 Git commit:        fa5a0e8139
 Built:             Sun Oct  1 19:58:37 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          dev
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.0
  Git commit:       bddd892e9108a7944df8fca9fde821bb97616bab
  Built:            Thu Mar 14 19:29:50 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.14-1
  GitCommit:        dcf2847247e18caba8dce86522029642f60fe96b
 runc:
  Version:          1.1.12-1
  GitCommit:        51d5e94601ceffbbd85688df1c928ecccbfa4685
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.0-beta.1-450-gfa5a0e8139.m
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.0-1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.24.7-1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 2
  Paused: 0
  Stopped: 8
 Images: 66
 Server Version: dev
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc runc-patched
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dcf2847247e18caba8dce86522029642f60fe96b
 runc version: 51d5e94601ceffbbd85688df1c928ecccbfa4685
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1016-azure
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.33GiB
 Name: dev2
 ID: ccb953ef-cff6-4d73-94de-24e41bbff5c2
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 36
  Goroutines: 62
  System Time: 2024-03-14T19:44:24.766150687Z
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

I'm using the last build off master that actually successfully lists images for me (It seems there's another bug on master introduced by 52a80b4

Using the current master (cdf70c0) things are actually a bit slower and even more API requests are made to containerd (more than 1000 more spans!)

Attached jaeger json, this is from a different capture than reported above, but similar results
traces-1710445565373.json.zip

From the flamegraph you'll see there's nothing that's taking a lot of time in particular, just the act of processing things sequentially:

[cpuguy83]

/cc @vvoland

[thaJeztah]

For the slowness, my first thinking was this PR, but that was already in rc1;

• c8d: Adjust "image list" to return only a single item for each image store entry #45967

This one could be related, but was already in rc2;

• c8d/list: Add test and combine size #47449

[thaJeztah]

Quick check on my machine running 26.0.0-rc2 with 38 images showed nearly 900ms on a first run which indeed doesn't seem very fast no.

docker image ls | wc -l
      38

time docker image ls -a

real	0m0.855s
user	0m0.042s
sys	0m0.022s