StorageDriver gcs not registered when updating 2.8.1 -> 2.8.2

[Gerrit91]

I tried to update a registry running in Kubernetes from version 2.8.1 to 2.8.2. The new release image does not come up and instead fails with the following stack trace:

panic: StorageDriver not registered: gcs                                                                                                                                                                           
                                                                                                                                                                                                                   
goroutine 1 [running]:                                                                                                                                                                                             
github.com/docker/distribution/registry/handlers.NewApp({0xe5e8a0?, 0xc00040c180?}, 0xc0003cd880)                                                                                                                  
        github.com/docker/distribution/registry/handlers/app.go:127 +0x2411                                                                                                                                        
github.com/docker/distribution/registry.NewRegistry({0xe5e8a0?, 0xc00040c180?}, 0xc0003cd880)                                                                                                                      
        github.com/docker/distribution/registry/registry.go:161 +0x145                                                                                                                                             
github.com/docker/distribution/registry.glob..func1(0xc0000341a0?, {0xc00012d380, 0x1, 0x1})                                                                                                                       
        github.com/docker/distribution/registry/registry.go:119 +0x18f                                                                                                                                             
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x1369f60, {0xc00012d340?, 0x2?, 0x2?})                                                                                            
        github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x44f                                                                                                                         
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x136a100)                                                                                                                         
        github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0xaa                                                                                                                          
main.main()                                                                                                                                                                                                        
        github.com/docker/distribution/cmd/registry/main.go:23 +0x25 

[milosgajdos]

Very strange because it should be registering itself. None of this code was touched in the last release.

Here's the relevant code:

• distribution/cmd/registry/main.go

  Line 13 in 7c354a4

  _ "github.com/docker/distribution/registry/storage/driver/gcs"

• distribution/registry/storage/driver/gcs/gcs.go

  Line 80 in 7c354a4

  factory.Register(driverName, &gcsDriverFactory{})

Can you paste the config you are using? It's really hard to make any assumptions without it.

[deleteriousEffect]

@milosgajdos GCS is behind a build flag

distribution/registry/storage/driver/gcs/gcs.go

Line 13 in 7c354a4

//go:build include_gcs

,
maybe the build didn't include that?

[milosgajdos]

maybe the build didn't include that?

Ah yeah, that may be the case though they're exported via env vars, but I can't see where this env var is used:

distribution/.github/workflows/ci.yml

Line 12 in 7c354a4

DOCKER_BUILDTAGS: "include_oss include_gcs"

Go build tags make me sad 🫠

That being said, I struggle to see how that worked in the previous release 🤔

distribution/.github/workflows/ci.yml

Line 12 in b5ca020

DOCKER_BUILDTAGS: "include_oss include_gcs"

@Gerrit91 can you confirm GCS driver worked in the previous release?

[Gerrit91]

Hey. Thanks for participating.

Here is the configuration including some Kubernetes commands, which hopefully illustrates the problem in more detail:

❯ k get secrets registry-config -o jsonpath='{.data.config\.yml}' | base64 -d
---
version: 0.1
storage:
  delete:
    enabled: true
  redirect:
    disable: false
  gcs:
    bucket: ...
    credentials: {"type": "service_account", ...}
http:
    addr: :5000
    secret: ...
auth:
  token:
    realm: https://...
    service: "token-service"
    autoredirect: false
    issuer: "docker-auth"
    rootcertbundle: /etc/docker/registry/certs/server.crt
notifications:
  endpoints: [{"name": "docker-registry-ui", "url": "http://registry-ui:8000/api/events", "headers": {"Authorization": ["Bearer ..."]}, "timeout": "1s", "threshold": 5, "backoff": "10s", "ignoredmediatypes": ["application/octet-stream"]}]

❯ k get deployments.apps registry -o yaml | grep image:
      - image: registry:2.8.1

❯ k edit deployments.apps registry 
# changing only the image version to registry:2.8.2

❯ k get po
NAME                             READY   STATUS             RESTARTS     AGE
registry-748cf4d8c7-ltm86        0/1     CrashLoopBackOff   1 (4s ago)   9s
registry-748cf4d8c7-xh7nz        0/1     CrashLoopBackOff   1 (2s ago)   6s
registry-f9bd665f4-bfqnl         1/1     Running            0            36m
registry-f9bd665f4-d9p7f         1/1     Running            0            29m

❯ k logs registry-748cf4d8c7-ltm86
panic: StorageDriver not registered: gcs

goroutine 1 [running]:
github.com/docker/distribution/registry/handlers.NewApp({0xe5e8a0?, 0xc00049f770?}, 0xc000492e00)
        github.com/docker/distribution/registry/handlers/app.go:127 +0x2411
github.com/docker/distribution/registry.NewRegistry({0xe5e8a0?, 0xc00049f770?}, 0xc000492e00)
        github.com/docker/distribution/registry/registry.go:161 +0x145
github.com/docker/distribution/registry.glob..func1(0xc0000b0170?, {0xc000406cc0, 0x1, 0x1})
        github.com/docker/distribution/registry/registry.go:119 +0x18f
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x1369f60, {0xc000406c80?, 0x2?, 0x2?})
        github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x44f
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x136a100)
        github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0xaa
main.main()
        github.com/docker/distribution/cmd/registry/main.go:23 +0x25

[marcusirgens]

Adding the build tags to the go build invocation seems to resolve this issue. I'm struggling to understand how this has worked previously?

diff --git a/Dockerfile b/Dockerfile
index fb54b681..f46e905a 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -27,7 +27,7 @@ RUN --mount=type=bind,target=/go/src/github.com/docker/distribution,rw \
     --mount=type=cache,target=/root/.cache/go-build \
     --mount=target=/go/pkg/mod,type=cache \
     --mount=type=bind,source=/tmp/.ldflags,target=/tmp/.ldflags,from=version \
-      set -x ; xx-go build -trimpath -ldflags "$(cat /tmp/.ldflags) ${LDFLAGS}" -o /usr/bin/registry ./cmd/registry \
+      set -x ; xx-go build -tags "${BUILDTAGS}" -trimpath -ldflags "$(cat /tmp/.ldflags) ${LDFLAGS}" -o /usr/bin/registry ./cmd/registry \
       && xx-verify --static /usr/bin/registry
 
 FROM scratch AS binary

It looks like this might have happened by mistake in #3903. The Dockerfile used to build the v2.8.1 tag clearly passes the BUILDTAGS argument to Go's build command, which no longer is the case for the Dockerfile used to build the v2.8.2 tag.

edit: @milosgajdos kindly let me know that the failing GCS driver code is only failing on main, so I've updated my comment to reflect this.

[milosgajdos]

@marcusirgens you're mentioning a PR on main. 2.8.x tags are done against release/2.8 branch. It seems there is a different issue at play. Also, as for the main branch, we need to update the ancient GCS deps. There is a PR hanging: #3702

[marcusirgens]

@marcusirgens you're mentioning a PR on main. 2.8.x tags are done against release/2.8 branch. It seems there is a different issue at play. Also, as for the main branch, we need to update the ancient GCS deps. There is a PR hanging: #3702

Thanks, I was not aware that main did not contain the latest code. I'll switch to the 2.8 branch and update my comment accordingly.

[valkmit]

Can confirm this is still an issue. Why was the PR referencing this dropped?

[davidspek]

@valkmit The PR was accidentally closed and has been re-opened so we can cut a new patch release once it's merged.

[davidspek]

@valkmit @Gerrit91 The PR to re-enable the build flags has been merged now. Once a new release is cut I will close this issue.

[davidspek]

@valkmit @Gerrit91 v2.8.3 was just released that has the GCS storage driver enabled. As such I'll be closing this issue. If there still is a problem please open a new issue so we can track it accordingly.