(aws-ecr-assets): DockerImageAsset cacheTo property causes "unknown flag: --cache-to" error

[jamesmcglinn]

Describe the bug

CDK DockerImageAsset has a recently-added cacheTo property.

This option should allow the Docker build cache to be exported, in order to be used to improve performance of future builds.

But instead using this property in a CDK CodePipeline causes a CodeBuild error.

Expected Behavior

The cacheTo property should save a Docker build cache to the selected destination, in this case ECR.

Current Behavior

Instead the build process fails with the following unknown flag: --cache-to error:

[23:55:56] [20%] debug: docker --config /tmp/cdkDockerConfigH10b build --tag cdkasset-2991adba35b --file Dockerfile --platform linux/arm64 --cache-to 'type=registry,ref=${Token[TOKEN.608]}.dkr.ecr.${Token[TOKEN.607]}.${Token[AWS.URLSuffix.5]}/${Token[TOKEN.602]}:tag,mode=max' .
unknown flag: --cache-to

Reproduction Steps

const cache = new ecr.Repository(this, 'Cache');
const image = new DockerImageAsset(this, 'Image', {
  directory: join(__dirname, '../'),
  file: 'Dockerfile',
  platform: Platform.LINUX_ARM64,
  cacheTo: {
    type: 'registry',
    params: {
      ref: `${cache.repositoryUri}:tag`,
      mode: 'max',
    },
  }
});

Possible Solution

This seems to be because the CodeBuild Docker build process isn't using BuildKit.

I've tried using the following config but it still runs docker build rather than docker buildx build. Adding the same environment to selfMutationCodeBuildDefaults also doesn't seem to help.

codeBuildDefaults: {
  buildEnvironment: {
    environmentVariables: {
      DOCKER_BUILDKIT: {
        value: '1',
      },
    },
  },
},

Additional Information/Context

Some properties have a comment referencing BuildKit as a requirement, perhaps this property also needs a similar comment:

    /**
     * Build secrets.
     *
     * Docker BuildKit must be enabled to use build secrets.
     *
     * @see https://docs.docker.com/build/buildkit/
     *
     * @default - no build secrets
     *
     * @example
     *
     * {
     *   'MY_SECRET': DockerBuildSecret.fromSrc('file.txt')
     * }
     */
    readonly buildSecrets?: {
        [key: string]: string;
    };

Although I haven't been able to find documentation on enabling BuildKit for DockerImageAsset.

CDK CLI Version

2.69.0

Framework Version

No response

Node.js Version

16

OS

LinuxBuildImage.STANDARD_6_0

Language

Typescript

Language Version

No response

Other information

The pull requests adding the cacheTo property: #24024 & #24524

[pahud]

--cache-from and --cache-to seem to be only supported with buildx
https://docs.docker.com/build/cache/backends/

Unfortunately CodeBuild doesn't natively support buildx at this moment. I am not sure if there's any workaround but we definitely should improve our docs on this.

[jamesmcglinn]

Even with a custom build image with buildx installed and a container driver selected, I'm not sure there's a way to tell DockerImageAsset() to use the container driver rather than the default docker driver which doesn't support --cache-from and --cache-to.

CodeBuild output looks like:

[Container] 2023/04/01 03:19:59 Running command docker buildx inspect --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.9s done
#1 creating container buildx_buildkit_container0
#1 creating container buildx_buildkit_container0 1.5s done
#1 DONE 4.4s
Name:          container
Driver:        docker-container
Last Activity: 2023-03-31 22:47:24 +0000 UTC
Nodes:
Name:      container0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Buildkit:  v0.11.5
Platforms: linux/arm64, linux/arm/v7, linux/arm/v6
[Container] 2023/04/01 03:20:04 Running command docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
container *  docker-container                             
  container0 unix:///var/run/docker.sock running v0.11.5  linux/arm64, linux/arm/v7, linux/arm/v6
default      docker                                       
  default    default                     running 23.0.2   linux/arm64, linux/arm/v7, linux/arm/v6
...
[03:20:06] [20%] build: Building Docker image at /codebuild/output/src859267221/src/asset.7e06e78f69d4b662e11929b140b14977c62cffffef37165d44d1758798805483
[03:20:06] [20%] debug: docker --config /tmp/cdkDockerConfig4IyR8e build --tag cdkasset-7e06e78f69d4b662e11929b140b14977c62cffffef37165d44d1758798805483 --file apps/app/Dockerfile --platform linux/arm64 --cache-to
[40%] fail: docker --config /tmp/cdkDockerConfig4IyR8e build --tag cdkasset-7e06e78f69d4b662e11929b140b14977c62cffffef37165d44d1758798805483 --file apps/app/Dockerfile --platform linux/arm64 --cache-to 'type=registry,ref=${Token[TOKEN.605]}.dkr.ecr.${Token[TOKEN.604]}.${Token[AWS.URLSuffix.1]}/${Token[TOKEN.599]}:app,mode=max' . exited with error code 1: ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

[pahud]

AWS CDK honors process.env.CDK_DOCKER when it bundles the image assets. I am not sure if this would be helpful?

aws-cdk/packages/aws-cdk-lib/core/lib/private/asset-staging.ts

Line 200 in a2c633f

const prog = process.env.CDK_DOCKER ?? 'docker';

[jamesmcglinn]

AWS CDK honors process.env.CDK_DOCKER when it bundles the image assets. I am not sure if this would be helpful?

Yes overriding CDK_DOCKER with the script below calling docker-buildx directly works for this build step.

#!/bin/bash
if [ "$3" == "build" ]
  then set -- "${@:4} --load"
fi

~/.docker/cli-plugins/docker-buildx build $@

[noreason]

Yes overriding CDK_DOCKER with the script below calling docker-buildx directly works for this build step.

#!/bin/bash
if [ "$3" == "build" ]
  then set -- "${@:4} --load"
fi

~/.docker/cli-plugins/docker-buildx build $@

Do you mind showing in a bit more detail how you got this working?

It seemed logical enough to me, create the script, tell CodeBuild to use it, but I must be doing something wrong because I can't get the script to execute. When it's time for docker to run, it just throws the error [100%] fail: Unable to execute 'docker' in order to build a container asset. Please install 'docker' and try again.

[jamesmcglinn]

@noreason I created a custom image based on the official AWS CodeBuild images (in my case al2-aarch64-standard-2.0) by adding the instructions below to the Dockerfile (before ENTRYPOINT). Make sure you get the right binary for your architecture:

ENV BUILDX_VERSION="0.10.4"
ENV BUILDX_ARCH="arm64"
RUN set -ex \
    && wget -q "https://github.com/docker/buildx/releases/download/v$BUILDX_VERSION/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" -O "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" \
    && chmod a+x "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" \
    && mkdir -p ~/.docker/cli-plugins \
    && mv "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" ~/.docker/cli-plugins/docker-buildx \
    && docker buildx install

In the Dockerfile copy the custom script above into the image around where it's copying in the entrypoint script:

COPY dockerd-entrypoint.sh /usr/local/bin/
COPY customdockerx /usr/local/bin/
COPY legal/THIRD_PARTY_LICENSES.txt /usr/share/doc

Build the image, push it to ECR or Docker Hub, and configure it as your custom build image. You'll need to include these commands in your buildspec PRE_BUILD phase:

/usr/local/bin/dockerd-entrypoint.sh
docker buildx create --name=container --driver=docker-container --use --bootstrap

And set these environment variables:

DOCKER_BUILDKIT=1
BUILDX_BUILDER=container
CDK_DOCKER=customdockerx

Be aware ECR doesn't support cache manifests yet so if you're not using your own or an external image repository you'll need to wait for aws/containers-roadmap#876.

[noreason]

@noreason I created a custom image based on the official AWS CodeBuild images (in my case al2-aarch64-standard-2.0) by adding the instructions below to the Dockerfile (before ENTRYPOINT). Make sure you get the right binary for your architecture:

ENV BUILDX_VERSION="0.10.4"
ENV BUILDX_ARCH="arm64"
RUN set -ex \
    && wget -q "https://github.com/docker/buildx/releases/download/v$BUILDX_VERSION/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" -O "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" \
    && chmod a+x "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" \
    && mkdir -p ~/.docker/cli-plugins \
    && mv "/tmp/buildx-v$BUILDX_VERSION.linux-$BUILDX_ARCH" ~/.docker/cli-plugins/docker-buildx \
    && docker buildx install

In the Dockerfile copy the custom script above into the image around where it's copying in the entrypoint script:

COPY dockerd-entrypoint.sh /usr/local/bin/
COPY customdockerx /usr/local/bin/
COPY legal/THIRD_PARTY_LICENSES.txt /usr/share/doc

Build the image, push it to ECR or Docker Hub, and configure it as your custom build image. You'll need to include these commands in your buildspec PRE_BUILD phase:

/usr/local/bin/dockerd-entrypoint.sh
docker buildx create --name=container --driver=docker-container --use --bootstrap

And set these environment variables:

DOCKER_BUILDKIT=1
BUILDX_BUILDER=container
CDK_DOCKER=customdockerx

Be aware ECR doesn't support cache manifests yet so if you're not using your own or an external image repository you'll need to wait for aws/containers-roadmap#876.

Thank you for this explanation. Bummer the amount of workaround needed to get this going.