Action not fetching images when behind HTTP proxy on self-hosted runner

[grahamb]

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

When running on a self-hosted runner (RHEL8) behind a HTTP proxy, build jobs fail when attempting to fetch the Docker image referenced in the FROM line in the Dockerfile:

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: node:16.19.1-alpine3.17: failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/16.19.1-alpine3.17": dial tcp 3.216.34.172:443: i/o timeout

When running the same docker buildx build command on the runner manually with a checkout of the repo, the build succeeds.

Configuration

• Repository URL (if public): private repo
• Build URL (if public): private repo

Dockerfile:

FROM node:16.19.1-alpine3.17 as build
ARG GITHUB_NPM_TOKEN
ENV NODE_ENV production
USER root
WORKDIR /usr/src/app
[logfile.txt](https://github.com/docker/build-push-action/files/10845193/logfile.txt)

COPY --chown=node:node package*.json ./
COPY --chown=node:node . .
RUN chown node:node /usr/src/app && apk update && apk add --update-cache dumb-init jq && npm -g i pm2
ARG SENTRY_DSN_ARG=""
ENV SENTRY_DSN $SENTRY_DSN_ARG
USER node
ENV NODE_JQ_SKIP_INSTALL_BINARY=true
RUN printf "//npm.github.sfu.ca/:_authToken=${GITHUB_NPM_TOKEN}\n@its:registry=https://npm.github.sfu.ca/" > .npmrc && \ 
  npm set unsafe-perm true && \
  npm config set noproxy "localhost,127.0.0.1,github.sfu.ca,*.sfu.ca,*.its.sfu.ca,docker.github.sfu.ca,www.sfu.ca" && \
  npm install --include=dev  && \
  npm run build && \
  rm /usr/src/app/.npmrc

FROM node:16.19.1-alpine3.17
WORKDIR /usr/src/app
COPY --chown=node:node --from=build /usr/src/app /usr/src/app
RUN chown node:node /usr/src/app && apk add --update-cache dumb-init jq && npm -g i pm2
USER node
EXPOSE 3000
CMD ["dumb-init", "node", "/usr/src/app/server/index.js"]

Build YAML file

name: Build and push to SFU GHCR

on:
  workflow_dispatch:
  push:
    branches:
      - "main"

permissions:
  packages: write
  contents: read

jobs:
  docker:
    runs-on: [self-hosted, lcpd, rhel8]
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3

      - name: Log in to GitHub Docker registry
        uses: docker/login-action@v2
        with:
          registry: containers.github.sfu.ca
          username: ${{ github.actor }}
          password: ${{ github.token }}

      - name: Generate tags
        id: date
        env:
          TAG: containers.github.sfu.ca/its/github-tools/github-tools
        run: |
          echo "TAG_VER=${{ env.TAG }}:$(date +'%Y%m%d%H%M%S')" >> $GITHUB_ENV
          echo "TAG_LATEST=${{ env.TAG }}:latest" >> $GITHUB_ENV

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2.4.1
        with:
          buildkitd-flags: --debug

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          file: Dockerfile-production
          push: true
          context: .
          build-args: |
            GITHUB_NPM_TOKEN=${{ secrets.ITS_DOCKER_PACKAGE_TOKEN }}
            SENTRY_DSN=${{ secrets.PRODUCTION_SENTRY_DSN }}
          tags: |
            ${{env.TAG_VER}}
            ${{env.TAG_LATEST}}

GitHub Runner .env file:

LANG=en_US.UTF-8
http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
no_proxy=sfu.ca,github.sfu.ca

Runner user's ~/.docker/config.json

{
  "auths": {},
  "proxies": {
    "default": {
      "httpProxy": "http://bby-vcontrol-proxy.its.sfu.ca:8080",
      "httpsProxy": "http://bby-vcontrol-proxy.its.sfu.ca:8080",
      "noProxy": "localhost,127.0.0.1,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,containers.github.sfu.ca"
    }
  }
}

Logs

logfile.txt

[subigre]

I have the exact same issue on my side 😞
I am using ARC and I am running the self-hosted runner on a K8s cluster behind a corporate proxy.

Using docker/build-push-action@v4, it fails with the following:

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: summerwind/actions-runner:ubuntu-22.04: failed to do request: Head "https://registry-1.docker.io/v2/summerwind/actions-runner/manifests/ubuntu-22.04": dial tcp 44.205.64.79:443: i/o timeout 
Error: buildx failed with: ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: summerwind/actions-runner:ubuntu-22.04: failed to do request: Head "https://registry-1.docker.io/v2/summerwind/actions-runner/manifests/ubuntu-22.04": dial tcp 44.205.64.79:443: i/o timeout

But, it works fine if I use docker build and docker push commands.

[crazy-max]

Where is the GitHub Runner .env file being set? FWIW, proxy env vars are handled by Buildx so if they are propagated in your workflow it should work.

[crazy-max]

@grahamb If you want to propagate an .env file in your workflow you can add a step like:

      -
        name: Propagate .env
        run: |
          for l in "$(cat .env)"; do
            echo "${l?}" >> $GITHUB_ENV
          done

@subigre Can you post a repro with env vars being used and your workflow please and also enable BuildKit debug logs.

If you can also add this step inside your workflow to check the context:

      -
        name: Dump context
        uses: crazy-max/ghaction-dump-context@v2

[subigre]

@crazy-max, here is the workflow I used:

name: Docker

on:
  workflow_dispatch:

env:
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push:
  
  runs-on: [ self-hosted ]
  
  steps:
    - name: Checkout repository
      uses: actions/checkout@v3
   
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v2
    
    - name: Set up Docker context
      run: docker context create builders 

   - name: Set up Docker Buildx
     uses: docker/setup-buildx-action@v2 
     with:
       endpoint: builders
    
    - name: Login to GitHub Container Registry
      uses: docker/login-action@v2
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: false
        tags: ${{ env.IMAGE_NAME }}

And the output logs:
output-logs.txt

In addition, here are the logs from Set up job step and it is indicated that our proxy is configured:

Current runner version: '2.303.0' 
Runner name: 'runner-deployment-njjbm-gvpcm' 
Runner group name: 'Default' 
Machine name: 'runner-deployment-njjbm-gvpcm' 
GITHUB_TOKEN Permissions 
Secret source: Actions 
Runner is running behind proxy server 'http://10.0.0.1:3128/' for all HTTP requests. 
Runner is running behind proxy server 'http://10.0.0.1:3128/' for all HTTPS requests. 
Prepare workflow directory 
Prepare all required actions 
Getting action download info 
Download action repository 'actions/checkout@v3' (SHA:2541b1294d2704b0964813337f33b291d3f8596b) 
Download action repository 'docker/setup-qemu-action@v2' (SHA:e81a89b1732b9c48d79cd809d8d81d79c4647a18) 
Download action repository 'docker/setup-buildx-action@v2' (SHA:4b4e9c3e2d4531116a6f8ba8e71fc6e2cb6e6c8c) 
Download action repository 'docker/login-action@v2' (SHA:f4ef78c080cd8ba55a85445d5b36e214a81df20a) 
Download action repository 'docker/build-push-action@v4' (SHA:3b5e8027fcad23fda98b2e3ac259d8d67585f671) 
Complete job name: build-and-push

Finally, here are the info display in the Docker info by docker/build-push-action@v4:

Server: Docker Engine - Community 
Engine: 
Version: 23.0.1 
API version: 1.42 (minimum version 1.12) 
Go version: go1.19.5 
Git commit: bc3805a 
Built: Thu Feb 9 19:48:50 2023 
OS/Arch: linux/amd64 
Experimental: false 
containerd: 
Version: v1.6.16 
GitCommit: 31aa4358a36870b21a992d3ad2bef29e1d693bec 
runc: 
Version: 1.1.4 
GitCommit: v1.1.4-0-g5fd4c4d 
docker-init: 
Version: 0.19.0 
GitCommit: de40ad0 
/usr/local/bin/docker info 
Client: 
Context: default 
Debug Mode: false 
Plugins: 
buildx: Docker Buildx (Docker Inc., v0.10.4) 
Server: 
Containers: 1 
Running: 1 
Paused: 0 
Stopped: 0 
Images: 2 
Server Version: 23.0.1 
Storage Driver: overlay2 
Backing Filesystem: xfs 
Supports d_type: true 
Using metacopy: false 
Native Overlay Diff: true 
userxattr: false 
Logging Driver: json-file 
Cgroup Driver: cgroupfs 
Cgroup Version: 1 
Plugins: 
Volume: local 
Network: bridge host ipvlan macvlan null overlay 
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog 
Swarm: inactive 
Runtimes: io.containerd.runc.v2 runc 
Default Runtime: runc 
Init Binary: docker-init 
containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec 
runc version: v1.1.4-0-g5fd4c4d 
init version: de40ad0 
Security Options: 
seccomp 
Profile: builtin 
Kernel Version: 4.18.0-425.10.1.el8_7.x86_64 
Operating System: Alpine Linux v3.17 (containerized) 
OSType: linux 
Architecture: x86_64 
CPUs: 4 
Total Memory: 31.37GiB 
Name: runner-deployment-njjbm-gvpcm 
ID: 9599b5f1-31f9-48a2-b5ff-19dde04d00c4 
Docker Root Dir: /var/lib/docker 
Debug Mode: false 
HTTP Proxy: http://10.0.0.1:3128/
HTTPS Proxy: http://10.0.0.1:3128/
No Proxy: localhost,127.0.0.1
Registry: https://index.docker.io/v1/ 
Labels: 
Experimental: false 
Insecure Registries: 
127.0.0.0/8 
Live Restore Enabled: false 
Product License: Community Engine 
WARNING: bridge-nf-call-iptables is disabled 
WARNING: bridge-nf-call-ip6tables is disabled

[crazy-max]

on a K8s cluster behind a corporate proxy.

@subigre How have you setup your cluster?

endpoint: builders

What is this endpoint? I don't see any logs about it in output-logs.txt.

[Wenzel]

Hi,

I can confirm this issue on my end as well.
Specifying the proxy args using ~/.docker/config.json or the Action's build-args parameter doesn't work.

[crazy-max]

I just made a test in this repo to check if proxy envs are propagated to RUN instructions in #872 and looks good: https://github.com/docker/build-push-action/actions/runs/5175871069/workflow?pr=872#L1021-L1025 (logs: https://github.com/docker/build-push-action/actions/runs/5175651929/jobs/9323506310?pr=872#step:7:214)

build jobs fail when attempting to fetch the Docker image referenced in the FROM line in the Dockerfile:

@grahamb Proxy config support for registry pull or git clones should work when setting up BuildKit using the setup-buildx-action like #872: https://github.com/docker/build-push-action/actions/runs/5175871069/workflow#L1066-L1067 (logs: https://github.com/docker/build-push-action/actions/runs/5175871069/jobs/9324012887#step:5:137).

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver-opts: |
            env.http_proxy=http://1.2.3.4:3128
            env.https_proxy=http://1.2.3.4:3128
          buildkitd-flags: --debug

You might also be interested in moby/buildkit#3750.

@dvdksn Maybe some docs on setting up proxy config in GHA would be useful.

Closing this issue since that should answer this question, but feel free to left a comment with good repro (workflow, BuildKit logs) if you still encounter an issue. Thanks.