Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tar errors on cache restore after toolchain installation #424

Open
2 of 5 tasks
matthewhughes-uw opened this issue Sep 15, 2023 · 11 comments · May be fixed by #438
Open
2 of 5 tasks

Tar errors on cache restore after toolchain installation #424

matthewhughes-uw opened this issue Sep 15, 2023 · 11 comments · May be fixed by #438
Assignees
Labels
bug Something isn't working

Comments

@matthewhughes-uw
Copy link

matthewhughes-uw commented Sep 15, 2023

Description:

When the version of Go in the go directive of go.mod is different from the version in the toolchain directive there is a cache conflict which causes errors on cache extraction.

Note: I think this is related to, but different from #403 since that appears to be covering the situation where some other tool is also working on the cache, but this issue is purely for the action running on a hosted runner without any additional changes to the environment etc.

Action version:

actions/setup-go@v4

Platform:

  • Ubuntu
  • macOS (untested)
  • Windows (untested)

Runner type:

  • Hosted
  • Self-hosted

Tools version:

  • go 1.21.0
  • toolchain go1.21.1

Repro steps:

See repo https://github.com/matthewhughes-uw/setup-go-test/, very basic repo with a single workflow that just runs this action (https://github.com/matthewhughes-uw/setup-go-test/blob/main/.github/workflows/go.yaml):

First run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195831453/job/16821285537

  • Installs Go 1.21.0 (version detected from go-version-file
  • Go detects toolchain and performs a toolchain install of Go 1.21.1 (see output go: downloading go1.21.1 (linux/amd64))
  • This places some files under ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.1.linux-amd64
  • Finds no existing cache, doesn't try to extract anything
  • Caches /home/runner/go/pkg/mod (which contains the files above) and /home/runner/.cache/go-build

Second run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195844389/job/16821321968

  • First two steps as above
  • Finds an existing cache, attempts to extract it, but because of the toolchain installation step files under ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.1.linux-amd64 are already present on the machine, causing errors from tar

Expected behavior:

Caching of the toolchain is handled gracefully, no errors between runs that don't contain any extra caching logic outside of what the action provides by default

Actual behavior:

Errors on cache extraction (see repro steps)

@matthewhughes-uw matthewhughes-uw added bug Something isn't working needs triage labels Sep 15, 2023
@matthewhughes-uw matthewhughes-uw changed the title Tar errors on cache restore with differing go and toolchain directives Tar errors on cache restore after toolchain installation Sep 15, 2023
@dsame
Copy link
Contributor

dsame commented Sep 15, 2023

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

@dsame dsame self-assigned this Oct 16, 2023
@dsame dsame linked a pull request Oct 24, 2023 that will close this issue
2 tasks
@dsame dsame linked a pull request Oct 24, 2023 that will close this issue
2 tasks
@dsame dsame linked a pull request Oct 24, 2023 that will close this issue
2 tasks
@dsame dsame removed a link to a pull request Oct 25, 2023
2 tasks
@thall
Copy link

thall commented Jan 8, 2024

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

do you have any updates around this to share?

@HaraldNordgren
Copy link

@matthewhughes-uw Any updates on this? After updating to Go version:

--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,8 @@
 module github.com/company/repo

-go 1.20
+go 1.21.1
+
+toolchain go1.21.6

 require (

We start having these warnings in the build:

/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/README: Cannot open: File exists
...

@HaraldNordgren
Copy link

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.

  • It first downloads and caches 1.21.1
  • Then tries to download 1.21.6 as toolchain version
  • But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
  • And boom... it 1.21.1 already exists in the cache from the first step! 🧨
15s
Run actions/setup-go@v4
  with:
    go-version-file: go.mod
    check-latest: true
    token: ***
    cache: true
Setup go version spec [1](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:1).[2](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:2)1.1
Attempting to resolve the latest version from the manifest...
matching 1.21.1...
Resolved as '1.21.1'
Attempting to download 1.21.1...
matching 1.21.1...
Acquiring 1.21.1 from https://github.com/actions/go-versions/releases/download/1.21.1-6107260229/go-1.21.1-linux-x64.tar.gz
Extracting Go...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487 -f /home/runner/work/_temp/b0a91240-cd7a-42cc-bc74-0757c51b5e9b
Successfully extracted go to /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487
Adding to the cache ...
Successfully cached go to /opt/hostedtoolcache/go/1.21.1/x64
Added go to the path
Successfully set up Go version 1.21.1
go: downloading go1.21.6 (linux/amd64)
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOMODCACHE
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOCACHE
/home/runner/go/pkg/mod
/home/runner/.cache/go-build
Received 151688858 of 151688858 (100.0%), 166.5 MBs/sec
Cache Size: ~145 MB (151688858 B)
/usr/bin/tar -xf /home/runner/work/_temp/22489b85-a7d8-49[3](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:3)7-8083-e821[4](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:4)04[5](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:5)3fc7/cache.tzst -P -C /home/runner/work/hive/hive --use-compress-program unzstd
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.[6](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:6).linux-amd64/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.[21](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:22).6.linux-amd64/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/README: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/update.bash: Cannot open: File exists
...

@matthewhughes-uw
Copy link
Author

matthewhughes-uw commented Jan 18, 2024

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.

* It first downloads and caches 1.21.1
* Then tries to download 1.21.6 as toolchain version
* But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
* And boom... it 1.21.1 already exists in the cache from the first step! 🧨

Note: the toolchain download is not done by setup-go, it's done by Go itself, e.g.

module example.com/my-mod

go 1.21.0

toolchain go1.21.1

Then if you try to run a go command using a version that doesn't match the toolchain the toolchain will be downloaded (creating new caches to ensure we download the new toolchain, and not just silently use an existing one):

$ mkdir gocache gomodcache
$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 env
go: downloading go1.21.1 (linux/amd64)

So unless setup-go is configured to use the go version (or perhaps, any newer version) specified in the toolchain directive then there will be this download. setup-go then continues to use the version it was configured with to run any future go commands (which may just invoke the toolchain version in the background)

$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version
go version go1.21.1 linux/amd64

EDIT: I'm not sure if setup-go could just set the GOTOOLCHAIN env var to ensure it always use the configured version (docs https://go.dev/doc/toolchain)

The GOTOOLCHAIN environment setting can force a specific Go version, overriding the go and toolchain lines (and possibly just use the version from the toolchain directive by default)

$ GOTOOLCHAIN=go1.21.0 GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version 
go version go1.21.0 linux/amd64

(from a quick test this skipped any toolchain download)

@HaraldNordgren
Copy link

HaraldNordgren commented Jan 18, 2024

@matthewhughes-uw That makes sense. What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

Because if this didn't happen, then there would be no cache problem.

How I solved it for now is to upgrade using go get go, upgrading to 1.21.6 locally, which then removes the toolchain line altogether, since Go and Toolchain versions are now the same.

Then the caching issue does not appear.

@matthewhughes-uw
Copy link
Author

matthewhughes-uw commented Jan 22, 2024

What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

There's a mix of Go and setup-go downloading bits. What I understand from your previous point:

  • It first downloads and caches 1.21.1: This is setup-go, and ends with the line Successfully set up Go version 1.21.1
  • Then tries to download 1.21.6 as toolchain version: setup-go then invokes go (specifically /opt/hostedtoolcache/go/1.21.1/x64/bin/go) which triggers a download of the toolchain, this is the line go: downloading go1.21.6 (linux/amd64)
  • But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache for go1.21.1: This is setup-go downloading the requested cache (not go doing anything)
  • And boom... it 1.21.1 already exists in the cache from the first step! 🧨: the issue is toolchain@v0.0.1-go1.21.6.linux-amd64/ exists in the cache (the toolchain downloaded from the 2nd step)

So the issue is we populate some of go1.21.1's GOMODCACHE and then restore the cache, which then complains because there are already some files there

You could remove the toolchain directive from your go.mod (unless you do have a requirement for a specific version of Go greater than what the go directive specifies when working in the module, see https://go.dev/blog/toolchain#toolchain-management)

i4ki added a commit to terramate-io/terramate that referenced this issue Feb 2, 2024
Set the same Go version as specified in the `go.mod`
because of the issue below:
- actions/setup-go#424

Signed-off-by: Tiago Natel <t.nateldemoura@gmail.com>
i4ki added a commit to terramate-io/terramate that referenced this issue Feb 2, 2024
Set the same Go version as specified in the `go.mod`
because of the issue below:
- actions/setup-go#424

Additionally, the rand.Seed() was updated to use
rand.New(rand.NewSource(seed)) because the former was
removed from Go1.20.

Signed-off-by: Tiago Natel <t.nateldemoura@gmail.com>
i4ki added a commit to terramate-io/terramate that referenced this issue Feb 2, 2024
Set the same Go version as specified in the `go.mod`
because of the issue below:
- actions/setup-go#424

Additionally, the rand.Seed() was updated to use
rand.New(rand.NewSource(seed)) because the former was
removed from Go1.20.

Signed-off-by: Tiago Natel <t.nateldemoura@gmail.com>
i4ki added a commit to terramate-io/terramate that referenced this issue Feb 2, 2024
Set the same Go version as specified in the `go.mod`
because of many similar issues related to the setup-go
cache unable to work when go.mod specifies a different
version.

- actions/setup-go#424

Additionally, the rand.Seed() was updated to use
rand.New(rand.NewSource(seed)) because the former was
removed from Go1.20.

Signed-off-by: Tiago Natel <t.nateldemoura@gmail.com>
@spencerschrock
Copy link

One way of avoiding this could be if actions/setup-go had a way of reading the toolchain directive instead of the go directive. Although it's still useful to test against the minimum version specified by the go directive, so I don't think changing the behavior of go-version-file is a good solution.

You could do your own toolchain version parsing in an earlier step, and set an env var or something:

uses: actions/setup-go@v5
with:
    go-version: ${{ env.GO_VERSION }}

matthewhughes-uw pushed a commit to matthewhughes-uw/setup-go that referenced this issue Feb 8, 2024
`go version` is run before downloading the cache, but if this is run
with a version of `go` that triggers a Toolchain download[1], e.g. if
the installed Go is 1.20.0 but `go.mod` has a toolchain directive
`go1.20.1` then a toolchain is downloaded to e.g.
`$GOMODCACHE/golang.org/toolchain@v0.0.1-go1.21.1.linux-amd64`, if this
file already exists in the cache (e.g. this is the second run of this
action we not cache invalidation) then the cache download will try and
overwrite these files resulting in noisy errors like:

    /usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/mkzip.go: Cannot open: File exists

Instead, force `go` to use the local toolchain (i.e. the one the one
that shipped with the go command being run) via setting the
`GOTOOLCHAIN` environment variable[1]:

> When GOTOOLCHAIN is set to local, the go command always runs the
bundled Go toolchain.

This addresses actions#424

[1] https://go.dev/doc/toolchain#select
@matthewhughes-uw
Copy link
Author

I think this can be avoided by just telling Go to use the local toolchain when running go version before downloading the cache, see: #456

I think there's another can of worms/separate issue around which version of Go should this action use when a toolchain directive is present, e.g. at the moment if you specify go-version but have a go.mod specifying a newer-version of Go I expect any go command will run using the toolchain version, which may be unexpected

i4ki added a commit to terramate-io/terramate that referenced this issue Feb 12, 2024
## What this PR does / why we need it:

Set the same Go version as specified in the `go.mod`
because of many similar issues related to the setup-go
cache unable to work when go.mod specifies a different
version.

- actions/setup-go#424

Additionally, the rand.Seed() was updated to use
rand.New(rand.NewSource(seed)) because the former was
removed from Go1.20.

## Which issue(s) this PR fixes:
no

## Special notes for your reviewer:

## Does this PR introduce a user-facing change?
```
no
```
@lukeed
Copy link

lukeed commented Apr 7, 2024

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

// go.mod
go 1.22.2
# workflows/ci.yml
- uses: actions/setup-go@v5
  with:
    go-version: 1.22.2

@matthewhughes-uw
Copy link
Author

matthewhughes-uw commented Apr 9, 2024

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

You could also set the GOTOOLCHAIN env var to local:

    - name: Configure env for toolchain ignore
      shell: bash
      run: echo "GOTOOLCHAIN=local" >> "$GITHUB_ENV"
    - name: Setup Go
      uses: actions/setup-go@v5

see #460 for more details

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants