Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] "Generate builder" and "Run sigstore/cosign-installer" steps failing with error updating to TUF remote mirror: invalid key #3350

Closed
jkreileder opened this issue Mar 19, 2024 · 23 comments · Fixed by #3355
Labels
status:triage Issue that has not been triaged type:bug Something isn't working

Comments

@jkreileder
Copy link
Contributor

The "Generate builder" and "Run sigstore/cosign-installer" steps have started failing for my workflows. This used to work fine, not sure if it is just an intermittent error or something more fundamental:

Here's a build that worked 18 hours ago but is failing now (i.e. without any code changes): https://github.com/jkreileder/cf-ips-to-hcloud-fw/actions/runs/8339143012 (corresponding workflow)

  1. Generate builder error:
Verifying artifact slsa-generator-container-linux-amd64: FAILED: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}

FAILED: SLSA verification failed: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 6.
  1. Run sigstore/cosign-installer@6e04d228eb30da1757ee4e1dd75a0ec73a653e06 error:
Error: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
main.go:74: error during command execution: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 1.
@jkreileder jkreileder added status:triage Issue that has not been triaged type:bug Something isn't working labels Mar 19, 2024
@tirumerla
Copy link
Contributor

Started seeing the same just now.

@brad-getpassport
Copy link

brad-getpassport commented Mar 19, 2024

+1 Seeing this across all our builds in the last 2 hours... this is holding up our release candidate...

Generating ephemeral keys...
Retrieving signed certificate...
creating signer: getting signer: getting key from Fulcio: getting CTFE public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 1.

@jenstroeger
Copy link

‼️ @ianlewis @laurentsimon please advise! This is breaking all our build pipelines, and talking with folks their pipelines are also dead in the water ‼️

@haydentherapper
Copy link
Contributor

haydentherapper commented Mar 20, 2024

@laurentsimon @ianlewis You just need to bump Cosign to at least v2.2.0, latest is v2.2.3.

I'd also recommend checking your renovate settings, I would have expected this bump to get picked up automatically.

haydentherapper added a commit to haydentherapper/slsa-github-generator that referenced this issue Mar 20, 2024
Versions of Cosign before v2.2.0 are not compatible with the latest TUF
root.

Fixes slsa-framework#3350

Signed-off-by: Hayden Blauzvern <hblauzvern@google.com>
@behnazh-w
Copy link
Contributor

@haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

@suzuki-shunsuke
Copy link
Contributor

suzuki-shunsuke commented Mar 20, 2024

haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

According to the announcement, v2.2.0 should support a new TUF Trust root, so I don't think we need to update Cosign in SLSA Verifier.

@suzuki-shunsuke
Copy link
Contributor

suzuki-shunsuke commented Mar 20, 2024

I see. The latest slsa-github-generato v1.9.0 uses the old slsa-verifier which uses the old cosign v2.0.2

VERIFIER_RELEASE: v2.3.0 # The version of the verifier to download.

https://github.com/slsa-framework/slsa-verifier/blob/c9abffe4d2ab2ffa0b2ea9b2582b84164f390adc/go.mod#L21

The main branch uses the latest slsa-verifier v2.4.1.

VERIFIER_RELEASE: v2.4.1 # The version of the verifier to download.

So we need to release a new version of slsa-github-generator, then the issue would be solved.

@suzuki-shunsuke
Copy link
Contributor

suzuki-shunsuke commented Mar 20, 2024

@laurentsimon @ianlewis
I'm sorry to bother you, but could you release a new version of slsa-github-generator to solve this issue?
#3350 (comment)

@laurentsimon
Copy link
Collaborator

Hi, yes let's do that. I think we need to revert a few breaking PRs we made recently in the slsa-github-generator. slsa-verifier should be ready to be released. @ramonpetgrave64 could you send PRs to temporarily revert the few breaking PRs we made recently?

@laurentsimon
Copy link
Collaborator

Note: slsa-verifier's latest version uses cosign 2.2.0 https://github.com/slsa-framework/slsa-verifier/blob/v2.4.1/go.mod so need not be updated.

laurentsimon pushed a commit that referenced this issue Mar 20, 2024
Versions of Cosign before v2.2.0 are not compatible with the latest TUF
root.

Fixes
#3350

# Summary

...

## Testing Process

...

## Checklist

- [ ] Review the contributing [guidelines](./../CONTRIBUTING.md)
- [ ] Add a reference to related issues in the PR description.
- [ ] Update documentation if applicable.
- [ ] Add unit tests if applicable.
- [ ] Add changes to the [CHANGELOG](./../CHANGELOG.md) if applicable.

---------

Signed-off-by: Hayden Blauzvern <hblauzvern@google.com>
Signed-off-by: Bob Callaway <bcallaway@google.com>
Co-authored-by: Bob Callaway <bcallaway@google.com>
laurentsimon added a commit that referenced this issue Mar 20, 2024
# Summary

Update changelog for #3350

## Testing Process

...

## Checklist

- [ ] Review the contributing [guidelines](./../CONTRIBUTING.md)
- [ ] Add a reference to related issues in the PR description.
- [ ] Update documentation if applicable.
- [ ] Add unit tests if applicable.
- [ ] Add changes to the [CHANGELOG](./../CHANGELOG.md) if applicable.

---------

Signed-off-by: laurentsimon <laurentsimon@google.com>
@sgammon
Copy link
Contributor

sgammon commented Mar 20, 2024

I see this is closed, but v1.9.0 is marked as the latest release... in the release notes update, it says:

error updating to TUF remote mirror: invalid
This will occur only when generating provenance with all builders and generators.
Affected versions: all versions up and including v1.9.0

Can this bug be reopened until a release occurs? Or is there a way to work around this, or can we use the generator at main?

cc / @laurentsimon @ianlewis

@laurentsimon
Copy link
Collaborator

Generator at main won't pass verification. We need to cut the release. @kpk47 is working on it

@sgammon
Copy link
Contributor

sgammon commented Mar 20, 2024

Thank you @laurentsimon for being so responsive on this issue; I understand the commit auto-closed the bug. It's helpful to have something to follow. Good luck and thank you for your hard work securing software supply chains.

@jkreileder
Copy link
Contributor Author

Works again for me with the un-finalized v1.10.0 pre-release.

@laurentsimon
Copy link
Collaborator

Release is available v1.10.0 https://github.com/slsa-framework/slsa-github-generator/releases/tag/v1.10.0

stephenfuqua added a commit to Ed-Fi-Alliance-OSS/slsa-github-generator that referenced this issue Mar 21, 2024
Versions of Cosign before v2.2.0 are not compatible with the latest TUF
root.

Fixes
slsa-framework/slsa-github-generator#3350

...

...

- [ ] Review the contributing [guidelines](./../CONTRIBUTING.md)
- [ ] Add a reference to related issues in the PR description.
- [ ] Update documentation if applicable.
- [ ] Add unit tests if applicable.
- [ ] Add changes to the [CHANGELOG](./../CHANGELOG.md) if applicable.

---------

Signed-off-by: Hayden Blauzvern <hblauzvern@google.com>
Signed-off-by: Bob Callaway <bcallaway@google.com>
Co-authored-by: Bob Callaway <bcallaway@google.com>
@ddworken
Copy link
Contributor

I upgraded to v1.10.0 and that fixed generation of new provenance files, but I'm wondering: Is there a way to make old versions of slsa-verifier still work? From what I can tell, this change appears to make all old provenances unverifiable which seems like a significant problem.

@behnazh-w
Copy link
Contributor

behnazh-w commented Mar 24, 2024

@ddworken The latest version of slsa-verifier is still is able to verify old provenances. Why do you need to use an old version of slsa-verifier?

@haydentherapper @bobcallaway Probably unrelated to this issue but what is the Sigstore/Cosign solution for revoked root keys and in general how does Sigstore provide backward compatibility with old versions of certificates, and therefore verifying existing legitimate provenances that were signed with revoked keys?

@ddworken
Copy link
Contributor

ddworken commented Mar 24, 2024

The issue is that the old version of slsa-verifier isn't able to verify releases that were formerly valid. My project uses SLSA for releases and secure updates. I have daily tests running on Github Actions and it went from passing to failing in line with this bug:

Screenshot 2024-03-24 at 4 03 42 PM

You can see the test failure here. I also confirmed that this appears to apply to the main slsa-verifier binary, so this can easily be reproduced:

# From inside of the slsa-verifier git repo
# Check out the old version of slsa-verifier
git checkout v1.3.2
# Download a file and an old attestation
wget https://github.com/ddworken/hishtory/releases/download/v0.277/hishtory-linux-amd64
wget https://github.com/ddworken/hishtory/releases/download/v0.277/hishtory-linux-amd64.intoto.jsonl
# Run slsa-verifier
go run ./cli/slsa-verifier -artifact-path hishtory-linux-amd64 -provenance hishtory-linux-amd64.intoto.jsonl -source github.com/ddworken/hishtory

This fails with the output:

slsa-verifier output
Getting rekor entry error error verifying tlog entry: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://sigstore-tuf-root.storage.googleapis.com",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 170,
			"len": 721,
			"expiration": "29 Mar 24 16:08 UTC",
			"error": ""
		}
	}
}: unable to fetch Rekor public keys from TUF repository, trying Redis search index to find entries by subject digest
FAILED: SLSA verification failed: could not find a matching valid signature entry: got unexpected errors updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://sigstore-tuf-root.storage.googleapis.com",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 170,
			"len": 721,
			"expiration": "29 Mar 24 16:08 UTC",
			"error": ""
		}
	}
}: unable to fetch Rekor public keys from TUF repository: verifying tlog entry 24296fb24b8ad77a614df024f5e997739525746b275409b28df6a6b7285c7128f4f8e429a5cb1e42, updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://sigstore-tuf-root.storage.googleapis.com",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 170,
			"len": 721,
			"expiration": "29 Mar 24 16:08 UTC",
			"error": ""
		}
	}
}: unable to fetch Rekor public keys from TUF repository: verifying tlog entry 24296fb24b8ad77af7d9a9a7dbb2e7c25c5d46e41770969259f3d72fd20f0016bd5b1a54229bcbba
exit status 2

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

Why do you need to use an old version of slsa-verifier?

This is because my project embeds slsa-verifier in order to provide secure updates. So when slsa-verifier suddenly breaks in this way, it means that any clients with the old version of slsa-verifier become effectively stranded since they cannot upgrade to new versions.

@haydentherapper
Copy link
Contributor

haydentherapper commented Mar 24, 2024

Probably unrelated to this issue but what is the Sigstore/Cosign solution for revoked root keys and in general how does Sigstore provide backward compatibility with old versions of certificates, and therefore verifying existing legitimate provenances that were signed with revoked keys?

@behnazh-w Supporting validity windows was the motivation behind the design of the trust root spec. For each root of trust, a validity time period is specified. If a compromise were to occur, rather than revoking the root material entirely and assuming we can determine the window when compromise occurred, the root material would simply be specified as valid only up to the compromise.

All Sigstore clients (eg sigstore-python, js, etc) but Cosign support ingesting the roots of trust in this format currently. Cosign will be updated to read it soon.

@laurentsimon
Copy link
Collaborator

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

You're correct. The TUF root updates need an update of slsa-verifier to v2.4.1. We could backport the fixes, but it would still require you to update your slsa-verifier version.

@ddworken
Copy link
Contributor

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

You're correct. The TUF root updates need an update of slsa-verifier to v2.4.1. We could backport the fixes, but it would still require you to update your slsa-verifier version.

Got it, thanks for confirming! I'm wondering: Is this class of breakage something you expect to continue to need to happen with slsa-verifier? If so, it seems like this may be an important caveat to document (maybe under the "Known Isssues" heading) since having a previously working verifier continue to work seems like a reasonable expectation.

@laurentsimon
Copy link
Collaborator

We don't expect this to happen. @haydentherapper can comment on the specifics of cosign.

@haydentherapper
Copy link
Contributor

Breaking changes are not something that is going to happen frequently and if they do, there will be a significant period of time to have clients migrate over.

V0ldek added a commit to rsonquery/rsonpath that referenced this issue Mar 28, 2024
SLSA breaking change requires an update of the action to 1.10:
slsa-framework/slsa-github-generator#3350
rsoberano-ld added a commit to launchdarkly/cpp-sdks that referenced this issue Apr 2, 2024
…trieving Rekor public keys (#386)

Provenance generation fails due to Rekor public key errors were
identified as a known issue and fixed in version 1.10.0 per:
slsa-framework/slsa-github-generator#3350
rsoberano-ld added a commit to launchdarkly/ruby-server-sdk that referenced this issue Apr 4, 2024
…trieving Rekor public keys (#265)

**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [ ] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions

**Describe the solution you've provided**

Provenance generation fails due to Rekor public key errors were
identified as a known issue and fixed in version 1.10.0 per:
slsa-framework/slsa-github-generator#3350
rsoberano-ld added a commit to launchdarkly/python-server-sdk that referenced this issue Apr 4, 2024
…trieving Rekor public keys (#280)

**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [ ] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions


**Describe the solution you've provided**

Provenance generation fails due to Rekor public key errors were
identified as a known issue and fixed in version 1.10.0 per:
slsa-framework/slsa-github-generator#3350
rsoberano-ld added a commit to launchdarkly/haskell-server-sdk that referenced this issue Apr 4, 2024
…trieving Rekor public keys (#82)

**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [ ] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions

**Describe the solution you've provided**

Provenance generation fails due to Rekor public key errors were
identified as a known issue and fixed in version 1.10.0 per:
slsa-framework/slsa-github-generator#3350
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status:triage Issue that has not been triaged type:bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants