Add scaffolding to stand up Trillian on k8s. #2754

vaikas · 2022-06-07T11:45:34Z

Signed-off-by: Ville Aikas vaikas@chainguard.dev

Here's the pieces for scaffolding and for createtree. I split them into two different directories because then applying just the scaffolding bits is easier and more well contained. Apply all the everythings :)

Createtree is a standalone job that makes creating trees in the k8s cluster easier and then because the entry is stuffed into configmap, it's easy to 'depend' on that for other jobs that may need it. For example, we have also a setup for spinning up a CTLog on top of this in sigstore, but wasn't sure if that might be of interest. I noticed that in README.md one of the steps after spinning up Trillian is to install CTLog on top of it.

https://github.com/sigstore/scaffolding/tree/main/config/ctlog

After creating the README-SCAFFOLDING.md in the examples dir, I noticed the PR template sez to add it to docs/ but I'm not sure if this belongs there since it's related to running tests, etc.

Lastly, the added kind test in github workflows only checks the configmap that the treeID was created, might be good to add some examples to test the grpc api as well, but figured that could be a followup if that makes sense.

Checklist

I have updated the CHANGELOG.
I have updated documentation accordingly (including the feature implementation matrix).

roger2hk · 2022-06-07T12:59:07Z

/gcbrun

vaikas · 2022-06-07T13:05:45Z

@roger2hk Thanks for the gcbrun.

trillian-pr-tests are failing but I can't see why, can somebody share what the problem is so I can try to fix it :)
https://github.com/google/trillian/pull/2754/checks?check_run_id=6774522500

pav-kv · 2022-06-07T18:16:31Z

running golangci-lint
level=warning msg="[runner] Can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\""
level=error msg="Running error: 1 error occurred:\n\t* can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\"\n\n"

go.mod

pav-kv

Oups, I did not intend to approve this PR just yet, still needs review. Please ignore it.

mhutchinson

I've left a few nits from a first glance. Do you want to rebase this onto HEAD, and meanwhile this week I'll try setting up an environment so I can follow the instructions here to really make sure it makes sense to me :-)

(apologies for late response, life has been busy the last 3 weeks!)

cmd/createtree-k8s/main.go

vaikas · 2022-06-27T14:12:15Z

Thanks for the comments! Sorry for tardy response, was out on PTO, will pick this back up this week! @mhutchinson totes understood about life sometimes getting very busy :)

AlCutter · 2022-06-28T11:18:26Z

/gcbrun

examples/deployment/kubernetes/README-SCAFFOLDING.md

mhutchinson · 2022-06-29T14:46:25Z

examples/deployment/kubernetes/README-SCAFFOLDING.md

+```
+3. Then create a tree in the Trillian:
+```shell
+ko apply -BRf ./examples/deployment/kubernetes/createtree


Feel free to add an issue (assigned to me) and TODO here to swap this over to use a pre-built image so that ko isn't needed.

mhutchinson · 2022-06-29T15:24:09Z

/gcbrun

mhutchinson · 2022-06-29T15:24:40Z

/gcbrun

mhutchinson

LGTM, modulo a few comments.

mhutchinson · 2022-06-29T15:26:48Z

.github/workflows/kind-scaffolding-e2e-createtree.yml

+    - name: Setup mirror
+      uses: chainguard-dev/actions/setup-mirror@main
+      with:
+        mirror: mirror.gcr.io


What's the purpose of this mirror?

In some cases esp. in test environments this can reduce flakiness by not hitting rate limits on pulling containers. So by default we set this up before having to add it later :)

.github/workflows/kind-scaffolding-e2e-createtree.yml

examples/deployment/kubernetes/scaffolding/log-server/placeholder.go

mhutchinson · 2022-07-08T08:38:02Z

/gcbrun

mhutchinson · 2022-07-11T10:22:20Z

This is failing running the lint checks:

Already have image (with digest): gcr.io/trillian-opensource-ci/trillian_testbase
running gofmt
running goimports
running golangci-lint
level=warning msg="[runner] Can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\""
level=error msg="Running error: 1 error occurred:\n\t* can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\"\n\n"

mhutchinson · 2022-07-11T10:24:09Z

golangci/golangci-lint#1920 may be relevant. Holler if you can't work this out and I'll take a deeper look.

mhutchinson · 2022-08-03T09:47:46Z

/gcbrun

mhutchinson · 2022-08-03T10:05:05Z

This is still causing some problem with the linter. Fortunately it can be reproduced locally:

golangci-lint version
golangci-lint has version v1.47.3 built from (unknown, mod sum: "h1:ri7A2DgtFpxgqcMSsU3qIT0IBm/SCdYgXlvmJx4szUU=") on (unknown)

golangci-lint run
WARN [runner] Can't run linter goanalysis_metalinter: buildir: failed to load package v1: could not load export data: no export data for "k8s.io/client-go/applyconfigurations/meta/v1"
ERRO Running error: 1 error occurred:
	* can't run linter goanalysis_metalinter: buildir: failed to load package v1: could not load export data: no export data for "k8s.io/client-go/applyconfigurations/meta/v1"

mhutchinson · 2022-08-03T10:07:29Z

I think this is because client-go is min version go1.18: https://github.com/kubernetes/client-go/blob/master/go.mod

golangci/golangci-lint#1920 (comment)

edit: but then again we're using v0.23.8, which is go1.16: https://github.com/kubernetes/client-go/blob/v0.23.8/go.mod

mhutchinson · 2022-08-03T10:36:46Z

Hah, I've just spent ages messing about with looking into the linters and go versions. The answer appears to be as simple as running go build to find the root cause:

go build ./...
# k8s.io/client-go/applyconfigurations/meta/v1
../../go/pkg/mod/k8s.io/client-go@v0.23.8/applyconfigurations/meta/v1/unstructured.go:64:38: cannot use doc (variable of type *"github.com/googleapis/gnostic/openapiv2".Document) as type *"github.com/google/gnostic/openapiv2".Document in argument to proto.NewOpenAPIData

mhutchinson · 2022-08-03T10:47:03Z

Root cause is probably kubernetes/client-go#1084. @vaikas can I leave this with you?

vaikas · 2022-08-03T16:39:45Z

Thank you so much @mhutchinson for digging into this. I'll keep looking at it, oh what joy! 🤣

vaikas · 2022-08-03T19:21:54Z

Ok, added a replace to go.mod and now things are building:

vaikas@villes-mbp trillian % go mod tidy -go=1.17 -compat=1.17
vaikas@villes-mbp trillian % git status
On branch k8s-createtree
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   go.mod
	modified:   go.sum
no changes added to commit (use "git add" and/or "git commit -a")
vaikas@villes-mbp trillian % go build ./...
vaikas@villes-mbp trillian %

Let's try this now :)

mhutchinson · 2022-08-04T10:15:59Z

/gcbrun

mhutchinson · 2022-08-04T10:31:37Z

Error is

Already have image (with digest): gcr.io/trillian-opensource-ci/trillian_testbase
running gofmt
running goimports
running golangci-lint
level=warning msg="[runner] Can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\""
level=error msg="Running error: 1 error occurred:\n\t* can't run linter goanalysis_metalinter: buildir: failed to load package version: could not load export data: no export data for \"sigs.k8s.io/release-utils/version\"\n\n"

Looks like a similar issue to the last one but with a different module. Oddly though, I can't reproduce this one locally with go build or golangci-lint. Will have a quick poke at this, but due to RL considerations I may need to leave this for @AlCutter to take over next week.

JAORMX · 2022-12-13T06:30:10Z

@vaikas if you want to revisit this PR, the cloudbuild results are now viewable so it would be easier to debug.

vaikas · 2022-12-15T23:18:05Z

oooooh, yes, thank you for the pointer, I'll try to carve some time in the next couple of days :)

…

On Mon, Dec 12, 2022 at 10:30 PM Juan Antonio Osorio < ***@***.***> wrote: @vaikas <https://github.com/vaikas> if you want to revisit this PR, the cloudbuild results are now viewable so it would be easier to debug. — Reply to this email directly, view it on GitHub <#2754 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWB45GGBHAOLK7P5B3TB6DWNAJX3ANCNFSM5YCUENJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

mhutchinson · 2023-01-09T12:04:05Z

I'm back! Getting this PR in would be a great accomplishment for my return if you're up for it @vaikas :-)

vaikas · 2023-01-09T14:56:54Z

Welcome back, happy new year!!!! Yes, I'll make time for it, this time I really mean it.

…

On Mon, Jan 9, 2023 at 2:04 PM Martin Hutchinson ***@***.***> wrote: I'm back! Getting this PR in would be a great accomplishment for my return if you're up for it @vaikas <https://github.com/vaikas> :-) — Reply to this email directly, view it on GitHub <#2754 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWB45CMHJYWF46YALXJZCTWRP5EBANCNFSM5YCUENJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

vaikas · 2023-02-08T23:30:50Z

@mhutchinson would you mind hitting the magic test button (/gcbrun) for me please? :)

.github/workflows/kind-scaffolding-e2e-createtree.yml

roger2hk · 2023-02-08T23:48:14Z

/gcbrun

JAORMX · 2023-02-09T16:25:37Z

.github/workflows/kind-scaffolding-e2e-createtree.yml

+      matrix:
+        k8s-version:
+        - v1.22.x
+        - v1.23.x


Any reason not to add newer kube versions?

LOL, oh dear :) Yes, done.

roger2hk · 2023-02-09T16:29:56Z

/gcbrun

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

roger2hk · 2023-02-21T20:47:40Z

/gcbrun

vaikas requested a review from a team as a code owner June 7, 2022 11:45

vaikas requested a review from jiggoha June 7, 2022 11:45

pav-kv approved these changes Jun 7, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

pav-kv suggested changes Jun 7, 2022

View reviewed changes

mhutchinson self-assigned this Jun 22, 2022

mhutchinson self-requested a review June 22, 2022 14:01

mhutchinson reviewed Jun 22, 2022

View reviewed changes

cmd/createtree-k8s/main.go Outdated Show resolved Hide resolved

cmd/createtree-k8s/main.go Outdated Show resolved Hide resolved

vaikas force-pushed the k8s-createtree branch from ce634a1 to afed0e0 Compare June 28, 2022 08:42

vaikas mentioned this pull request Jun 28, 2022

Update go mod to 1.17 with: go mod tidy -go=1.17 #2768

Merged

2 tasks

mhutchinson reviewed Jun 28, 2022

View reviewed changes

examples/deployment/kubernetes/README-SCAFFOLDING.md Show resolved Hide resolved

vaikas force-pushed the k8s-createtree branch from 33cc6a2 to 89b3d79 Compare June 29, 2022 09:05

mhutchinson reviewed Jun 29, 2022

View reviewed changes

mhutchinson closed this Jun 29, 2022

mhutchinson reopened this Jun 29, 2022

mhutchinson reviewed Jun 29, 2022

View reviewed changes

vaikas force-pushed the k8s-createtree branch from d2211fe to 07b5e0c Compare July 7, 2022 17:45

vaikas mentioned this pull request Jul 7, 2022

Build and use image for ./examples/deployment/kubernetes/createtree #2774

Open

mhutchinson approved these changes Jul 8, 2022

View reviewed changes

vaikas force-pushed the k8s-createtree branch from 07b5e0c to 78d0621 Compare July 27, 2022 18:15

vaikas changed the title ~~WIP: Add scaffolding to stand up Trillian on k8s.~~ Add scaffolding to stand up Trillian on k8s. Jul 30, 2022

vaikas mentioned this pull request Jul 31, 2022

Make the release artifacts more granular. sigstore/scaffolding#252

Closed

v-porytskyi approved these changes Dec 12, 2022

View reviewed changes

vaikas force-pushed the k8s-createtree branch from b269b37 to c8225b6 Compare February 7, 2023 21:41

roger2hk reviewed Feb 8, 2023

View reviewed changes

.github/workflows/kind-scaffolding-e2e-createtree.yml Outdated Show resolved Hide resolved

.github/workflows/kind-scaffolding-e2e-createtree.yml Outdated Show resolved Hide resolved

JAORMX reviewed Feb 9, 2023

View reviewed changes

JAORMX approved these changes Feb 9, 2023

View reviewed changes

vaikas added 5 commits February 21, 2023 12:38

WIP: Add scaffolding to stand up Trillian on k8s.

9bafa5b

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

do not need override anymore.

03ecc4f

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

use build-ko/setup-ko, go 1.19, add package comment.

b7d95be

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

test with k8s 1.24, 1.25

d7c4370

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

rebase

5177416

Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

vaikas force-pushed the k8s-createtree branch from 7ff1402 to 5177416 Compare February 21, 2023 20:39

Add scaffolding to stand up Trillian on k8s. #2754

Are you sure you want to change the base?

Add scaffolding to stand up Trillian on k8s. #2754

Conversation

vaikas commented Jun 7, 2022

Checklist

roger2hk commented Jun 7, 2022

vaikas commented Jun 7, 2022

pav-kv commented Jun 7, 2022

pav-kv left a comment

Choose a reason for hiding this comment

mhutchinson left a comment

Choose a reason for hiding this comment

vaikas commented Jun 27, 2022

AlCutter commented Jun 28, 2022

mhutchinson Jun 29, 2022

Choose a reason for hiding this comment

mhutchinson commented Jun 29, 2022

mhutchinson commented Jun 29, 2022

mhutchinson left a comment

Choose a reason for hiding this comment

mhutchinson Jun 29, 2022

Choose a reason for hiding this comment

vaikas Jul 7, 2022

Choose a reason for hiding this comment

mhutchinson commented Jul 8, 2022

mhutchinson commented Jul 11, 2022

mhutchinson commented Jul 11, 2022

mhutchinson commented Aug 3, 2022

mhutchinson commented Aug 3, 2022

mhutchinson commented Aug 3, 2022 • edited

mhutchinson commented Aug 3, 2022

mhutchinson commented Aug 3, 2022

vaikas commented Aug 3, 2022

vaikas commented Aug 3, 2022

mhutchinson commented Aug 4, 2022

mhutchinson commented Aug 4, 2022

JAORMX commented Dec 13, 2022

vaikas commented Dec 15, 2022 via email

mhutchinson commented Jan 9, 2023

vaikas commented Jan 9, 2023 via email

vaikas commented Feb 8, 2023

roger2hk commented Feb 8, 2023

JAORMX Feb 9, 2023

Choose a reason for hiding this comment

vaikas Feb 9, 2023

Choose a reason for hiding this comment

roger2hk commented Feb 9, 2023

roger2hk commented Feb 21, 2023

mhutchinson commented Aug 3, 2022 •

edited