bug(aws,gcp,k8s,azure): Test caching in CI does not work

[candiduslynx]

Which problem is this feature request solving?

Every time we update huge plugins (like AWS) we retest all pf the code instead of only testing the affected parts.

Describe the solution you'd like

Only testing the affected parts (need to make sure, though, that the libraries updates trigger the tests, too).
Possibly, this can be achieved by caching Go build cache & test cache (see this workflow as an example).

Relevant info:
https://pkg.go.dev/cmd/go#hdr-Build_and_test_caching
https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
https://github.com/marketplace/actions/cache

https://github.com/marketplace/actions/golang-with-cache
https://github.com/mvdan/github-actions-golang

Pull request (optional)

• I can submit a pull request

[candiduslynx]

I'll be introducing reusable workflows in #9518 to make the change easier.

[candiduslynx]

Desired outcome

Every plugin test workflow should utilize build & test cache.
This will enable us to significantly shorten the test time (for AWS it allowed running tests for 20s only instead of 10 minutes).

Proposed solution

Add build & test caching for every plugin build we have.
The cache should be tied to the plugin, Go version & OS.

Cache should be restored every time we do something Go-related (build, test, lint), and saved after testing from the default (main) branch only (as it's inherited by other branches as well).
Noteworthy, the build & test cache should be saved regardless of changes if we're running in the default (main) branch.

Additional steps

Use reusable workflows

We can utilize reusable workflows to extract the repeating parts from plugin workflows.
This allows us to:

1. Shorten the boilerplate code
2. Perform changes only in single place when we decide to make changes

This change is extracted into a separate #9655 issue.

Update runner logic

Currently we use large runners for releases. The reason for this is to prime the build cache. However, this cache is only used during releases and validate-release workflows.

I want to utilize large runners on the default (main) branch only, as I propose to save the cache only from the default (main) branch.

[erezrokah]

After spending quite a bit of time reviewing and working on this issue, we realized setup-go already caches build and test results. We had a bug in our Ci with plugins that have a test-policies job (AWS, GCP, Azure, K8S), that we used the same cache key for the build & test job and the test policies job. Since the test policies job is usually faster the cache from that job was the one that "took", essentially only caching build results and not test results.

The resolution is to use a separate cache key for those jobs, by disabling setup-go cache, and using the cache action directly.

We've also submitted a PR to add support for this use case in setup-go, see actions/setup-go#366

I updated the issue title to reflect the conclusion, and will discuss re-usable workflows separately