FluxCD integration idea: allow TestTriggers to execute tests on CustomResource Conditions

[cedricmckinnie]

Describe the enhancement you'd like to see

To summarize, I would love to see the ability of TestTriggers to execute Tests based on Name, Selectors and Conditions of CustomResource objects. This would allow TestTriggers to run Tests for advanced operator configuration rollouts i.e. when using FluxCD.

I also want to resurface my previous comment on this as a new thread as I see this as more of an enhancement to TestTriggers rather than a new feature-request. I think this will provide a relatively quick and simple path to better FluxCD integration (and likely other advanced operators) without having to re-architect much if at all.

Additional context

FluxCD controllers wait for resources that are being synced to complete reconciliation whether that attempt succeeds or fails. The controllers then write to the Status Conditions of the respective HelmRelease or Kustomization object once the whole set of resources that is managed by that HR or KS has attempted to deploy rather than just a Pod, Deployment, DaemonSet etc. within it. This would be powerful because:

• It would allow for you to capture a whole group of resources without having to write granular triggers for each native k8s object.
• You can leverage existing Testkube functionality to implement it.
• It could open the door for testing other advance operators aside from FluxCD

Example usage would not require any change to the CRD or CLI inputs other than expanding the list of accepted resource values to include CustomResources i.e.

apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
  name: testtrigger-example
spec:
  resource: <HelmRelease|Kustomization> or perhaps any CR object
  resourceSelector:
    namespace: foo-ns
    name: my-helmrelease
  event: <created|modified|deleted>
  conditionSpec:
    timeout: 100
    conditions:
    - reason: ReconciliationSucceeded
      status: "True"
      type: Ready
    - reason: UpgradeSucceeded
      status: "True"
      type: Released
  action: run
  execution: test
  testSelector:
    name: test-to-run
    namespace: testkube

Many thanks for the consideration!

[vsukhin]

Thank you @cedricmckinnie Definitely, good idea!

[cedricmckinnie]

Just had an other interesting idea but I'm not sure how feasible it would be.

Do you think it would be a good idea to capture details about the objects that were created/modified/deleted and provide them as input data to the test execution runtime?

This would make the test execution aware of what object triggered it to run. At the very least the .metadata.name/namespace/labels/annotations and the triggering action could be quite useful as context data during test execution for a Test invoked by a TestTrigger.

[vsukhin]

It might be interesting.knowing context details can be useful in some cases

[gberche-orange]

@dejanzele

Other use-cases where CRD support for triggers would be useful beyond the original Fluxcd use-case:

• triggering tests on changes of CRs such as
  • A/B testing CR such as https://keptn.sh/ and its CRDs a https://lifecycle.keptn.sh/docs/yaml-crd-ref/
  • cloud service subscriptions, such as https://cloud.google.com/config-connector/docs/reference/overview
  • ...

I'd even say that ability to trigger tests on CRs is likely more useful than triggering on the current Resource - pod, deployment, statefulset, daemonset, service, ingress, event, configmap

As a workaround, I'll next try to trigger on side effects of CR controllers, such as events generated by flux, see some list at fluxcd/flux2#3628

[hrishin]

@vsukhin is possible to assign the ticket. More than happy to contribute here.

Can give a brief about general design, as we have started working on this issue for our use case. Maybe its worth to consider the use case like trigger on general CRDs as well (One suggested by @cedricmckinnie @gberche-orange).

[vsukhin]

Sure, feel free to contribute! It would be great

[gberche-orange]

Follow up on #3712 (comment)

As a workaround, I'll next try to trigger on side effects of CR controllers, such as events generated by flux, see some list at fluxcd/flux2#3628

Unfortunately, the event trigger is scoped on the associated resource (e.g Deployment) and thus can not yet be specified for the flux Kustomization (apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 kind: Kustomization)

[cedricmckinnie]

Looks like dynamic informers are on the way which I believe should make it easy to watch any kind of custom resources the same way watcher.go is currently handling native resources. It might even reduce the need to use each individual informer type.

https://github.com/kubernetes/kubernetes/blob/ec9a8ffb237e391ce9ccc58de93ba4ecc2fabf42/staging/src/k8s.io/client-go/dynamic/dynamicinformer/informer_test.go

Seems like there are a few blockers like this:

kubernetes/kubernetes#79610

[vsukhin]

good news, indeed @cedricmckinnie

[cedricmckinnie]

Any chance a regular watch.Interface could be used just for CRDs as an experimental feature until dynamic informers are fixed? Just want to confirm before I spend the time trying to implement.

kubernetes/kubernetes#79610 (comment)

[vsukhin]

We might, but I can't give you any ETA

[cedricmckinnie]

@vsukhin Submitted some changes for using dynamic informers to watch CR status conditions. I've tested it with HelmReleases to some extent in my cluster and it seems to work quite well.