Use a single cache for all dynamic controllers (i.e. XRs and claims)

[negz]

Description of your changes

Fixes #5338
Fixes #2645
Closes #5463
Closes #5468
Closes #5422

It's possible this will fix #5400, #5151, #5533, and #5228 given that it makes large changes to how realtime compositions work. I'd rather not assume though - we can see if folks can reproduce if/when this PR is merged.

Crossplane uses a "controller engine" to dynamically start claim and XR controllers when a new XRD is installed, and stop them when the XRD is uninstalled.

Before this PR, each controller gets at least one cache. This is because when I built the controller engine, you couldn't stop a single informer within a cache (a cache is basically a map of informers by GVK). Instead, we stop entire caches.

When realtime composition is enabled, there are even more caches. One per composed resource GVK. A GVK routed cache routes cache lookups to these various delegate caches.

Meanwhile, controller-runtime recently made it possible to stop an informer within a cache. It's also been possible to remove an event handler from an informer for some time (since Kubernetes 1.26):

• 🌱 Proposal for dynamic informer cache kubernetes-sigs/controller-runtime#2285
• ⚠ Support registration and removal of event handler kubernetes-sigs/controller-runtime#2046

This PR uses a single client, backed by a single cache, across all dynamic controllers (specifically the definition, offered, claim, and XR controllers). Each controller tracks what watches ("sources" in controller-runtime) it has active and stops them when it doesn't need them anymore. The controller engine also garbage collects custom resource informers when their CRDs are deleted.

Compared to the current implementation, this:

• Takes fewer global locks when realtime compositions are enabled. Locking is now mostly at the controller scope.
• Works with the breaking changes to source.Source introduced in controller-runtime v0.18. :)

I also think this makes the realtime composition code a little easier to follow by consolodating it into the ControllerEngine (and into one repo), but that's pretty subjective.

Here's a questionable Mermaid diagram of how it works:

graph TD
	subgraph ITC[engine.InformerTrackingCache]
		I1[XFoo Informer]
		I2[XBar Informer]
		I3[ComposedResource Informer]
	end
	
	subgraph CE[engine.ControllerEngine]
		subgraph C1[XFoo Controller]
			R1[XFoo Reconciler] -- Get --> I1
			R1 -- Get --> I3
			S1[XFoo Source] -- Watch --> I1
			S1 -- Enqueue --> R1
			SC1[ComposedResource Source] -- Watch --> I3
			SC1 -- Enqueue --> R1
		end

		subgraph C2[XBar Controller]
			R2[XBar Reconciler] -- Get --> I2
			R2 -- Get --> I2
			S2[XFoo Source] -- Watch --> I2
			S2 -- Enqueue --> R2
			SC2[ComposedResource Source] -- Watch --> I3
			SC2 -- Enqueue --> R2
		end
	end

	subgraph C[cache.Cache]
	  I4[XRD Informer]
	end

	subgraph M[manager.Manager]
		subgraph C3[XRD Controller]
			R3[XRD Reconciler] -- Start --> C1
			R3 -- Start --> C2
			R3 -- Get --> I4
			S3[XRD Source] -- Watch --> I4
			S3 -- Enqueue --> R3
		end
	end

It's conceptually pretty similar to a typical set of controller-runtime controllers run by a controller.Manager, but with the ability to stop watches and informers.

Reviewer Note

This PR depends on crossplane/crossplane-runtime#689.

Unfortunately, this required bumping a bunch of dependencies, mostly to get the latest controller-runtime. Those dependency bumps in turn required a bunch of other little fixes (CRD generation issues, new linter warnings, etc).

I also opted to move the ControllerEngine implementation from c/cr to c/c. I think that will make changes like this easier in future.

While this all makes this PR pretty huge, it's all broken out into in separate commits. The last commit in the PR contains the bulk of the changes - reviewers can focus on that.

I'm opening this as draft since it's a big change. I still need to add more tests for the new code (e.g. the new engine implementation).

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added or updated unit tests.
• Added or updated e2e tests.
• Linked a PR or a docs tracking issue to document this change.
• Added backport release-x.y labels to auto-backport this PR.

Need help with this checklist? See the cheat sheet.

[negz]

CC @sttts - this makes big changes to the realtime compositions implementation.

[negz]

The changes to this file are all mechanical - just changing how we pass a client.

Previously we passed a whole manager just to get its client. By passing a client specifically we can pass in a special client that is backed by our cache.

[negz]

I think we should do this in a future PR since this one is already very large.

[negz]

Can't merge until this is gone. I like that there's a linter for this now.

[negz]

Looks like the E2E tests are passing pretty reliably now. The only thing blocking moving this out of draft is writing some more unit tests.

[turkenh]

I also opted to move the ControllerEngine implementation from c/cr to c/c. I think that will make changes like this easier in future.

I am working on implementing watches in provider-kubernetes, and the PR is based on the previous real-time composition implementation, which is being refactored here. While the provider-kubernetes use case is more straightforward (no controller start/stop at runtime), I believe the new structure is more usable with the unified/shared cache then the previous implementation. When I check the functionality introduced here, I have a feeling like I could leverage things like TrackingInformers/InformerTrackingCache and StoppableSource, and maybe even Controller Engine with a single controller (TBD) 🤔 So, I would need them to be importable, at least not under internal if not back to crossplane-runtime 🙏

[negz]

So, I would need them to be importable

@turkenh How would you feel about duplicating them to start with? We could then consolidate later.

I'm hopeful we can get the tracking cache and stoppable source functionality added upstream to controller-runtime. If that happens I could imagine a world where:

• ControllerEngine is only used by c/c, lives in c/c (internal)
• Provider-kubernetes also uses tracking cache and stoppable sources, which live in controller-runtime