Cluster API v1.0 + v1beta1

[vincepri]

Hi everyone,

Let's chat about our roadmap to v1beta1 APIs and v1.0 🚀 . In the past few years we've gone through several iterations of our alpha APIs:

• v1alpha1 has been the start of everything: try this out and see if things work out.
• v1alpha2 was the foundational release, we've re-architected most of our APIs and controllers to function around contracts and object references. We made composition a primary concept and started building on top of it.
• v1alpha3 is the current stable release (at the time of writing), this release was driven by new features, like health checks or control plane lifecycle management.
• v1alpha4 is the current development version, focused primarily on stability and iterative improvements.

Our projects has been effectively operating like a GA project, most of our APIs have been supported for 1 year (or more in the case of v1alpha3), and we've started providing conversion webhooks between our types starting from v1alpha2.

While there is still lots to do, we've heard multiple feedback coming from different sources that new users don't necessarily trust the "alpha" in our API versioning, which makes total sense. If one is building a new production project, why trust an alpha API?

As we focus more on stability and keep our foundations in place, I'd like to put forward a plan to get to v1beta1 and release our 1.0 (!!) release by KubeCon US this year.

How would we get to v1beta1?

So far, we've always planned a roadmap based on the features we wanted to build. This time around, a possible alternative would be to instead focus on the lingering issues and technical debt. What's missing in the current APIs? How do we setup ourselves for success? What's currently missing in Cluster API to be a production ready environment?

What about new features? Let's pick a few good ones that are the most pressing and the most important.

Our current processes and compatibility promises don't need to change. Truthfully, we've been operating at a beta level (by Kubernetes definition, minor some bits) for quite a while. The only thing left for us is to make it official.

Thoughts, questions, concerns?

Let's open this discussion up and please bring your thoughts to the table. We'd like to hear from you if you're contributing today, just using or exploring Cluster API. Feedback is welcome!

[vincepri]

There is still lots to unpack here, I'll make sure to update the post as we make some progress during community meetings.

[jackfrancis]

Thank you @vincepri for everything you've done over the past few years to get us to this point!

[CecileRobertMichon]

recording of today's office hours where this was discussed: https://www.youtube.com/watch?v=WmCXrMRIsGY
meeting notes are here.

[detiber]

Our current processes and compatibility promises don't need to change. Truthfully, we've been operating at a beta level (by Kubernetes definition, minor some bits) for quite a while. The only thing left for us is to make it official.

The one place that we haven't is Rule #4a here: https://kubernetes.io/docs/reference/using-api/deprecation-policy/

Specifically:

Rule #4a: Other than the most recent API versions in each track, older API versions must be supported after their announced deprecation for a duration of no less than:

• GA: 12 months or 3 releases (whichever is longer)
• Beta: 9 months or 3 releases (whichever is longer)
• Alpha: 0 releases

We have been operating somewhere between Alpha and Beta here in that we support conversion from the n-1 release for the purposes of upgrades. If we are to adhere to the upstream Beta API policy, we would have to maintain conversion support for older versions much longer than we do today, and have to support more than just the upgrade use case, but also external components interacting with the older types.

If things were primarily changing within specific types (or moving common fields shared by child types to parent types), this wouldn't cause me too much hesitation, as we could generally recreate the data as needed. I'm not sure there is an answer when we are adding new abstractions or moving functionality from one abstraction to another (like the introduction of the Control Plane provider in v1alpha3, the proposed LoadBalancer provider, etc). Are we willing to say that we are done building new abstractions and/or going to require any new abstractions provide forward/backwards compatible conversions that would support the 3 release deprecation policy?

[detiber]

After having done another round of review of the upstream deprecation policy: https://kubernetes.io/docs/reference/using-api/deprecation-policy/

One of the things that stands out to me is that some parts of the deprecation policy are to support version skew between clients, users, external systems, and even internal k8s subsystems. It might be a good idea to fork it and make modifications that are relevant more specifically to our use cases.

Do we need to think about things like version skew between various components of CAPI?
Do we need to think about how external systems interact with CAPI (such as cluster-autoscaler)?
Do we need to think about what type of version skew we want to support for users interacting with CAPI?

Up until now we've lumped a lot of concerns around "stability" and "length of support" into one bucket "alpha". I think it is important that we don't just try to move the types and version numbers of binaries/releases to a more "stable" looking number, but as we do so more explicitly state what it means as we do so.

[MarcelMue]

Are we willing to say that we are done building new abstractions and/or going to require any new abstractions provide forward/backwards compatible conversions that would support the 3 release deprecation policy?

I would personally be happy if we could agree to this forward/backward compatible conversion policy. My main concern is the one you have already stated though:
Is it technically possible & realistic in cases where we move functionality from one abstraction to another?

[sbueringer]

I also think it's important to discuss how beta or GA (and the respective stability guarantees) are handled in various providers. While I think that most of the providers are currently keeping up with the main project, I'm not entirely sure all can / have enough maintainers to do so.

For example, in CAPO we're imho currently able to keep up with the main project. But I think it will be probably a completely different discussion, if we have to keep up with the main project and also have to provide stability guarantees for example of 3 releases (I guess we're talking about probably a year) including up and downwards compatibility.

I want to emphasize, that in my opinion, this should not block the main project and the providers, which are able to do so, from graduating to beta or GA and provide stability guarantees accordingly. I just think we should consider the variance we have between providers regarding production readiness and ability to maintain long term stability guarantees.

It's probably enough to have ClusterAPI-wide guidelines what alpha/beta/GA means and individual providers can graduate accordingly on their own pace.

[devigned]

With the outstanding work in Cluster Class, Load Balancer Provider, etc., do we still feel like we are going to hit a beta by the end of the year? Would we want to have a alpha5 release with Cluster Class to get some mileage prior to cutting beta1?

This might be an unpopular point of view, but we may have to draw a line for big feature work and be strict with our cutoff dates if we are going to meet our goal of getting to beta by the end of the year, or more agressively, by KubeCon US.

[sbueringer]

I think it would be nice to get a stable "core" to beta and beyond and be able to add new features independently on top of that which can go through the graduation process again. But yeah, the main controversial point is, when is the core stable enough and what is part of the core.

[CecileRobertMichon]

+1 to thinking about target dates for the next big releases so users, contributors, and providers can plan around it.

I think it would be nice to get a stable "core" to beta and beyond and be able to add new features independently on top of that which can go through the graduation process again

This ^ if we keep growing what is "core" it's a never-ending process. We could start graduating some of our APIs which have been treated as beta for a while (according to these definitions https://kubernetes.io/docs/reference/using-api/#api-versioning), without slowing down development of new features and APIs, like ClusterClass. Related discussion: #4928 (comment)

[alexeldeib]

Big +1 to #4351 (reply in thread), we have already started down this path with experimental features like MachinePool, AKS, etc (okay no others come to mind immediately), I think the natural evolution is maturing those into already-defined stability classes (alpha, beta), and tightening up core pieces.

[vincepri]

+1 to all that has been said.

Given the work on ClusterClass and our continuous focus on stability and user experience it's fair to say that end of year for beta is probably not doable everything considered.

I'll echo @CecileRobertMichon, we should meet in the next meetings and try to outline what are the requirements to get to beta, and what should be our target date. Beta APIs come also with a greater support matrix, although the pace of development might increase given that we can finally start using minor versions for breaking code changes (excluding APIs).

To list some of the current priorities:

• ClusterClass and (++) dynamic object patching
• Clusterctl and the Cluster API operator
• External etcd
• Kubelet secure authentication
• Load balancer provider
• MachinePool

And then there are some other priorities which should be further prioritized, although we haven't started looking at those items yet in details:

• OS Image information based on more than just Kubernetes versions; needs something generic across providers and still allow generic filterings for fields/properties that are cloud specific.
• Better failure domain handling and support.

What else did I miss from the above?

[CecileRobertMichon]

cross-referencing the mailing list thread: https://groups.google.com/g/kubernetes-sig-cluster-lifecycle/c/YBCBJzTO1o8?pli=1

Hi folks,

For the past few months, the Cluster API community has discussed the roadmap to v1beta1 and v1.0. The path thus far has been:

• v1alpha1 was the start of everything: try this out and see if things work out.
• v1alpha2 was the foundational release, we've re-architected most of our APIs and controllers to function around contracts and object references. We made composition a primary concept and started building on top of it.
• v1alpha3 is currently in maintenance mode. This release was driven by new features like health checks and control plane lifecycle management.
• v1alpha4 is the current stable version and is focused primarily on stability and iterative improvements.
  • New features are going to be included in v0.4.x in the future behind feature gates, like the ClusterClass work described in this proposal.
    Despite our API remaining in alpha state, the project is effectively operating with beta or higher guarantees. Most of our APIs have been supported for 1 year or longer, as in the case of v1alpha3. We have conversion webhooks between our types starting from v1alpha2, an extensive test matrix expanding with each new release as well as user and developer documentation.
    After the September 8th community meeting, we’d like to propose the following:
• Code and API freeze starting on October 1st, 2021
  • Bug fixes, improvements to the existing codebase, and existing feature gates functionalities are excluded from the freeze.
• To v1.0
  • Create the v1.0 Milestone.
    • Identify release-blocking issues and prioritize accordingly.
  • Create a release-0.4 branch to support v0.4 / v1alpha4.
  • Convert our current types to v1beta1 and provide conversion support from v1alpha3 and v1alpha4.
  • Cut the final release.
• Update our Cluster API Enhancement Proposal (CAEP) template and contributing guide to add a section around breaking API or behavior changes.
• Plan a roadmap and backlog grooming sessions for future minor releases, based on 1.x.
  • Amend the roadmap document with proposal freeze dates for each minor release.
  • Continue working on security improvements, ClusterClass, Runtime Extensions, MachinePool Machines, and all other proposals in 1.x releases.
  • Any proposals including breaking API changes are delayed until the community is ready to start v1beta2, or later.

As always, please feel free to reach out with any questions, comments, or concerns.

The Cluster API Maintainers.

[fabriziopandini]

We are on v1beta1 since some time now, but we forgot to close this discussion!