Evaluate Kubernetes 1.29 changes

[maltesander]

I would like to know if there are any changes in Kubernetes 1.29 that the SDP could potentially make use of (e.g. sidecars), will require changes or will break things on our end.

This should be timeboxed and take up at most 4h of research and reading.
The result of this should be a comment on this issue or follow-up issues listing the high level points of things we could, should and should not do.

Must read:

• https://kubernetes.io/blog/2023/11/16/kubernetes-1-29-upcoming-changes/

[maltesander]

I summarized the upcoming changes.
Except for the registry changes i do not think we have to do anything.

Registry k8s.gcr.io redirect to registry.k8s.io

See https://kubernetes.io/blog/2023/03/10/image-registry-redirect/ and https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/
E.g. images like dns/k8s-dns-node-cache, ingress-nginx/controller are affected.

=> This may affect images we use in demo / integration tests or CSI sidecars

Introducing Kubernetes Community-Owned Package Repositories: pkgs.k8s.io

https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/

=> This may affect images we use in demo / integration tests

Deprecations and removals for Kubernetes v1.29

Full list https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#deprecation

Prestop lifecycle

A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination (kubernetes/kubernetes#119026).

=> Since we use this now for graceful shutdown in several products could be used instead of manual sleep.

Cron

Creation of new CronJob objects containing TZ or CRON_TZ in .spec.schedule, accidentally enabled in 1.22, is now disallowed. Use the .spec.timeZone field instead, supported in 1.25+ clusters in default configurations.

Removal of in-tree integrations with cloud providers

The feature gates DisableCloudProviders and DisableKubeletCloudCredentialProviders will both be set to true by default for Kubernetes v1.29. This change will require that users who are currently using in-tree cloud provider integrations (Azure, GCE, or vSphere) enable external cloud controller managers, or opt in to the legacy integration by setting the associated feature gates to false.

=> Admin tasks

Removal of the v1beta2 flow control API group

flowcontrol.apiserver.k8s.io/v1beta2 will no longer be served (https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-29). v1beta3 has been promoted to stable.

=> Afaik not used by us

Deprecation of the status.nodeInfo.kubeProxyVersion field for Node

This field is not accurate and is set by kubelet, which does not actually know the kube-proxy version, or even if kube-proxy is running.

=> Not used by us

[adwk67]

Regarding the registries: the three that we use have been copied to Nexus/OCI:

• https://repo.stackable.tech/#browse/browse:docker:v2%2Fk8s%2Fsig-storage%2Fcsi-node-driver-registrar%2Ftags%2Fv2.5.0
• https://repo.stackable.tech/#browse/browse:docker:v2%2Fk8s%2Fsig-storage%2Fcsi-provisioner%2Ftags%2Fv3.1.0
• https://github.com/stackabletech/docker-images/blob/main/airflow/Dockerfile#L9
  I couldn't find any other usages with grep.

[siegfriedweber]

Another highlight:

• The SidecarContainers feature has graduated to beta and is enabled by default.
  ⇒ The logging side-car container can be simplified.

[lfrancke]

Thanks!
As we have a working implementation of the logging sidecar I'd suggest we wait until we don't support K8s versions without this feature anymore?
Or is there a benefit in supporting it earlier. For now we'd probably need feature/version detection making the code even more complicated.

[siegfriedweber]

As we have a working implementation of the logging sidecar I'd suggest we wait until we don't support K8s versions without this feature anymore?

Yes, let's wait. The current approach works well. It is only a little bit complicated and will be much easier with native sidecar containers.