Istio Ambient support

[peterj]

Istio Ambient mode is a different deployment model from the “traditional” (sidecar) mode of Istio. The ambient mode (sidecar-less) doesn’t require injecting sidecars into the deployments.

Here are the high-level differences between the two modes:

• The concerns handled by the sidecar proxy in the sidecar Istio are split into two components in Istio ambient: 

  • Ztunnel (handles L4 concerns, mTLS, authorization policies without any HTTP)

    • Ztunnel is installed automatically when profile=ambient

  • Waypoint proxy (handles L7 concerns, i.e. traffic splitting, matching, header manipulation, etc., more or less everything that gets defined in the VirtualService)

    • Waypoint is optional (not installed by default) and it can be deployed per service account (handles all workloads using the same service account) or  per namespace (handles L7 proxying for all workloads in the namespace)  

• Ingress gateway isn’t installed by default anymore when using profile=ambient

  • It might be worth migrating over to Kubernetes Gateway APIs and deploying the Istio ingress gateway like that, as we’ll have to use the Gateway APIs to deploy waypoint proxies anyway

• Kubernetes Gateway API is used for ingress (and waypoint proxy) deployments

• Any L7 VirtualServices or AuthorizationPolicies must have a “targetRef” section that specifies which waypoint proxy handles the L7 configuration

Waypoint proxies

Any VirtualService or AuthorizationPolicy that uses HTTP concepts will require a waypoint proxy. Given that there are 3 namespaces (that I identified so far), I’d suggest a per-namespace deployment of a waypoint proxy.

In addition to the waypoint proxy, the resources will have to be updated to use the waypoint proxies: 

Component	Namespace	Notes
dex	auth
central-dashboard	kubeflow
jupyter-web-app	kubeflow
volumes-web-app	kubeflow
katib-ui	kubeflow
ml-pipeline-ui	kubeflow
metadata-envoy-service	kubeflow
kfp-tekton	kubeflow
kubebench-dashboard	kubeflow
profiles-kfam	kubeflow
tensorboards-web-app-service	kubeflow
kserve-models-web-app	kserve

Work items

• Migrate to Kubernetes Gateway API

  • Update the YAML (/common/istio*) for deploying ingress and local ingress to use the Gateway API 

• Move to the latest Istio (Ambient will be beta in the next release (1.22))

  • We can still continue with the sidecar mode here

• Move to Ambient mode

  • Identify components that need waypoint proxy

  • Deploy waypoint proxies 

  • Switch the Istio profile from default → ambient (or have an option of doing one or the other) - since Ambient will still be Beta, we shouldn’t make it a default option

[ca-scribner]

What are the components that need to change for this? Would the profile controller need to create waypoint proxies for each user's namespace?

[peterj]

Waypoint proxies are automatically created when the Gateway resource gets created (so is a bit simpler than crafting deployments/services). You can configure it in such a way that there's 1 instance per namespace that handles all L7 traffic for that namespace.