Headless Service Per Replica KEP

[Edwinhr716]

What type of PR is this?

/kind feature

What this PR does / why we need it

KEP for #173 .

Based on original KEP #177. Added suggestions, and modified the way LWS_LEADER_ADDRESS is set.

Which issue(s) this PR fixes

Fixes #

Special notes for your reviewer

Does this PR introduce a user-facing change?

[liurupeng]

maybe we create a headless service per statefulset

[Edwinhr716]

So the leader will have one, while the workers will have a different one?

[liurupeng]

yes, ideally we only need the headless service to generate the dns record

[Edwinhr716]

What are the benefits of creating a new one per statefulset instead? Creating a new one per sts creates more headless services

[ahg-g]

The headless service name will be part of the fully qualified host name of the pod, so how many and what name we give the service matters. It is also why I am suggesting that the API to be a knob that decides on the subdomain policy because this is the user facing side effect.

The pod hostname is <pod-name>.<svc-name>

So if the new "policy" is UniquePerReplica, then each replica including the leader should be part of the same headless service so that the middle part above is common among the pods of a replica. Example:

Replica 0:

• my-lws-0.my-lws-0
• my-lws-0-1.my-lws-0

Replica 1:

• my-lws-1.my-lws-1
• my-lws-1-1.my-lws-1

If we create a headless service per sts including a special one for leaders, then we can't achieve the above since the leader will have a different subdomain name. Here is what it will look like:

Replica 0:

• my-lws-0.my-lws
• my-lws-0-1.my-lws-0

Replica 1:

• my-lws-1.my-lws
• my-lws-1-1.my-lws-1

I am fine with the above, but what would we call such a domain name "policy"? it wouldn't be UniquePerReplica.

[ahg-g]

UniquePerReplicaWorkers maybe?

[Edwinhr716]

That makes sense to me, but it does rely on the assumption that if the subdomain is UniquePerReplicaWorkers, that means that the leaders must have their own subdomain.

[kerthcet]

What about leaderSharedWorkerDedicated vs shared or leaderWorkerShared

[Edwinhr716]

Of the three I prefer leaderSharedWorkerDedicated

[ahg-g]

LeadersSharedWorkersDedicated

[liurupeng]

how to select the generated pods to these headless service?

[Edwinhr716]

It uses the GroupIndexLabelKey as selector, so that way all pods that are part of the same replica are selected to the same headless service

[liurupeng]

we need to add the details how pods will be selected by which headless service as well

[ahg-g]

The headless service name will be part of the fully qualified host name of the pod, so how many and what name we give the service matters. It is also why I am suggesting that the API to be a knob that decides on the subdomain policy because this is the user facing side effect.

The pod hostname is <pod-name>.<svc-name>

So if the new "policy" is UniquePerReplica, then each replica including the leader should be part of the same headless service so that the middle part above is common among the pods of a replica. Example:

Replica 0:

• my-lws-0.my-lws-0
• my-lws-0-1.my-lws-0

Replica 1:

• my-lws-1.my-lws-1
• my-lws-1-1.my-lws-1

If we create a headless service per sts including a special one for leaders, then we can't achieve the above since the leader will have a different subdomain name. Here is what it will look like:

Replica 0:

• my-lws-0.my-lws
• my-lws-0-1.my-lws-0

Replica 1:

• my-lws-1.my-lws
• my-lws-1-1.my-lws-1

I am fine with the above, but what would we call such a domain name "policy"? it wouldn't be UniquePerReplica.

[Edwinhr716]

Changed implementation to creating a headless service per sts. Still have to come up with a name for the new subdomainPolicy.
@ahg-g @liurupeng PTAL

[kerthcet]

We have too many labels, can't we determine by the workerIndex?

[Edwinhr716]

We can use workerIndex = 0 to select the leaders, but cannot do workerIndex != 0 to select the workers since AFAIK selector does not have that capability.

[kerthcet]

IIUC, if uniquePerReplica, which means the leader pod and the worker Pods should have the same svc name, but here they're different. my-lws-0.my-lws vs my-lws-0-1.my-lws-0

[Edwinhr716]

The implementation is for headless service per sts, still figuring out a naming scheme so left the old name to discuss it further

[kerthcet]

I prefer the uniquePerReplica option. It makes sense for the leader and worker to have the subdomain in common.

+1, or it's weird I think, unless we have other use cases like visiting the workers only.

[ahg-g]

LeadersSharedWorkersDedicated

[ahg-g]

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, Edwinhr716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment