Add IP mode field to loadbalancer status ingress #97681

Sh4d1 · 2021-01-04T09:50:03Z

Signed-off-by: Patrik Cyvoct patrik@ptrk.io

What type of PR is this?
/kind feature

What this PR does / why we need it:
Implements kubernetes/enhancements#1392
Which issue(s) this PR fixes:

Fixes #79783
Fixes #66607
Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add IP mode field to loadbalancer status ingress

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/pull/1392

/assign @thockin

k8s-ci-robot · 2021-01-04T09:50:11Z

@Sh4d1: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sh4d1 · 2021-01-04T09:51:05Z

pkg/apis/networking/v1/defaults.go

@@ -43,3 +45,15 @@ func SetDefaults_NetworkPolicy(obj *networkingv1.NetworkPolicy) {
 		}
 	}
 }
+
+func SetDefaults_Ingress(obj *networkingv1.Ingress) {


cc @liggitt this should fix #92312 (comment)

I did not realize this struct was reused for Ingress! I don't think this is right. It happens to work for the fields that exist today, but I don't know of any Ingress implementation that is VIP-like.

I think we should probably clone the definition of type LoadBalancerStatus into the networking apigroup.

@bowei @rramkumar1 @aledbf - agree or disagree?

And if we do that, it should be a standalone prefactoring commit. :)

I think we should probably clone the definition of type LoadBalancerStatus into the networking apigroup.

+1

Wouldn't that be a breaking change in client-go? 🤔

Sh4d1 · 2021-01-04T10:20:09Z

/test pull-kubernetes-e2e-kind-ipv6

Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>

Sh4d1 · 2021-01-04T11:20:30Z

/retest

fejta-bot · 2021-01-04T11:26:48Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

Sh4d1 · 2021-01-04T12:57:11Z

/retest

Sh4d1 · 2021-01-04T13:45:39Z

/test pull-kubernetes-e2e-ubuntu-gce-network-policies

Sh4d1 · 2021-01-04T16:37:23Z

/retest

thockin

Thanks for bringing this back.

thockin · 2021-01-04T17:04:51Z

staging/src/k8s.io/api/core/v1/types.go

+	// delivering traffic with the destination IP and port set to the node's IP and nodePort or to the pod's IP and targetPort.
+	// This field can only be set when the ip field is also set, and defaults to "VIP" if not specified.
+	// +optional
+	IPMode *LoadBalancerIPMode `json:"ipMode,omitempty" protobuf:"bytes,3,opt,name=ipMode"`


I am concerned that we are using tag 3 here, when tag 4 is already used. I can't see why PortStatus would have not used 3, and I must have missed that at review time.

@janosi - you added PortStatus in c970a46 - did you use 4 for a reason?

Interesting, I remember that both PR were close in time, and this had to be reverted #96454 , is it possible that is the reason? that this PR predated the PortStatus one?

So should I bump to 5 ? or keep 3? 🤔

Ahh, I am sorry :( I am pretty sure I missed to fallback to 3 when the other PR was reverted :(

It's fine, as long as we understand what happened :)

thockin · 2021-01-04T17:17:45Z

pkg/proxy/iptables/proxier.go

@@ -1160,8 +1160,8 @@ func (proxier *Proxier) syncProxyRules() {

 		// Capture load-balancer ingress.
 		fwChain := svcInfo.serviceFirewallChainName
-		for _, ingress := range svcInfo.LoadBalancerIPStrings() {
-			if ingress != "" {
+		for _, ingress := range svcInfo.LoadBalancerIngress() {


For future - it might have made these reviews a touch easier if there was a single "prefactoring" commit that made this (effectively no-op) structural change, and then the "real" commit would have been that much easier to see.

Noted! (I can still split it if needed 😄 )

@Sh4d1 ,

One of my colleagues has a vested interest in moving this forward. Is it possible to go ahead and split things up as @thockin suggested? I imagine when he is able to conduct a Prod Readiness review that he'll need to start from scratch anyway due to the nearly two year gap in between the above comment and now. Since it would make it easier, maybe split them? I can help too if you like.

thockin · 2021-01-04T17:21:22Z

pkg/proxy/iptables/proxier.go

-					)
-					// jump to service firewall chain
-					writeLine(proxier.natRules, append(args, "-j", string(fwChain))...)
+					if !utilfeature.DefaultFeatureGate.Enabled(features.LoadBalancerIPMode) || *ingress.IPMode == v1.LoadBalancerIPModeVIP {


How about capturing this as a helper function like:

func isVIP(ing *LoadBalancerIngress) bool { if !utilfeature.DefaultFeatureGate.Enabled(features.LoadBalancerIPMode) { return true // backwards compat } return ing.IPMode == v1.LoadBalancerIPModeVIP }

This makes the eventual cleanup easier. We should probably be doing more of this.

thockin · 2021-01-04T17:23:03Z

pkg/proxy/ipvs/proxier.go

-		for _, ingress := range svcInfo.LoadBalancerIPStrings() {
-			if ingress != "" {
+		for _, ingress := range svcInfo.LoadBalancerIngress() {
+			if ingress.IP != "" && (!utilfeature.DefaultFeatureGate.Enabled(features.LoadBalancerIPMode) || *ingress.IPMode == v1.LoadBalancerIPModeVIP) {


same comment re: helper function

thockin · 2021-01-04T17:26:15Z

pkg/proxy/service.go

-
-	if len(ips) > 0 {
-		ipFamilyMap = utilproxy.MapIPsByIPFamily(ips)
+	correctIngresses, incorrectIngresses := utilproxy.FilterIncorrectLoadBalancerIngress(service.Status.LoadBalancer.Ingress, sct.ipFamily)


We switched from a "filter incorrect" model to a "map by family" model because it is easier to think about and more copyable as a pattern. Can we convert this to that model, please?

thockin · 2021-01-04T17:30:39Z

pkg/proxy/util/utils.go

+	var invalidIngresses []v1.LoadBalancerIngress
+
+	for _, ing := range ingresses {
+		correctIP := MapIPsByIPFamily([]string{ing.IP})[ipFamily]


This feels like going out of the way to use the existing API, but it doesn't really fit. You're going to waste time mapping by IP when you KNOW there's just one IP. I think it would be cleaner to iterate the input list, map by family, then return that map.

thockin · 2021-01-04T17:38:20Z

pkg/apis/networking/v1/defaults.go

@@ -43,3 +45,15 @@ func SetDefaults_NetworkPolicy(obj *networkingv1.NetworkPolicy) {
 		}
 	}
 }
+
+func SetDefaults_Ingress(obj *networkingv1.Ingress) {


I did not realize this struct was reused for Ingress! I don't think this is right. It happens to work for the fields that exist today, but I don't know of any Ingress implementation that is VIP-like.

I think we should probably clone the definition of type LoadBalancerStatus into the networking apigroup.

@bowei @rramkumar1 @aledbf - agree or disagree?

thockin · 2021-01-04T17:39:13Z

pkg/apis/networking/v1/defaults.go

@@ -43,3 +45,15 @@ func SetDefaults_NetworkPolicy(obj *networkingv1.NetworkPolicy) {
 		}
 	}
 }
+
+func SetDefaults_Ingress(obj *networkingv1.Ingress) {


And if we do that, it should be a standalone prefactoring commit. :)

k8s-ci-robot · 2021-10-21T11:25:22Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-10-22T15:58:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Sh4d1
To complete the pull request process, please ask for approval from thockin after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

api/OWNERS
pkg/apis/OWNERS
pkg/features/OWNERS
pkg/proxy/OWNERS
pkg/registry/OWNERS
staging/src/k8s.io/api/OWNERS
staging/src/k8s.io/cloud-provider/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2021-10-22T16:01:17Z

@Sh4d1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-e2e-gci-gce-ipvs	`7d62217`	link	false	`/test pull-kubernetes-e2e-gci-gce-ipvs`
pull-kubernetes-node-e2e-containerd	`7d62217`	link	true	`/test pull-kubernetes-node-e2e-containerd`
pull-kubernetes-e2e-gce-alpha-features	`7d62217`	link	false	`/test pull-kubernetes-e2e-gce-alpha-features`
pull-kubernetes-e2e-kind	`7d62217`	link	true	`/test pull-kubernetes-e2e-kind`
pull-kubernetes-integration	`7d62217`	link	true	`/test pull-kubernetes-integration`
pull-kubernetes-e2e-gce-ubuntu-containerd	`7d62217`	link	true	`/test pull-kubernetes-e2e-gce-ubuntu-containerd`
pull-kubernetes-e2e-ubuntu-gce-network-policies	`7d62217`	link	false	`/test pull-kubernetes-e2e-ubuntu-gce-network-policies`
pull-kubernetes-e2e-kind-ipv6	`7d62217`	link	true	`/test pull-kubernetes-e2e-kind-ipv6`
pull-kubernetes-conformance-kind-ga-only-parallel	`7d62217`	link	true	`/test pull-kubernetes-conformance-kind-ga-only-parallel`
pull-kubernetes-e2e-gci-gce-ingress	`7d62217`	link	false	`/test pull-kubernetes-e2e-gci-gce-ingress`
pull-kubernetes-dependencies	`7d62217`	link	true	`/test pull-kubernetes-dependencies`
pull-kubernetes-unit	`7d62217`	link	true	`/test pull-kubernetes-unit`
pull-kubernetes-verify-govet-levee	`7d62217`	link	true	`/test pull-kubernetes-verify-govet-levee`
pull-kubernetes-verify	`7d62217`	link	true	`/test pull-kubernetes-verify`
pull-kubernetes-e2e-gce-100-performance	`7d62217`	link	true	`/test pull-kubernetes-e2e-gce-100-performance`
pull-kubernetes-typecheck	`7d62217`	link	true	`/test pull-kubernetes-typecheck`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dims · 2022-01-10T17:04:56Z

Is this PR still needed, please rebase if so (or we can close it?)

nschad · 2022-01-10T17:14:08Z

Is this PR still needed, please rebase if so (or we can close it?)

I think they are waiting for #106242 to be accepted/merged.

thockin · 2022-01-10T19:22:04Z

It's still in my queue

…

On Mon, Jan 10, 2022 at 9:14 AM Niclas Schad ***@***.***> wrote: Is this PR still needed, please rebase if so (or we can close it?) I think they are waiting for #106242 <#106242> to be accepted. — Reply to this email directly, view it on GitHub <#97681 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVAYCTWWFEW3DFP2L33UVMHW3ANCNFSM4VSYUJKA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

thockin · 2022-01-21T23:34:31Z

It seems unlikely thatr I will get to this as a take-over for the 1.24 cycle

k8s-triage-robot · 2022-04-21T23:36:09Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-05-22T00:22:43Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-06-21T01:22:35Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-06-21T01:22:50Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

polefishu · 2022-07-26T03:23:55Z

pkg/proxy/iptables/proxier.go

@@ -1230,7 +1232,7 @@ func (proxier *Proxier) syncProxyRules() {
 						"-A", string(kubeExternalServicesChain),


Do not wirte reject rule when IPMode == v1.LoadBalancerIPModeProxy?

nschad · 2022-11-03T08:09:29Z

Hey,

now that #106242 is merged should we proceed with this PR or did we move the functionality to a different PR?

thockin · 2022-11-03T16:02:20Z

If someone wants to pick it up, great! Otherwise it is in my queue for "eventually"

…

On Thu, Nov 3, 2022 at 1:09 AM Niclas Schad ***@***.***> wrote: Hey, now that #106242 is merged should we proceed with this PR or did we move the functionality to a different PR? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

akutz · 2023-02-02T19:57:04Z

If someone wants to pick it up, great! Otherwise it is in my queue
for "eventually"

On Thu, Nov 3, 2022 at 1:09 AM Niclas Schad @.***> wrote:

Hey,

now that #106242 is merged should we proceed with this PR or did we move the functionality to a different PR?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you modified the open/close state.Message ID: @.***>

I am taking a look. Tim, I may reach out on Slack. Thanks!

aojea · 2023-03-22T19:20:01Z

@thockin Can we consider re-opening this? My understanding is this still likely needs a new owner (I'm not sure if @Sh4d1 is still planning on moving forward with this).

this is a Pull request , the new owner should open a new one 😄

CharlieR-o-o-t · 2023-08-22T16:45:27Z

@thockin @Sh4d1 I want to pick it up? Can I?

thockin · 2023-08-22T16:54:58Z

This one has been implemented now.

CharlieR-o-o-t · 2023-08-23T08:58:06Z

@thockin sorry, I trying to figure out why this PR has been closed without merge.
Was this KEP implemented in another PR?

Also, don't see any related changes in "master".

aojea · 2023-08-23T11:36:06Z

#119937

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 4, 2021

k8s-ci-robot assigned thockin Jan 4, 2021

k8s-ci-robot requested review from andrewsykim and caseydavenport January 4, 2021 09:50

k8s-ci-robot added sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 4, 2021

Sh4d1 commented Jan 4, 2021

View reviewed changes

Add IP mode field to loadbalancer status ingress

7d62217

Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>

Sh4d1 force-pushed the kep_1860_round_2 branch from 9fc0514 to 7d62217 Compare January 4, 2021 10:31

thockin added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 4, 2021

thockin reviewed Jan 4, 2021

View reviewed changes

thockin reopened this Oct 22, 2021

thockin mentioned this pull request Nov 9, 2021

Copy LoadBalancerStatus from core to networking #106242

Merged

thockin removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Nov 10, 2021

zuzzas mentioned this pull request Feb 10, 2022

cert-manager does not work with Ingress Nginx's proxy-protocol mode deckhouse/deckhouse#841

Closed

2 tasks

joschi36 mentioned this pull request Mar 17, 2022

Ipvs exclude external ip 2 swisstxt/kubernetes#2

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 21, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 22, 2022

k8s-ci-robot closed this Jun 21, 2022

polefishu reviewed Jul 26, 2022

View reviewed changes

thockin mentioned this pull request Jun 29, 2023

Make Kubernetes aware of the LoadBalancer behaviour kubernetes/enhancements#1860

Open

		@@ -1230,7 +1232,7 @@ func (proxier *Proxier) syncProxyRules() {
		"-A", string(kubeExternalServicesChain),

Add IP mode field to loadbalancer status ingress #97681

Add IP mode field to loadbalancer status ingress #97681

Conversation

Sh4d1 commented Jan 4, 2021

k8s-ci-robot commented Jan 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sh4d1 commented Jan 4, 2021

Sh4d1 commented Jan 4, 2021

fejta-bot commented Jan 4, 2021

Sh4d1 commented Jan 4, 2021

Sh4d1 commented Jan 4, 2021

Sh4d1 commented Jan 4, 2021

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Oct 21, 2021

k8s-ci-robot commented Oct 22, 2021

k8s-ci-robot commented Oct 22, 2021

dims commented Jan 10, 2022

nschad commented Jan 10, 2022 • edited

thockin commented Jan 10, 2022 via email

thockin commented Jan 21, 2022

k8s-triage-robot commented Apr 21, 2022

k8s-triage-robot commented May 22, 2022

k8s-triage-robot commented Jun 21, 2022

k8s-ci-robot commented Jun 21, 2022

polefishu Jul 26, 2022 • edited

Choose a reason for hiding this comment

nschad commented Nov 3, 2022

thockin commented Nov 3, 2022 via email

akutz commented Feb 2, 2023

aojea commented Mar 22, 2023

CharlieR-o-o-t commented Aug 22, 2023 • edited

thockin commented Aug 22, 2023

CharlieR-o-o-t commented Aug 23, 2023

aojea commented Aug 23, 2023

nschad commented Jan 10, 2022 •

edited

polefishu Jul 26, 2022 •

edited

CharlieR-o-o-t commented Aug 22, 2023 •

edited