Support exposing extra TCP ports in Spark Driver via K8s Ingress

[hiboyang]

Purpose of this PR

See #1984: Support exposing extra TCP ports in Spark Driver via K8s Ingress

Proposed changes:

• Add driverIngressOptions in Spark application CRD to specify parameters for extra ingress
• Refactor Spark UI options related code to reuse code between spark UI and spark driver ingress
• Add unit test for spark driver ingress

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• [ x] Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

This is to support some use cases like exposing Spark Connect server side port in Driver to network traffic outside of the K8s cluster. Another use case is to run Jupyter Notebook Server or Kernel Gateway inside Spark Driver, so user could use Jupyter Notebook within the Spark application.

Checklist

Before submitting your PR, please review the following:

• [ x] I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• [ x] I have added tests that prove my changes are effective or that my feature works.
• [ x] Existing unit tests pass locally with my changes.

Additional Notes

N/A

[yuchaoran2011]

/lgtm

[yuchaoran2011]

/lgtm

[yuchaoran2011]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yuchaoran2011

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuchaoran2011]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment