KEP-3488: Implement Enforcement Actions and Audit Annotations #115973

jpbetz · 2023-02-22T19:22:59Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implements the Enforcement Actions and Audit Annotations sections of CEL for Admission Control.

Special notes for your reviewer:

Adding validationActions as a required field is a intentional (alpha stability) breaking change. We want developers setting this explicitly going forward.

Something we missed when writing the KEP: If multiple bindings for a policy match the same resource, I'd decided to have all the unique audit annotation values produced by the auditAnnotations concatenated into a comma-separated value. For simple cases where all the bindings produce the same audit annotation value, this results in that single value being published to the audit event. For cases where bindings produce different values, the values are concatenated into a comma-separated list so no information is lost. Better ideas on how to best combine the values into a single audit annotation value are welcome.

Deviations from the KEP:

Switched from {policyName}/validation_failure to the validation.policy.admission.k8s.io/validation_failure to adhere better to the conventions of audit annotations.
Custom audit annotations remain in the form {ValidatingPolicyDefinition name}/{auditAnnotation key}. We had wanted to offer a way to prevent a validating or mutating webhook from producing the exact same audit annotation as a ValidatingPolicyDefinition, but the structure of audit annotation keys makes this awkward because only one / is allowed and there is a length restriction on audit annotation keys.

Does this PR introduce a user-facing change?

Adds auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
Adds validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

KEP

jpbetz · 2023-02-22T19:26:25Z

/sig api-machinery
/priority important-soon
/api-review

jpbetz · 2023-02-22T19:26:38Z

/triage accepted

pkg/apis/admissionregistration/types.go

staging/src/k8s.io/api/admissionregistration/v1alpha1/types.go

pkg/apis/admissionregistration/types.go

staging/src/k8s.io/api/admissionregistration/v1alpha1/types.go

pkg/apis/admissionregistration/validation/validation.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/interface.go

pkg/apis/admissionregistration/types.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/policy_decision.go

k8s-triage-robot · 2023-02-22T22:26:34Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

jpbetz · 2023-02-22T23:49:19Z

@cici37 Feedback applied, thanks!

jpbetz · 2023-02-23T02:17:04Z

/retest

sftim · 2023-02-23T10:06:34Z

We had wanted to offer a way to prevent a validating or mutating webhook from producing the exact same audit annotation as a ValidatingPolicyDefinition, but the structure of audit annotation keys makes this awkward because only one / is allowed and there is a length restriction on audit annotation keys.

Does anything stop me from abusing this to write a well-known annotation key, or to try to fool someone into thinking I did?

jpbetz · 2023-03-03T06:01:26Z

Rebased and API review feedback applied!

pkg/apis/admissionregistration/validation/validation.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/cel/compile.go

cici37 · 2023-03-06T19:08:20Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/cel/filter.go

@@ -70,9 +70,6 @@ func (a *evaluationActivation) Parent() interpreter.Activation {

 // Compile compiles the cel expressions defined in the ExpressionAccessors into a Filter
 func (c *filterCompiler) Compile(expressionAccessors []ExpressionAccessor, hasParam bool) Filter {
-	if len(expressionAccessors) == 0 {
-		return nil
-	}


Why to remove this nil check?

Now that we have both validations and auditAnnotations, it is possible for one of the calls to Compile to have an empty slice. We could write special case cost to avoid the compile call, but it is less error prone to make the compile calls safe by not returning an error here.

Here is not really a err handling but rather a short circus. I am ok with either since it is nit and not a blocker any way :)

cici37 · 2023-03-06T19:11:33Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/controller.go

@@ -165,6 +170,8 @@ func (c *celAdmissionController) Run(stopCh <-chan struct{}) {
 	wg.Wait()
 }

+const maxAuditAnnotationValueLength = 10 * 1024


Why this is different than the number in validation?
Also, maybe have a comment to remind keeping it same as it is in validation if it should be the same.
A following up PR might be add this to celconfig file we used in #115747

An expressions that concatenate strings will often produce a longer value than the expression, so the expression limit is set at 5kb and the annotation value limits is set at 10k.

cici37 · 2023-03-06T19:13:58Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/controller.go

+type validationFailureValue struct {
+	Message           string                      `json:"message"`
+	Policy            string                      `json:"policy"`
+	Binding           string                      `json:"binding"`


Maybe use PolicyName and BindingName here since we only assign name not the entire policy/binding?

I was trying to save space since I'm already concerned with the data volume produced by these audit annotation values. It should be clear enough? Let me know if you feel strongly about it.

Just thought we have Policy/Binding all over and it's easily confused in terms of code maintenance.
It is not a merging blocker though :)

cici37 · 2023-03-06T19:17:29Z

staging/src/k8s.io/apiserver/pkg/admission/plugin/validatingadmissionpolicy/validator.go

+	for i, evalResult := range auditAnnotationEvalResults {
+		var auditAnnotationResult = &auditAnnotationResults[i]
+		// TODO: move this to generics
+		validation, ok := evalResult.ExpressionAccessor.(*AuditAnnotationCondition)


Is that possible that evalResult.ExpressionAccessor is nil and evalResult.Error saved the err?
Looks at the code path checks evalResult.ExpressionAccessor.(*AuditAnnotationCondition) before evalResult.Error and the error might be ignored?

This is the same order as for validations above. This !ok check is a failure that should never be reached if the code is working as expected (and will go away when we switch to using generics here).

liggitt

A couple nits/suggestions, but API types, validation, and docs lgtm.

Once the compile/admission/impl bits are reviewed/approved, tag me back in and I can approve.

pkg/apis/admissionregistration/validation/validation.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/cel/filter_test.go

cici37 · 2023-03-06T23:27:50Z

Looks like the tests messages need to be updated with the code updates. Couple nits. Overall LGTM :)

jpbetz · 2023-03-06T23:56:57Z

Feedback applied and commits squashed to minimal set (with codegen separated out)

cici37 · 2023-03-07T03:35:38Z

/lgtm
/approve

/test pull-kubernetes-e2e-kind-ipv6

k8s-ci-robot · 2023-03-07T03:35:44Z

LGTM label has been added.

Git tree hash: fbbfb21bd33f40a55ccdbfc530ec543351172d47

liggitt · 2023-03-07T03:39:48Z

/approve
for API changes

k8s-ci-robot · 2023-03-07T03:40:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cici37, jpbetz, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~api/OWNERS~~ [liggitt]
~~pkg/apis/OWNERS~~ [liggitt]
~~pkg/generated/openapi/OWNERS~~ [liggitt]
~~pkg/registry/admissionregistration/OWNERS~~ [jpbetz,liggitt]
~~staging/src/k8s.io/api/OWNERS~~ [liggitt]
~~staging/src/k8s.io/apiserver/OWNERS~~ [liggitt]
~~staging/src/k8s.io/client-go/applyconfigurations/OWNERS~~ [jpbetz,liggitt]
~~test/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jpbetz changed the title ~~Enforcement actions~~ KEP-3488: Implement Enforcement Actions and Audit Annotations Feb 22, 2023

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Feb 22, 2023

k8s-ci-robot requested review from caesarxuchao and cici37 February 22, 2023 19:29

k8s-ci-robot added area/code-generation area/test kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Feb 22, 2023

jpbetz force-pushed the enforcement-actions branch from d8c042d to ea772d8 Compare February 22, 2023 19:49

cici37 reviewed Feb 22, 2023

View reviewed changes

jpbetz force-pushed the enforcement-actions branch from ea772d8 to d5537b1 Compare February 22, 2023 22:17

liggitt moved this from In progress to Changes requested in API Reviews Mar 2, 2023

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 2, 2023

jpbetz force-pushed the enforcement-actions branch from 924a506 to 1d8bda2 Compare March 3, 2023 03:28

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 3, 2023

liggitt moved this from Changes requested to In progress in API Reviews Mar 3, 2023

cici37 reviewed Mar 6, 2023

View reviewed changes

liggitt moved this from In progress to API review completed, 1.27 in API Reviews Mar 6, 2023

liggitt reviewed Mar 6, 2023

View reviewed changes

pkg/apis/admissionregistration/validation/validation.go Outdated Show resolved Hide resolved

pkg/apis/admissionregistration/validation/validation.go Outdated Show resolved Hide resolved

pkg/apis/admissionregistration/validation/validation.go Outdated Show resolved Hide resolved

jpbetz force-pushed the enforcement-actions branch from 5df742e to 5d7aa42 Compare March 6, 2023 22:40

cici37 reviewed Mar 6, 2023

View reviewed changes

staging/src/k8s.io/apiserver/pkg/admission/plugin/cel/filter_test.go Outdated Show resolved Hide resolved

jpbetz force-pushed the enforcement-actions branch from 7157751 to 012ef09 Compare March 6, 2023 23:56

jpbetz added 3 commits March 6, 2023 21:51

Implement validationActions and auditAnnotations

d221ddb

Generate code

932a4d9

Add integration tests

c2b3871

jpbetz force-pushed the enforcement-actions branch from 012ef09 to c2b3871 Compare March 7, 2023 02:56

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2023

k8s-ci-robot merged commit 0467542 into kubernetes:master Mar 7, 2023

k8s-ci-robot added this to the v1.27 milestone Mar 7, 2023

jpbetz mentioned this pull request Mar 9, 2023

Custom match criteria #116350

Merged

cici37 mentioned this pull request Mar 16, 2023

CEL for Admission Control kubernetes/enhancements#3488

Open

16 tasks

jpbetz mentioned this pull request Mar 17, 2023

KEP-3488 ValidatingAdmissionPolicy: Enforcement actions, audit annotations, and secondary authz kubernetes/website#40098

Merged

2 tasks

alexzielenski mentioned this pull request Jul 20, 2023

KEP-3488: Promote ValidatingAdmissionPolicy to Beta #118644

Merged

43 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-3488: Implement Enforcement Actions and Audit Annotations #115973

KEP-3488: Implement Enforcement Actions and Audit Annotations #115973

jpbetz commented Feb 22, 2023 •

edited

jpbetz commented Feb 22, 2023

jpbetz commented Feb 22, 2023

k8s-triage-robot commented Feb 22, 2023

jpbetz commented Feb 22, 2023

jpbetz commented Feb 23, 2023

sftim commented Feb 23, 2023

jpbetz commented Mar 3, 2023

cici37 Mar 6, 2023

jpbetz Mar 6, 2023

cici37 Mar 6, 2023

cici37 Mar 6, 2023

jpbetz Mar 6, 2023

cici37 Mar 6, 2023

jpbetz Mar 6, 2023

cici37 Mar 6, 2023

cici37 Mar 6, 2023

jpbetz Mar 6, 2023

liggitt left a comment

cici37 commented Mar 6, 2023

jpbetz commented Mar 6, 2023 •

edited

cici37 commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

liggitt commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

KEP-3488: Implement Enforcement Actions and Audit Annotations #115973

KEP-3488: Implement Enforcement Actions and Audit Annotations #115973

Conversation

jpbetz commented Feb 22, 2023 • edited

What type of PR is this?

What this PR does / why we need it:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

jpbetz commented Feb 22, 2023

jpbetz commented Feb 22, 2023

k8s-triage-robot commented Feb 22, 2023

jpbetz commented Feb 22, 2023

jpbetz commented Feb 23, 2023

sftim commented Feb 23, 2023

jpbetz commented Mar 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt left a comment

Choose a reason for hiding this comment

cici37 commented Mar 6, 2023

jpbetz commented Mar 6, 2023 • edited

cici37 commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

liggitt commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

jpbetz commented Feb 22, 2023 •

edited

jpbetz commented Mar 6, 2023 •

edited