Pre Core Policy Exemption Proposal

Best (?) Solutions

Publish separate CRD package:

use in the bundle like so:

kind: UDSBundle
metadata:
  name: user-bundle
packages:
 - name: exemption-crd
 - name: custom-init
 - name: core

pros:

• exemptions can live with package like normal

downsides:

• bit clunky how the CRD package works (files + actions)
• could get weird with upgrades

Make a mini exemption package to deploy after core:

use in the bundle like so:

kind: UDSBundle
metadata:
    name: user-bundle
packages:
  - name: custom-init
  - name: core
  - name: exemption

pros:

• no need to track versions for a separate crd package
• can be implemented right now by teams, without requiring anything from uds-core

downsides:

• weird timing if pods scale/cycle before core is up
• upgrades could be weird if the exemption has to be updated

Allow custom exemptions via values passed to uds-core:

use in the bundle like so:

kind: UDSBundle
metadata:
  name: user-bundle
packages:
- name: custom-init
- name: core
  overrides:
    uds:
     uds:
      - name: EXEMPTION
        path: exemptions

and then in uds-config.yaml:

options:
  architecture: amd64

variables:
   core:
      exemptions: |
        - policies:
            - ...policies
          matcher:
            namespace: ""
            name: ""
          title: ""
          description: ""

pros:

• would prevent most/all timing issues

downsides:

• exemption won't live with the corresponding package
• might be gross to recreate a CR with helm values in the uds-config.yaml

Current Leanings

We're leaning toward option 1 -- publish separate CRD package -- if testing bears out that there's no catastrophic upgrade issues. But of course we want to know what users of core would find to be a better experience.

Other Context

Reasons for rejecting other ideas:

• uds cluster config: isn't ideal to maintain exemptions that are not needed by core itself. has the potential, though probably small, to get out of hand with one off needs for exemptions. It introduces the need for core to test with the custom init packages etc... and also makes a potential unnecessary dependency on core to update when the pre-core exempted package updates.
• while there could still be a use case for namespace wide exemptions, at least for dev mode, it would be better for the policies that don't need to be exempted to still run against the pod in that namespace.
• deploying the exemption as part of the custom-init pkg but wrapped with the helm capability check could have worse side-effects from timing issues.
• deploying the CRD as part of the custom-init pkg would introduce a weird coupling in the pkg to core, such that it could affect the init pkg update and release needs.

testing the CRD package so far has been successful.

Manual tests:

• initial install
• cycle pod after initial install
• update deployment with new policy violation and add exemption to pre-core Exemption CR
• update Exemption CRD and update deployment Exemption CR with new field

I lost track of this. We are still needing a solution. After digging through this and the PR, I don't think this helps us with the rook-ceph specific issues as there are no exemptions for things like hostPath volumes.

We currently have a package that creates exemptions that we deploy post core install which sounds like our option 2? The main issue is the lack of exemption support for things that rook-ceph needs.

@anthonywendt just to check, are you seeing any specific policies you can't exempt? I just checked and we have exemptions for everything possible as far as I can tell. Ref: https://github.com/defenseunicorns/uds-core/tree/main/src/pepr/policies - RestrictVolumeTypes is the exemption/policy name for hostpath, probably in combo with RestrictHostPathWrite for something like Rook.

@anthonywendt just to check, are you seeing any specific policies you can't exempt? I just checked and we have exemptions for everything possible as far as I can tell. Ref: https://github.com/defenseunicorns/uds-core/tree/main/src/pepr/policies - RestrictVolumeTypes is the exemption/policy name for hostpath, probably in combo with RestrictHostPathWrite for something like Rook.

Yeah as you pointed out to me, the regex in our matcher for our exemption we are using wouldn't catch everything we wanted. I think we may be good once we update our exemption appropriately but I will test to verify.