Differential Updates with Zarf Packages

[Racer159]

Overview

There has been desire in the community to produce differential packages for Zarf updates to ease Day 2 operations and development cycle times. This discussion is to talk about the potential user flow / implementation for this feature. Notionally this could look like the following.

Flow

Given the below future Zarf package configuration (and associated local manifests - see examples/flux-test in the Zarf repo):

kind: ZarfPackageConfig
metadata:
  name: podinfo-flux
  description: "Deploy flux and then podinfo via flux"
  version: 6.3.3

components:
  - name: flux
    description: "Installs the flux CRDs / controllers to use flux-based deployments in the cluster"
    required: true
    import:
      path: oci://ghcr.io/defenseunicorns/flux:v0.33.0
      name: flux

  - name: podinfo-via-flux
    description: "Example deployment via flux using the famous podinfo example"
    required: true
    manifests:
      - name: podinfo-via-flux
        namespace: podinfo
        files:
          - podinfo-source.yaml
          - podinfo-kustomization.yaml
    repos:
      - https://github.com/stefanprodan/podinfo.git@6.3.3
    images:
      - ghcr.io/stefanprodan/podinfo:6.3.3
    actions:
      onDeploy:
        after:
          # This will use a wait action to wait for the podinfo pods to be ready
          - description: "Podinfo pods to be ready via wait action"
            wait:
              cluster:
                kind: pod
                name: app=podinfo
                namespace: podinfo
                condition: ready

You could then run the following:

zarf package create . --differential oci://ghcr.io/defenseunicorns/podinfo-flux:6.3.2

This would pull the zarf.yaml layer from the specified OCI-published version and compare each component's oci imports first, then it would compose the package and compare the repos, and images and would remove any that were tagged same (while keeping all charts, files, manifests, actions and other keys - if an image or a repo did not have a tag (or sha) it would be retained (i.e. if a full git repo or branch was specified)).

Once finished with that filtering, Zarf would proceed with package create as normal and would create differential Zarf package tarball with a name such as:

zarf-package-podinfo-flux-amd64-6.3.3-differential-6.3.2.tar.zst

This could also be published to oci with:

zarf package publish zarf-package-podinfo-flux-amd64-6.3.3-differential-6.3.2.tar.zst oci://ghcr.io/defenseunicorns

Which would also tag this as a differential package:

oci://ghcr.io/defenseunicorns/podinfo-flux:6.3.3-differential-6.3.2

If we were to inspect the package we would see that it has a new build key to mark it as differential, has the flux key omitted, and has a differentialMissing key of all omitted required components to display a warning on zarf package deploy if a component is not yet in cluster.

kind: ZarfPackageConfig
metadata:
  name: podinfo-flux
  description: "Deploy flux and then podinfo via flux"
  version: 6.3.3
build:
  terminal: guilty-spark
  user: wayne
  architecture: amd64
  differential: true
  differentialMissing:
  - flux
  timestamp: Tue, 14 Mar 2023 18:05:48 -0500
  version: v0.26.0
  migrations:
  - scripts-to-actions

components:
  - name: podinfo-via-flux
    description: "Example deployment via flux using the famous podinfo example"
    required: true
    manifests:
      - name: podinfo-via-flux
        namespace: podinfo
        files:
          - podinfo-source.yaml
          - podinfo-kustomization.yaml
    repos:
      - https://github.com/stefanprodan/podinfo.git@6.3.3
    images:
      - ghcr.io/stefanprodan/podinfo:6.3.3
    actions:
      onDeploy:
        after:
          # This will use a wait action to wait for the podinfo pods to be ready
          - description: "Podinfo pods to be ready via wait action"
            wait:
              cluster:
                kind: pod
                name: app=podinfo
                namespace: podinfo
                condition: ready

Notes

⚠️ NOTE: - the package examples provided assume that #1060 has been completed as a pre-req

⚠️ NOTE: - in the above implementation you would be prevented from composing a differential since (as proposed in #1060) skeleton packages are not the same as full packages:

# The following would not be possible - you cannot publish differential skeleton (composable) packages
components:
  - name: flux
    description: "Installs the flux CRDs / controllers to use flux-based deployments in the cluster"
    required: true
    import:
      path: oci://ghcr.io/defenseunicorns/flux:v0.33.0-differential-v0.32.0
      name: flux

⚠️ NOTE: - we would need to add a key denoting where a component was imported from since this information is removed when packages are built currently.

⚠️ NOTE: - when importing components it is possible to override/merge keys, if other keys were specified other than import we would likely need to keep that import (even if it were the same tag) and build it into the package - we could likely perform image and repo filtering for this however.

⚠️ NOTE: - matching would be done on component names and ordering - iterating through the new component list (NEW), and matching that against the pulled OCI zarf.yaml component list (OLD). If the new list had components not in the old one those would always be included, and if the old one had components not in the new list those would be skipped.

# Given the following oci-imported and regular components
## NEW
oci://ghcr.io/defenseunicorns/flux:v0.33.0
oci://ghcr.io/defenseunicorns/other:v0.1.0
podinfo

## OLD
oci://ghcr.io/defenseunicorns/flux:v0.33.0
oci://ghcr.io/defenseunicorns/hello:v0.2.1
podinfo
world

## RESULT
oci://ghcr.io/defenseunicorns/other:v0.1.0
podinfo

# Given an order change the following would result (since podinfo is found before flux)
## NEW
oci://ghcr.io/defenseunicorns/other:v0.1.0
podinfo
oci://ghcr.io/defenseunicorns/flux:v0.33.0

## OLD
oci://ghcr.io/defenseunicorns/flux:v0.33.0
oci://ghcr.io/defenseunicorns/hello:v0.2.1
podinfo
world

## RESULT
oci://ghcr.io/defenseunicorns/other:v0.1.0
podinfo
oci://ghcr.io/defenseunicorns/flux:v0.33.0

⚠️ NOTE: - this does not help with differential updates for dataInjections (which could also be large)

[runyontr]

It's not clear if the desire for this feature is:

• create smaller zarf packages for data transit to airgap environments
• reduce the time for iterating on an environment and have zarf be part of the developer experience.

From my current experience using and deploying zarf, the latter has come up several times while developing a package for BigBang. If you really want to focus on the first one, I'll move the rest of this comment into its own issue for conversation.

For instance, if I were to be playing around with values files as I debug adding a feature, this is the workflow I've been using:

1. Update the values files locally <--- takes a different amount of time depending on the situation
2. Run zarf package create --confirm <--- takes 2m47s to run. Most time is spent re-querying repo1, finding the images needed, checking all the images, and creating SBOMs, all of which are exactly the same as before:

zarf-v0.025.0-rc2 package create --confirm  106.32s user 17.91s system 74% cpu 2:46.21 total

3. Run zarf package deploy --confirm <--- takes 2m47s at a minimum. This was redeploying the same zarf package that had no changes to the environment and everything returned healthy immediately. There was probably 30ish seconds doing the health checks I've put in. 1m10s were spent checking images. While the registry did scale up, re-using the same tunnel prevented that from having an impact on the timing

zarf-v0.025.0-rc2 package deploy zarf-package-big-bang-amd64-1.54.0.tar.zst   8.98s user 4.37s system 7% cpu 2:47.59 total

Before using zarf, i would literally just run something like this:

 helm upgrade -i bigbang --create-namespace oci://registry.dso.mil/platform-one/big-bang/bigbang/bigbang --version 1.54.0 \
    -n bigbang \
    -f bigbang/credentials.yaml \
    -f bigbang/loki.yaml \
    -f bigbang/ingress-certs.yaml \
    -f bigbang/kyverno.yaml

So here's a situation where it takes 5:30 to update an environment to debug, when I really just needed to run a helm upgrade call if I wasn't using zarf which would have taken 10s.

I think there might be situations where we can get some better profiling data from zarf to see where the time is spent in both the create and deploy calls to help better understand how we can optimize. e.g.

1. re-using an existing zarf package's data instead of re-building (not sure how this plays with declarative nature of what we're trying to do, but just thinking)
2. look at the sha of the images.tar file and compare to the last time it ran
3. Add arguments to zarf package deploy --skip-images --skip-repos to skip those steps during deploy.

[Racer159]

Since this will likely be a complex feature to develop and use over time, my first thought is to focus on your second pain with the knowledge/intent to do it for production use cases in the future - differential packages will take some testing to make sure that they are reliable in production, but the dev use case could be iterated on more quickly - we will likely want to provide warnings when first introducing this feature.

[bburky]

With the recent OCI feature, I think pushing to OCI and deploying from OCI will already reuse docker image/layers and zarf components between updated packages. This means zarf package push, zarf package pull and zarf package deploy oci://... are already differential in some regards now. They won't upload/download unmodified OCI layers if they're found in the cache.

I'm not quite sure what all goes on during zarf package create though. If unmodified imported OCI components (#1060) are used between rebuilds, the download of the unmodified components themselves should be fast (unmodified layers aren't redownloaded if they were cached). I have a few questions though:

• Zarf will avoid downloading git repos and images from the original upstream and use the ones from the imported oci component. Is that right?
• Are things like SBOMs still recomputed for imported components on each zarf package create? Or does it reference the SBOMs embedded inside the component?
• Does this mean if I structure my zarf package as imported OCI components, I should get really fast rebuilds with zarf package create?

If the user can somehow intelligently mirror a docker registry into their airgap you also have incremental data transfer across the airgap today: only changed images and zarf components will be transferred.

[Racer159]

For imported components those only contain any files that were local to the build process (no remote components) so any images or git repos would be re-processed - because these are reprocessed SBOMs are also recalculated and No, OCI components thus far have mostly focused on reusability of configuration over speed.