Epinio K8S Controller

[manno]

Epinio is currently implemented as a CLI that talks to an API server, that knows how to instruct Kubernetes to create resources.

To be more transparent and useful to knowledgeable human operators, Epinio could add a controller, that reacts to custom resources. In this approach the API server would create custom resources, just like a human operator would.

To me it seems the API server would still be useful:

• because it would have a separate role based access (Epinio users).
• because a REST API is a more traditional interface, which can be used by WebUI's and such.
• The API simplifies the Kubernetes experience for less knowledgeable users

The Epinio CRDs would probably match the endpoints:

• "staging image" (= run buildpack)
• app deployment
• namespaces
• services (doesn't need a CRD?)
• ...

[agracey]

The API also does two other important things:

• Bridges the impedance mis-match between a procedural developer experience and the reconciliation loop based Kubernetes internals
• Gives a contract to port tooling to other implementations if ever needed

[agracey]

After discussing this with a few people, this could be related to #986 as another avenue to explore.

If we could instead use the installed chart instance's values as the source of truth for an app's configuration, we could drop the need for an additional CRD. This would reduce our surface area as well as give a whole package in an industry standard format to move between clusters.

[mattfarina]

transparent and useful to knowledgeable human operators

I've been wondering who these human operators are and what their needs may be. I've noticed there are two forms of human operators:

1. cluster admins - these are the people who administer the cluster. They manage things like cluster extensions.
2. workload operators - work with running the workloads in the cluster.

These two roles are often handled by different people and can be on different teams. Take an organization that has 10 global clusters. One team may operate those clusters while people on different teams run workloads in those clusters in a multi-tenant manner.

These two roles also have a different focus. One is interested in how Kubernetes is running and will likely have expertise in the VMs, networking, and storage Kubernetes works with. The other will have to working with and know the specific workloads and how they work within Kubernetes.

Note, neither of these is the epinio user who is using it to run their apps.

So, what kinds of things do they need?

• cluster admins are the ones who would install epinio into a cluster. If there are CRDs and custom non-namespaced controllers they are they ones with access to do this and who are interested in this. They need an easy experience to install, upgrade, monitor, and troubleshoot the control plane and components they are running.
• cluster admins will want to know what they are running. They will have the ability to see this once installed by inspecting k8s but I suspect they would like to see the details up front.
• workload admins are mostly (or all for 80% of cases) automated out. This is the idea of a PaaS and, I think, the thing that needs to work the best.
• workload admins do still have some edge cases such as taking the workload created by the PaaS and running it elsewhere. This is an edge case and outside the main use of a PaaS.

Kubernetes is a platform for building platforms. It's a better place to start; not the endgame. -- Kelsey Hightower

App Devs don't want to deal with k8s. They just want to run their code on the computer (which happens to be a cluster somewhere).

So, who needs what level of transparency? Here are some ideas...

• cluster admins want to know what and at what version things are running. This includes the base containers used by the workloads (this should be configurable)
• workload admins who want to pull the workloads out to run elsewhere (an edge case) will want to have enough transparency to export the package in a way they can run elsewhere. I would suggest that when running it elsewhere you can do that without any epinio components being present.

[manno]

That doesn't sound like “cluster” and “workload” admins would want to interact with the controller.
So, to summarize, apart from edge cases, this is seems out-of-scope for a PaaS.

Or do companies have “cloud native devop teams” that want to write their own controller logic to impose rules on deployments? I'm only guessing, do they do that for customization, to make it work with private cloud features, for security, or to do deployments in a batch?

Some of that customization might be solved by #986, when ops can edit the templates used for Epinio app deployments.

[mattfarina]

Or do companies have “cloud native devop teams” that want to write their own controller logic to impose rules on deployments? I'm only guessing, do they do that for customization, to make it work with private cloud features, for security, or to do deployments in a batch?

If we are talking the PaaS push to run code setup, I do not think anyway wants to batch that. You want to push your code and see it run.

When it comes to the case of taking that code, exporting it, and running it elsewhere, I think people will want to run it in a variety of ways. GitOps, manually installed, using terraform, etc. The export should be flexible enough it can work with their process of choice.

As for customization, how often do existing PaaS operators create their own buildpacks. I imagine it happens but it's rare. Before trying to figure out what to create here I would look to understand what they want and need to do. Then supply that, as a feature later, in the simplest manner possible. Don't need to over engineer this.

That's my 2 cents, anyway.