Why not use vanilla StatefulSet/Deployment + client-go instead of Operator SDK?

[Ari]

Hello everyone, we have been using Kubernetes and we mostly do our development using Kubernetes vanilla sets. We send the operations via client-go

For example, for Redis, we would just use StatefulSet, the same for MySQL. When I look at the operator SDK code, I see a lot of work goes into what seems to be basic operations.

We are looking at the Operator SDK to see if it can provide any value, but we're having a difficult time finding a use case. We also are looking at public operators, but generally they seem pretty complicated.

For example, Service, StatefulSet, PVC creation happens in the operator from CRD changes. You could just create all of those using client-go and remove the complexity of operators by a huge factor. 

The main benefit I see may be the fact that the operator-sdk handles/abstracts controller logic for you, such as queuing, or watching objects and sending updates to the cluster.

Can someone help me understand why you would develop a MySQL operator using CRDs and operators instead of auto generating the StatefulSet from a builder and shoving the update to client-go?

I look at MySQL operators and Redis operators and I see how I could create the exact behaviour using 100 lines of code and a StatefulSet/Service/PVC with a few InitContainers, but I see a huge codebase with complexity that could lead to failures of these services in production due to the complexity by introducing operators.

Any input is greatly appreciated, we want to be apart of the Operator movement, but having trouble understand when you would use this, especially since most examples basically just give you TLS and Backups, which I believe should not even be handled by the operator.

[Daniel Messer]

Hi Ari,

thanks for raising this topic. The WHY often gets lost in the high velocity of development in the ecosystem. It seems like your question is rather "Why use the Operator pattern" than "Why use the Operator SDK". They are both equally valid and interesting though, so here are my 2 cents:

Why use the Operator pattern?

- standardized interaction via a Kubernetes-native API

- mature API versioning, migration and validation features in contemporary Kubernetes

- easy addition to existing Kubernetes primitives via Mutating Webhooks

- nice client UX to custom extensions using Admission Webhooks (fail fast)

- "free" storage via etcd

- drives GitOps

- Kubernetes RBAC to limit who can use the service

- in general very low learning curve for Operator end users: they can re-use all known control concepts from existing Kubernetes types

Why use the Operator SDK?

- all the benefits of Kubebuilder (which is what the SDK uses for scaffolding) like all kinds of boilerplate code generation for types, deep copy, hooks, registration, etc

- integrated testing from unit tests, over integration tests (envtest) to functional tests (kuttl)

- integrated packaging to distribute updates in a very controlled way (via a graph, using OLM)

- migration support across controller-runtime releases

- a lot of helpers to quickly run a local controller

I think the general advice of using the right tool for the right job still stands. Not everything is worth writing an Operator for. Application-aware procedures like configuring TLS and running application-consistent backup all on their own could also be done without CRD. Initial cluster formation could also be done with initContainers. But it adds up quickly if you want to provide a good experience for users that you do not know at this point in time. And very soon you are shipping a lot of code along your application image that is for management and not for business logic.

With the Operator pattern you are providing a much more consistent user experience that is nowadays well understood. It's the same reason why Kubernetes itself, despite its complexity, is successful: all clusters may be set up differently but work the same. All user knowledge and learnings are transferable. You don't have to re-learn how you interact with them to deploy software, just because you are using Kubernetes distributions from entirely different providers on completely separate infrastructure.

Operators essentially resemble not only handy extensions to the cluster to automate things like certificate management, policies enforcement, quota configuration, security audit, etc but provide managed services for things like Databases, Message Queues, Service Meshes, basically any distributed system sufficiently complex enough that it cannot solely be managed by on-board Kubernetes controllers. This is where reconciliation in a loop to avoid drift, versioned APIs and repeatable UX become very important. 

If you are building an Operator it is very likely that some of the backing services and functionalities you need are already provided by other Operators out there. This is why we created OperatorHub.io and donated the entire Operator Framework to CNCF, so that the ecosystem continues to grow and it becomes even more seamless to re-use these Operators. This is not possible without some standardizations in the interfaces and interaction patterns. There are for sure alternatives out there but CRDs in general and the Operator pattern in particular have seen the broadest adoption and support so far.

A bit lengthy but hopefully I could provide some additional angles. Code complexity might be higher initially but that's what tools like Kubebuilder, Kudo, Operator-SDK etc are for.

/Daniel

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/6c39ae3f-9382-467f-a93e-27c2b4baa637n%40googlegroups.com.

--

Daniel Messer

Product Manager Operator Framework & Quay

Red Hat OpenShift

[Marc Boorshtein]

We've been developing and using our operator for 18 months.  If you really have just K8s standard objects (statefulset,deployment,etc) then an operator probably isn't for you.  What you gain in a tighter deployment you'll lose with trying to abstract away k8s native objects then having users make changes after the fact for every combination.  

Operators are going to come in handy when you:

1.  Need to orchestrate multiple complex relationships (ie storage abstracting away complex primitives)

2.  provide an operations API, ie an easy way for admins to trigger backups in a cloud native way

3.  need to do something that can't be done declaratively easily (for instance our tool needs a PKCS12 file, we use our operator to create one from k8s secrets so there's a cloud native way to do it)

operators shouldn't be used for configuration management.  there are better models (ie GitOps) that aren't as restrictive.  Also, if your app can work with the API directly, it makes it less necessary to have an external "operator" vs managing the watch inside your application.

There are advantages to using OLM (which isn't necessary for an operator to be an operator).  Especially if your target is openshift.  OLM hasn't gotten the momentum outside of openshift unfortunately (though it is a CNCF project now so maybe that will change?).  

hope that helps  

Marc Boorshtein
CTO Tremolo Security
marc.bo...@tremolosecurity.com
Twitter - @mlbiam / @tremolosecurity