Operator Image Testing

[Shawn Hurley]

Hello All,

As a practice, I as an admin would like some way to "test" the operator based on the image. This means that before I "publish" an operator to users of my cluster I would like to have some test that I can run to verify that the image I have is acceptable. 

This use case requires that the operator-sdk defines a interface to run these tests, and with the work already done for the e2e framework I think we have a good starting point. 

 My idea is that the operator-sdk has a second binary that it creates and installs in every operator-sdk built image. This binary, for now, let's call it operator-sdk-test, will run the packaged e2e that the user has defined for their operator. I suggest that we re-use most of the work already done and we can figure out the changes we need. 

This nice part of this is that the operator-sdk can help users have standards for the operator image, that other systems/users/admins can expect. 

What do you guys think?

Thanks,

Shawn Hurley

[Rob Szumski]

I can't speak to the technical implementation, but I am in full support of this idea. Being able to include tests in some fashion is important so that platform providers like RedHat can independently test Operators on their platform and companies that are looking to heavily bet on an Operator can do the same during their security audits and other sign off before rolling out new versions in their environment.

 - Rob 

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To post to this group, send email to operator-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CADtR-ZdPUU%3D0ScEGYjbgSP_gbJ5bMD1Tof9GeTSG5w%2B_U5nH8w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[David Soff]

Well, since an operator basically ingests events from one or more "topics" and then uses some internal state to produce new "events" on other "topics".

Going by Shawn's idea I think the test container would be a kubeapi server with a second process which would send in the input events. The operator would then do it's magic and would send the results back to the API server. The test process can then verify that the correct objects have been sent.

I hope this makes sense.

David

Op do 30 aug. 2018 om 20:51 schreef Rob Szumski <rszu...@redhat.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CABD1xTbvS5MLcBAXHx-HfaeSuH1kd3%3DNbOpP-F6Pd-4PDBDYSA%40mail.gmail.com.

[hta...@redhat.com]

We're tracking the design and work for this in #435 

There's a few open questions here that I would like clear up with regards to running the test-framework inside the cluster. Specifically the issue of permissions for the test-pod.

Normally when you run the test-framework locally against a k8s cluster its assumed it has enough permissions to create resources needed for the tests (Namespaces,Roles,Rolebindings etc).
But when the test-framework is running inside the cluster it will be using the pod's in cluster kubeconfig.

So the serviceaccount for the test-pod would need a ClusterRole and ClusterRoleBinding to create resources like namespaces for the tests.
The alternative is that the test-pod reuses the namespace that the admin provides it and runs all of the tests in the same namespace.

[Peter Wilcsinszky]

My two-cents:

Although it would be great if all the tests would be easy to run inside the cluster, I see things a bit differently. Current e2e framework is great for interacting with the API since it already has the permissions required to do so locally. As noted already it would be problematic to enable doing the same inside the cluster, at least would need special care all the way down. Why not leave it as simple as it is now and provide programatic extensions to run incluster tests decoupled and maybe provide a separate image that can be used to run e2e remotely.

For basic scenarios it would be possible to write utility functions hiding the complexity of configuring and starting a pod with prepackaged docker images for a simple test scenario. For example we could have a config for an alpine based image (with netcat builtin or installed on the fly) that would receive the endpoint host, port and the timeout parameters and would run a connection check against it. The framework could give us a simple interface to run this blackbox test and pass the required parameters. Another scenario would be something similar but in that case we could provide the command that would run in the given image.

In more complex scenarios (for example when there is a need to install an LDAP server or Kerberos KDC for testing authentication), the util could provide an interface for installing helm charts with custom values and optionally run helm test against those releases. I envision this like a configurable, self-contained test suite that is basically an umbrella chart. This chart would have dependencies on the things that the operator needs to interact with and would contain the tests as annotated helm test pods. 

For complex workflows, where there is a need for parallel execution and such a workflow engine like Argo could be supported.

In the end it could be packaged into a docker image that would be separate from the operator however that doesn't mean it wouldn't be possible to run it incluster. So that instead of providing the tests and test framework in the operator image I beleive it would be cleaner to provide it separately with the ability to run remotely for more flexibility.

Peter

--

You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To post to this group, send email to operator-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/6a8dd6e1-b01f-4b0b-a5e0-e598855b7e67%40googlegroups.com.