operators and ci/cd

[Nigel Jones]

I've built a simple go operator (based on memcached) and have ideas/plan for 

the capability I want to add.

However in addition I'd like to get a basic CI/CD process set up. Initially a simple compile, then ideally a test/deploy.

I'm using operator sdk 1.6.1 on macOS primarily, with code hosted in github. Whilst I am experimenting with the operator I am doing 
 - 'make generate' to ensure the go type definitions are up to date
 - 'make manifests' to ensure the CRDs are up to date
 - 'make install' to build the code
- 'make docker-build docker-push IMG=xxxx' to build the image
 - 'make deploy IMG=xxxx' to deploy the operator

Firstly, to get compile validation I tried the first 3 steps in a github action, having made use of the setup_go action. However it failed with '/home/runner/work/egeria-k8s-operator/egeria-k8s-operator/bin/controller-gen: Exec format error' which made me realise that presumably the skeleton project contains architecture dependent code, and my primarily build environment is in ubuntu

Later I was thinking of either using 'make run' to ensure the operator at least launches ok, and perhaps build the image/deploy to a 'KIND" environment within the build pipeline - and run more tests there (already do this on another project)

So I felt it a good time to ask :
 - what is best practice here?
 - what do you do for build validation?
 - how do you approach getting the right native code in place (if that is the issue)
 - how are you automating testing of your operator in your pipeline?

Many thanks
Nigel (@planetf1 on github)

[Nigel Jones]

On a related point, given the way the sdk is laying down a skeleton project to be filled in, what is best practice for

a) Updating to be aligned with later versions of the SDK

- Since I had a 'simple' project I ended up cut/pasting the controller and definitions to go from 1.0.0 to 1.6.1 (as the steps became more complex)
- for 1.6.1 to 1.6.2 I looked at the code diff, and the brief reference in the release notes to image versions going forward .1, and did the same

b) Deciding what gets checked in

 - Currently I am checking in most files from the built project, but am wondering what the appropriate mix of code gen and checkin is (which depends on the original question)

[Daniel Messer]

+operator-framework-sdk-dev 

To keep aligned with later k8s/SDK versions there is a specific section in the SDK docs here: https://sdk.operatorframework.io/docs/upgrading-sdk-version/ - did you find those useful?

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/cdf86070-c650-4dba-88cf-5f0b0f28683fn%40googlegroups.com.

--

Daniel Messer

Product Manager Operator Framework & Quay

Red Hat OpenShift

[Nigel Jones]

Thanks, yes I did come across that yesterday but couldn't find it today!
That being said, I note that the release notes mentioned updating various images to 1.6.2 - which was the only change I ended up making to my go based operator

Still looking for any comments on the build process :-)

[Camila Macedo]

Hi Nigel, 

I will try to share some detailed info, and I hope that helps you.

Pre-requirements

- To run make manifests, you need to have a controller-gen See (here[1]). 

- Also, you will need to have kustomize[2] 

- And then, to run make test[3], you will need to have the env test binaries[15]. 

So, you need to ensure that all requirements will be installed before. You can check here[13] how we set up the environment in the CI for SDK and here[14] how we do that in Kubebuilder. 

Why do we use Kind? 

In Kubebuilder and SDK, we use Kind to test the projects because it allows us to load the manager image that was built in the CI with `kind load docker-image --name`. So, instead of make docker-push you will use that. See here[4] for example. Then, I'd like to recommend you use that. Otherwise, your CI needs to be able to push and pull the image from the public registry. 

Use Envtest

Regarding recommendations, I'd like to suggest you test the controllers with Envtest. It is described in the Best Practices section. See here[5] and then, you will see that it has a link for Testing your Operator project[6].

E2e options

Regarding e2e tests, you can do something that is done in Kubebuilder/SDK using Kubebuilder tests utils lib[7] and ginkgo[8]/gomega[9], which makes that significantly easier, allows debugging via an IDE, etc ...  

You can check in the SDK, and Kubebuilder repos its examples to test the projects in the testdata. See here[10] and here[11]. 

Also, you can use the shell script as we did in the past; see an example[12] with the legacy layout for you to have a better idea.

Cheers,

[1] - https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/Makefile#L119-L120

[2] - https://github.com/kubernetes-sigs/kustomize

[3] - https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/Makefile#L82-L87

[4] - https://github.com/operator-framework/operator-sdk/blob/master/hack/tests/e2e-ansible-molecule.sh#L9-L13

[5] - https://sdk.operatorframework.io/docs/best-practices/common-recommendation/

[6] - https://sdk.operatorframework.io/docs/building-operators/golang/testing/

[7] - https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/utils

[8] - https://onsi.github.io/ginkgo/

[9] - https://onsi.github.io/gomega/

[10] - https://github.com/operator-framework/operator-sdk/tree/master/test/e2e/go

[11] - https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/v3

[12] - https://github.com/operator-framework/operator-sdk/blob/v0.18.0/hack/tests/e2e-helm.sh

[13] - https://github.com/operator-framework/operator-sdk/blob/f298f7c92ee154a0e8123fb13398a7f21720cf0e/Makefile#L147-L151

[14] - https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/setup.sh

[15] - https://master.book.kubebuilder.io/reference/envtest.html

CAMILA MACEDO

SR. SOFTWARE ENGINEER 

RED HAT Operator framework

Red Hat UK

She / Her / Hers

IM: cmacedo

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CAEQM%2BpNdpWPkTK-jAPPg8EmUwUvA3VqxZdEe60tPsTf5kMF9ZQ%40mail.gmail.com.

[Nigel Jones]

camila - thanks so much. Those links look just what I'm looking for. No doubt I'll have further questions in time!

[Camila Macedo]

Hi Nigel, 

Thank you for letting me know. I am glad that is what you are looking for. 

Also, I think would be nice to supplement the docs with these details and examples.

So, if you wish, I just would like to say that it would be very welcome your collaboration with it as well. You might have indeed better inputs after you are able to achieve the goal and based on your experience to do that.

Cheers, 

CAMILA MACEDO

SR. SOFTWARE ENGINEER 

RED HAT Operator framework

Red Hat UK

She / Her / Hers

IM: cmacedo

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/04d12c44-7a71-4e4f-8508-af5a1e669ab1n%40googlegroups.com.

[Nigel Jones]

I've noticed that when issuing 'make test'  there is an attempt to download the correct binaries, if not already present. However when run in Ubuntu (default shell is zsh) I get:

make test
/home/ubuntu/src/egeria-k8s-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/ubuntu/src/egeria-k8s-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
mkdir -p /home/ubuntu/src/egeria-k8s-operator/testbin
test -f /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh || curl -sSLo /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.2/hack/setup-envtest.sh
source /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/ubuntu/src/egeria-k8s-operator/testbin; setup_envtest_env /home/ubuntu/src/egeria-k8s-operator/testbin; go test ./... -coverprofile cover.out
/bin/sh: 1: source: not found
/bin/sh: 1: fetch_envtest_tools: not found
/bin/sh: 1: setup_envtest_env: not found
^Cmake: *** [Makefile:86: test] Interrupt

Even though I'm using zsh 'source' is a builtin, so it looks as if Make is using /bin/sh by default

If we add
SHELL=/bin/bash

into the 'Makefile' generated by the toolkit, then things work ie:

make test
/home/ubuntu/src/egeria-k8s-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/ubuntu/src/egeria-k8s-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
mkdir -p /home/ubuntu/src/egeria-k8s-operator/testbin
test -f /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh || curl -sSLo /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.2/hack/setup-envtest.sh
source /home/ubuntu/src/egeria-k8s-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/ubuntu/src/egeria-k8s-operator/testbin; setup_envtest_env /home/ubuntu/src/egeria-k8s-operator/testbin; go test ./... -coverprofile cover.out
fetching envtest to...@1.19.2 (into '/home/ubuntu/src/egeria-k8s-operator/testbin')
kubebuilder/bin/
kubebuilder/bin/etcd
kubebuilder/bin/kubectl
kubebuilder/bin/kube-apiserver
setting up env vars
?   github.com/odpi/egeria-k8s-operator [no test files]

Now in my case I know /bin/bash is fine - there may be a slight question here about how to ensure the path is always correct, but it looks as if

the Makefile is a little remiss in not include a declaration of 'SHELL' when it is dependent on the builtins that do not appear in 'sh' (which is Make's

default).

Shall I raise a github issue?

Nigel.

--

----
Nigel Jones

[jesus m. rodriguez]

Hello Nigel,

Feel free to add your findings to this existing issue.

https://github.com/operator-framework/operator-sdk/issues/4905

Sincerely,
jesus

> > Red Hat UK <https://www.redhat.com/>

> >
> > She / Her / Hers
> >
> > IM: cmacedo

> > <https://red.ht/sig>

> >
> >
> >
> > On Fri, Apr 23, 2021 at 11:42 AM Nigel Jones <nigel....@gmail.com>
> > wrote:
> >
> > > camila - thanks so much. Those links look just what I'm looking for. No
> > > doubt I'll have further questions in time!
> > >
> > > On Friday, 23 April 2021 at 11:36:14 UTC+1 cma...@redhat.com wrote:
> > >
> > > > Hi Nigel,
> > > >
> > > > I will try to share some detailed info, and I hope that helps you.
> > > >
> > > >

> > > > *Pre-requirements*

> > > >
> > > >
> > > > - To run make manifests, you need to have a controller-gen See (here

> > > > <https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/Makefile#L119-L120

> > > > >
> > > > [1]).
> > > >
> > > > - Also, you will need to have kustomize

> > > > <https://github.com/kubernetes-sigs/kustomize>[2]

> > > >
> > > > - And then, to run make test

> > > > <https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/Makefile#L82-L87

> > > > >[3],
> > > > you will need to have the env test binaries[15].

> > > > <https://master.book.kubebuilder.io/reference/envtest.html>

> > > >
> > > >
> > > > So, you need to ensure that all requirements will be installed before.
> > > > You can check here

> > > > <https://github.com/operator-framework/operator-sdk/blob/f298f7c92ee154a0e8123fb13398a7f21720cf0e/Makefile#L147-L151

> > > > >[13]
> > > > how we set up the environment in the CI for SDK and here

> > > > <https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/setup.sh>[

> > > > 14]
> > > > how we do that in Kubebuilder.
> > > >
> > > >

> > > > *Why do we use Kind? *

> > > >
> > > >
> > > > In Kubebuilder and SDK, we use Kind to test the projects because it
> > > > allows us to load the manager image that was built in the CI with `kind
> > > > load docker-image --name`. So, instead of make docker-push you will use
> > > > that. See here

> > > > <https://github.com/operator-framework/operator-sdk/blob/master/hack/tests/e2e-ansible-molecule.sh#L9-L13

> > > > >[4]
> > > > for example. Then, I'd like to recommend you use that. Otherwise, your CI
> > > > needs to be able to push and pull the image from the public registry.
> > > >
> > > >

> > > > *Use Envtest*

> > > >
> > > >
> > > > Regarding recommendations, I'd like to suggest you test the controllers
> > > > with Envtest. It is described in the Best Practices section. See here

> > > > <https://sdk.operatorframework.io/docs/best-practices/common-recommendation/>[5

> > > > ]
> > > > and then, you will see that it has a link for Testing your Operator
> > > > project[

> > > > <https://sdk.operatorframework.io/docs/building-operators/golang/testing/>
> > > > 6].
> > > >
> > > >
> > > > *E2e options*

> > > >
> > > >
> > > > Regarding e2e tests, you can do something that is done in
> > > > Kubebuilder/SDK using Kubebuilder tests utils lib

> > > > <https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/utils>[7]
> > > > and ginkgo <https://onsi.github.io/ginkgo/>[8]/gomega
> > > > <https://onsi.github.io/gomega/>[9], which makes that significantly

> > > > easier, allows debugging via an IDE, etc ...
> > > >
> > > >
> > > > You can check in the SDK, and Kubebuilder repos its examples to test the
> > > > projects in the testdata. See here

> > > > <https://github.com/operator-framework/operator-sdk/tree/master/test/e2e/go>[10
> > > > ]
> > > > and here
> > > > <https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/v3>

> > > > [11].
> > > >
> > > > Also, you can use the shell script as we did in the past; see an example

> > > > <https://github.com/operator-framework/operator-sdk/blob/v0.18.0/hack/tests/e2e-helm.sh

> > > > Red Hat UK <https://www.redhat.com/>

> > > >
> > > > She / Her / Hers
> > > >
> > > > IM: cmacedo

> > > > <https://red.ht/sig>

> > > > > > <https://groups.google.com/d/msgid/operator-framework/cdf86070-c650-4dba-88cf-5f0b0f28683fn%40googlegroups.com?utm_medium=email&utm_source=footer
> > > > > > >
> > > > > > .

> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Daniel Messer
> > > > >
> > > > > Product Manager Operator Framework & Quay
> > > > >
> > > > > Red Hat OpenShift
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "Operator Framework" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to operator-framew...@googlegroups.com.
> > > > >
> > > > To view this discussion on the web visit
> > > > > https://groups.google.com/d/msgid/operator-framework/CAEQM%2BpNdpWPkTK-jAPPg8EmUwUvA3VqxZdEe60tPsTf5kMF9ZQ%40mail.gmail.com

> > > > > <https://groups.google.com/d/msgid/operator-framework/CAEQM%2BpNdpWPkTK-jAPPg8EmUwUvA3VqxZdEe60tPsTf5kMF9ZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> > > > > >
> > > > > .

> > > > >
> > > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Operator Framework" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to operator-framew...@googlegroups.com.
> > > To view this discussion on the web visit

> > > https://groups.google.com/d/msgid/operator-framework/04d12c44-7a71-4e4f-8508-af5a1e669ab1n%40googlegroups.com
> > > <https://groups.google.com/d/msgid/operator-framework/04d12c44-7a71-4e4f-8508-af5a1e669ab1n%40googlegroups.com?utm_medium=email&utm_source=footer
> > > >
> > > .
> > >
> >
>
> --
> ----
> Nigel Jones
>

[Eric Stroczynski]

The linked bug was fixed in the most recent minor release.

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/3c559d263a0cfacbccf123cddbeb2906c7cd3104.camel%40redhat.com.