Multi-instance deployment in cluster using Kustomize

[Rod Oliver]

I have been experimenting with deploying multiple AWX instances in a
cluster, with each instance in its own namespace. I have been
following the process using Kustomize with AWX Operator v1.4.0 with
AWX v21.14.0.

I have seen that it is possible to install 2 different instances of
AWX alongside an instance of AWX Operator in 2 different namespaces.
However when I uninstall one of the AWX + AWX Operator instances using
Kustomize with "kustomization.yaml" as below, using the command
"kubectl delete -k ." that the other AWX is also uninstalled.

I suppose this is because the Operator installation for each deploys
non-namespace-scoped K8s objects and the uninstallation of one of the
Operator instances removes these "shared" objects, which triggers the
removal of the other AWX instance.

apiVersion: kustomize.config.k8s.io/v1beta1
images:
- name: quay.io/ansible/awx-operator
newTag: 1.4.0
kind: Kustomization
namespace: awx-dev25491
resources:
- github.com/ansible/awx-operator/config/default?ref=1.4.0
- awx_manifest.yml

I suppose that instead of installing the AWX instance with Kustomize I
could install it with the command "kubectl apply -f awx_manifest.yml"
and uninstall only that AWX instance with "kubectl delete -f
awx_manifest.yml" and then delete the namespace to get rid of the
namespace-scoped Operator objects. However that doesn't cleanly get
rid of all Operator objects and that even after uninstalling all AWX
instances and their namespaces, the non-namespace-scoped Operator
objects would need to be separately deleted.

Is there a clean way of deleting an Operator instance without
clobbering all AWX instances on that cluster?

[AWX Project]

Part of the problem here is that the AWX custom resource definition (CRD) is a global resource.  So when you run "kubectl delete -k .", when the operator is deleted it will delete the CRD in the process, thus deleting all of the child custom resources (CR) in the process.  

         customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted

So the trick will be finding a way to exclude deleting the 3 CRD's upon teardown.  You'll have the same problem with `make deploy` and `make undeploy`.  I'll think on a potential solution here..... The OLM install leaves the CRD behind when the operator is deleted for this reason (OperatorHub install).

Thanks,

AWX Team

[AWX Project]

You could do something like this to exclude things from the strategic merge of resources kustomize does before applying/deleting.  

Example `kustomization.yaml`

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization

resources:

# Find the latest tag here: https://github.com/ansible/awx-operator/releases

- github.com/ansible/awx-operator/config/default?ref=1.4.0

patches:

- path: delete-cluster-resources/clusterrolebinding.yaml

- path: delete-cluster-resources/clusterrole.yaml

# - path: delete-cluster-resources/awx.yaml

# - path: delete-cluster-resources/awxbackup.yaml

# - path: delete-cluster-resources/awxrestore.yaml

# Set the image tags to match the git version from above

images:

- name: quay.io/ansible/awx-operator

newTag: 1.4.0

# Specify a custom namespace in which to install AWX

namespace: awx

Example patch file in `delete-cluster-resources/clusterrolebinding.yaml`:

$patch: delete

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: awx-operator-proxy-rolebinding

Example patch file in `delete-cluster-resources/clusterrole.yaml`:

$patch: delete

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: awx-operator-proxy-role

However, I wasn't able to use this approach to exclude deletion of the CRD resources because it is not guaranteed that they exist (because of cleanup).  So I would recommend making a cleanup script for the time being.  

Something like:

#!/usr/bin/env bash

kubectl delete serviceaccount/awx-operator-controller-manager -n namespace

kubectl delete role.rbac.authorization.k8s.io/awx-operator-awx-manager-role -n namespace

kubectl delete role.rbac.authorization.k8s.io/awx-operator-leader-election-role -n namespace

kubectl delete rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding -n namespace

kubectl delete rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding -n namespace

kubectl delete configmap/awx-operator-awx-manager-config -n namespace

kubectl delete service/awx-operator-controller-manager-metrics-service -n namespace

kubectl delete deployment.apps/awx-operator-controller-manager -n namespace

Hopefully this helps,

AWX Team