OLM and multi namespaces scoped operator

[Austin Cunningham]

Hi, 

I have been trying to set an operator as multi-namespace scoped by setting the installMode in the CSV 

  installModes: 
    - supported: true
      type: OwnNamespace
    - supported: true
      type: SingleNamespace
    - supported: true
      type: MultiNamespace
    - supported: true
      type: AllNamespaces

and changing the operatorGroup from a single namespace to multiple in spec.targetNamespaces

spec: 
  targetNamespaces: 
    - 3scale-test
    - 3scale-crs

The operator pod is reporting rbac issues listing the custom resources 

E1010 09:52:38.504468 1 reflector.go:178] pkg/mod/k8s.io/clie...@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1beta1.OpenAPI: openapis.capabilities.3scale.net is forbidden: User "system:serviceaccount:3scale-test:3scale-operator" cannot list resource "openapis" in API group "capabilities.3scale.net" in the namespace "3scale-crs,3scale-test"
E1010 09:52:38.504636 1 reflector.go:178] pkg/mod/k8s.io/clie...@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1beta1.Backend: backends.capabilities.3scale.net is forbidden: User "system:serviceaccount:3scale-test:3scale-operator" cannot list resource "backends" in API 

OLM seems to copy the role and role binding from the original namespace and the RBAC looks to have access to the resources via the service account

kubectl auth can-i -n 3scale-test list backends --as system:serviceaccount:3scale-test:3scale-operator 
yes
kubectl auth can-i -n 3scale-crs list backends --as system:serviceaccount:3scale-test:3scale-operator 
yes

If I change the operatorGroup  spec.targetNamespaces to cluster scoped the operator pod restarts and starts to reconcile with out issue. 

Strangely if I go from cluster scoped back to multi-namespaced scope I don't see the rbac issue but the operator is no longer reconciling . 

 

Question: am I missing anything in the setup for multi namespace scope? 

Note have tried this with two seperate operators with the same result sdk versions

- v1.2.0

- v1.14.0

Issue here 

https://issues.redhat.com/browse/MGDAPI-4664

--

Austin Cunningham

Senior Software Engineer - ClouD Services

Red Hat

Communications House, Cork Road, Waterford X91NY33

Ireland

aucu...@redhat.com   

[Daniel Messer]

Hi Austin,

Thanks for reaching out. For this to work your operator needs to support accepting a comma-separated list of namespace in the WATCH_NAMESPACE env var and set up separate informers / watchers per namespace in its logic. You are currently likely just passing down this value into your operator pod which is why it tries to establish a watch on the namespace called "3scale-crs,3scale-test" as per the log output. This namespace doesn't exist.

HTH,
Daniel

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CAHE0bNG-1YQiykUSQk3f-YyuYenFPKq5BRMey-gpjw4FdNa2FA%40mail.gmail.com.

--

Daniel Messer

Product Manager Operator Framework & Quay

Red Hat OpenShift

[Austin Cunningham]

Thanks Daniel 

That makes sense

Guess I ignored that section of the docs 

    ...
watchNamespace, err := getWatchNamespace()
if err != nil {
    setupLog.Error(err, "unable to get WatchNamespace, " +
        "the manager will watch and manage resources in all Namespaces")
}

options := ctrl.Options{
    Scheme:             scheme,
    MetricsBindAddress: metricsAddr,
    Port:               9443,
    LeaderElection:     enableLeaderElection,
    LeaderElectionID:   "f1c5ece8.example.com",
    Namespace:          watchNamespace, // namespaced-scope when the value is not an empty string
}

// Add support for MultiNamespace set in WATCH_NAMESPACE (e.g ns1,ns2)
if strings.Contains(watchNamespace, ",") {
    setupLog.Info("manager set up with multiple namespaces", "namespaces", watchNamespace)
    // configure cluster-scoped with MultiNamespacedCacheBuilder
    options.Namespace = ""
    options.NewCache = cache.MultiNamespacedCacheBuilder(strings.Split(watchNamespace, ","))
}
...

 

One more question, should we care about config/manager/manager.yaml as WATCH_NAMESPACE will just be treated as a string whether the value is ns or ns1,ns2 ?

[Tony Wu]

Hi Austin,

One more question, should we care about config/manager/manager.yaml as WATCH_NAMESPACE will just be treated as a string whether the value is ns or ns1,ns2 ?

Don't worry about that as the string will be split by "," when being used to create `MultiNamespacedCacheBuilder` (see here).

And if you are integrating with OLM, the manager's WATCH_NAMESPACE env var will be replaced by the CSV with its `olm.targetNamespaces` annotation that reflects the targetNamespaces specified in the OperatorGroup.

Keep in mind that "MultiNamespace" installMode could easier lead to "OperatorGroups with intersecting provided APIs competing for the same resources in the set of intersecting namespaces" [0], so it is less recommended if your Operators could be installed more than once on a cluster.

Happy weekend!

Tony

---

(If your Operator will only be installed once on a cluster, let's proceed with caution.)

To better showcase how an Operator can dynamically watch namespaces with the help of the OLM, I've created a sample Memcached Operator based on the SDK Go tutorial. (I also list the github repo [1] and the bundle image [2] in case you'd like to take a look and try it on your own.)

I went ahead and installed it using the Operator SDK:
operator-sdk run bundle quay.io/tlwu2013/go-tutorial-hack-bundle:v0.0.1 -n tw5 --install-mode MultiNamespace=tw6,tw7

The above run bundle command basically is saying, I want:

• to install Operator in namespace: tw5
• the Operator be installed in "MultiNamespace" installMode, initially watching namespace: tw6,tw7
• this Operator allows me to later add an additional namespace: tw3 as one of the targetNamespaces after the installation
• That way, tenants of namespace: tw3 can create Memcached custom object there (and know they are able to do so by the CSV)

Step 1. after the "run bundle" cmd, in the OCP console, you can see the Operator is installed in namespace tw5 and managing namespace tw6,tw7

Step 2. I created a Memcached custom object in namespace: tw6

Step 3. The Memcached deployment got 3 pods spun up correctly by the Operator in namespace: tw6

Step 4a. You can see the manager's WATCH_NAMESPACE env var is replaced by the CSV with its `olm.targetNamespaces` annotation

Step 4b. as in CSV's YAML

Step 5. add namespace: tw3 as an additional targetNamespace (--> *This is when could easily cause intersecting/overlapping between the same provided APIs among OperatorGroups if your Operators are installed many times on cluster.)

Step 6. OLM would update CSV `olm.targetNamespaces` annotation accordingly and the changes would flow into the manager's WATCH_NAMESPACE env var for the additional namespaced cache.

Step 7. Created a Memcached custom object in namespace: tw3, and the Memcached deployment spun up 3 pods by the Operator correctly

[0] OperatorGroup Intersection

[1] github repo: go-tutorial-hack-multiNS

[2] bundle image of go-tutorial-hack-bundle:v0.0.1

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CAHE0bNEdKTZEL36S6CCLr1h_xEFrAbdkmdV69WVZWaq1BADPzA%40mail.gmail.com.

--

Tony Wu

OpenShift Product Management

Red Hat OpenShift

ton...@redhat.com

[Austin Cunningham]

Thanks Tony,

     Yes this is exactly why I asked the question. We need the operator to be able to switch dynamically on changes in the operator group and we are already using `olm.targetNamespaces`.  Thanks for the example 😃.  

Regards,

Austin