High memory usage for simple helm-operator

[Michele Dolfi]

Hello,

We used the tutorial for building an operator using helm. This is using the latest helm-operator v1.11.0.

We are installing the operator on a non-small OpenShift 4.7 cluster (60 nodes, +2k Pods).

While installing we had to go through multiple OOM crashes and constantly increase the memory limit.

We finally got the operator manager running, but it is constantly using >1.2GB of memory.

We believe there should be something weird. Can something be improved in the PROJECT or watches.yaml file?

Are there known cases which could increase the memory usage?

It looks like that the helm-operator manager is fetching all APIs on the cluster (or at least a huge fraction of those). Is this really needed? Could this be the bottleneck?

Best,

Michele

[Daniel Messer]

High memory usage in controllers is often rooted in large amount of caches the Informers are maintaining. The size increases proportionally with the amount of watched resource kind and watched names. Can you share your watches.yaml? cc @operator-framework-sdk-dev 

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/061cfc45-1252-4fb9-aba0-e7a13c188c64n%40googlegroups.com.

--

Daniel Messer

Product Manager Operator Framework & Quay

Red Hat OpenShift

[Michele Dolfi]

Sure, here is the content of the file.

We also tried setting watchDependentResources to false, but it didn't help.

# Use the 'create api' subcommand to add watches to this file.

- group: apps.cps.deepsearch.ibm.com

version: v1alpha1

kind: KgAmqp

chart: helm-charts/kgamqp

watchDependentResources: false

#+kubebuilder:scaffold:watch

[Daniel Messer]

Can you also describe how the Operator is deployed? if it's via OLM, what's the OperatorGroup configuration?

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/54c3ef4b-8d3b-4b44-9243-af6859afc5e6n%40googlegroups.com.

[Michele Dolfi]

It is deployed on a OCP 4.7 cluster via OLM.

We package a bundle + catalog. We install the CatalogSource, then trigger the install from the OperatorHub UI component in the OCP console.

Best,

Michele

[Camila Macedo]

Hi Michele,

Is that a Helm project, Am I right? Would the selectors be able to help out filter what should or not be watched? Did you try this feature? 

selector

The conditions that a resource’s labels must satisfy in order to get reconciled. For additional information see labels and selectors documentation.

More info: https://sdk.operatorframework.io/docs/building-operators/helm/reference/watches/

Cheers, 

CAMILA MACEDO

SR. SOFTWARE ENGINEER 

RED HAT Operator framework

Red Hat UK

She / Her / Hers

IM: cmacedo

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/CAEQM%2BpMTeq6mJk0uHUYVkjziPDxU7JzSkAKEXdx02y6gr4nH5g%40mail.gmail.com.

[Michele Dolfi]

Hi,

I understood the selector was used only if watchDependentResources is true.

What should we try in there:

- selectors to find the CRs to manage?

- selector to find the resources created by the Helm releases creation by the CRs?

- something else?

Labels selector usually don't allow to restrict to only some APIs, right? So it would still fetch all the APIs on the cluster, but the filter should avoid fetching too much data?

Best,

Michele