Cache size & memory consumption with owned ressources & cache

[olivier chantrel]

Hello,

we encountered memory issues while adding secondary ressources to our CR with go operator-sdk. Indeed, when secondary ressources (like secrets, connfigmaps...) are owned and watched by our CR controller, all these ressources are mounted in cache. This means that for a cluster-scoped operator, if our CR owns secret ressource all the secrets created in the cluster will be cached in memory. 

In our case (big cluster) the controller consumes around 2Go memory even if we have only few CR's (because of several thousands secrets in the cluster).

We had two solutions :

- detach secondary ressources from CR (no more ressources owned by CR) but this means secondary ressources are not watched anymore by our controller (for ex no reconciliation if secret is removed),

- implement custom cache informer to filter secondary ressources by labels (labels should be generated when controller creates secondary ressources).

Would it be possible to make this custom cache implementation more intuitive thanks to operator-sdk and why not automate it for cluster-scoped operators (caching so many ressources seems really unuseful) ?

Thanks,

Olivier

[Varsha Prasad Narsing]

Hello,

Controller-runtime currently exposes the option of filtering the resources to be cached using label selectors on objects. This can be applied while creating c-r `manager`. Here is the doc regarding it: https://github.com/kubernetes-sigs/controller-runtime/blob/master/designs/use-selectors-at-cache.md. I was also able to find an implementation example in OLM's codebase: https://github.com/operator-framework/operator-lifecycle-manager/blob/caab6c52ec532dc82c7178eebb0377bd80d1e82a/cmd/olm/manager.go#L53. This would help in reducing the cache size. Please let us know if this is helpful.

Thank you

Varsha

--
You received this message because you are subscribed to the Google Groups "Operator Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to operator-framew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/af57e92f-d8e8-4761-b9be-c6132d46aeden%40googlegroups.com.

--

Varsha Prasad

Software Engineer
Operator SDK Team
Red Hat Inc. San Francisco

[Olivier CHANTREL]

Hello, 

Thanks Varsha, you're right this is what I called "implement custom cache filter" but I was wondering if opeartor-sdk could simplify this as all clustered-scope operators would need this kind of filtering.

Thanks,

Olivier

[Varsha Prasad Narsing]

Hello Olivier, 

The current implementation in controller-runtime to expose filtering based on labels is easily pluggable in operator code scaffolded by SDK. For example:

If you would like to filter deployments based on labels, as an operator its easier to specify the same while creating the manager:

```

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		NewCache: cache.BuilderWithOptions(cache.Options{
			SelectorsByObject: cache.SelectorsByObject{
				&corev1.Deployment{}: {
					Label: XXXX,
				},
			},
		}),
	})

```

In addition to selecting the resource by specifying the object, we could also use `SelectorsByGVK` (https://github.com/kubernetes-sigs/controller-runtime/blob/v0.13.0/pkg/cache/internal/selector.go#L27). Controller-runtime internally takes care of creating informers for the specified resources, the only task required to do is to mention the objects and labels required to cache. We also have this PR (https://github.com/kubernetes-sigs/controller-runtime/pull/1962) open that would eventually allow users to specify cache funcs per namespace /cluster/ or all other namespaces (more flexibility in caching leading to better memory optimization). 

Since most of the internal implementation is done by controller-time, and users would just have to specify the objects and labels, there is not much that the Operator SDK could scaffold here. We are definitely looking to add more documentation on this topic. However, please let us know your thoughts or any ideas you have that can be implemented in SDK to improve the user experience.

Thank you

Varsha

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/96709079-d213-4994-bbac-da648c562bf7n%40googlegroups.com.

[Olivier CHANTREL]

Thanks for your answer Varsha,

indeed I was looking for a sample, it's quite clear now and I confirm it's quite easy to plug it on operator code.

Thanks

Olivier

[Austin Macdonald]

If anyone is interested in becoming a contributor, there is an open issue for documenting this! 

https://github.com/operator-framework/operator-sdk/issues/5382

To view this discussion on the web visit https://groups.google.com/d/msgid/operator-framework/22821695-34b3-41c6-bb11-1aa3f3b4f634n%40googlegroups.com.