High Log Volume on RabbitMQ-Operator on Kubernetes

Norbert Egger

unread,

Oct 1, 2021, 6:05:30 AM10/1/21

to rabbitmq-users

Hi,

we have deployed a RabbitMQ Cluster Operator on our Google Cloud Kubernetes Engine.

We recognized, that the operator produces an extremly high amount of log entries (hundrets every second) These have level info, here an example and screenshot is attached:

{"level":"info", "logger":"controller-runtime.manager.controller.rabbitmqcluster", "msg":"mutual TLS enabled, looking for CA certificate secret", "name":"mq", "namespace":"default", "reconciler group":"rabbitmq.com", "reconciler kind":"RabbitmqCluster", "secret":"ca-secret", "ts":1.6330816868842402E9}

Any suggestions, how i can i reduce this to only log e.g. warning?

Best regards

Norbert

log-example-rabbitmq-kubernetes-operatpor.png

david....@gmx.de

unread,

Oct 4, 2021, 4:42:44 AM10/4/21

to rabbitmq-users

Hey Norbert,

instead of simply decreasing log verbosity, it might be worth investigating why you see hundreds of log lines every second. You could check the metrics exposed by the Operator Prometheus port to understand how often it reconciles. How many RabbitmqClusters does your Operator manage? Are there very frequent updates to these RabbitmqClusters?

The specific log line you pasted above "mutual TLS enabled, looking for CA certificate secret" should be logged at debug level instead of info level.

The Operator uses https://github.com/go-logr/zapr.

You can therefore configure the Operator loggers via these command line flags.

To only output error logs, configure the Operator Deployment YAML as follows:

spec:

containers:

- args:

- --zap-log-level=error

command:

- /manager

Cheers,

David

david....@gmx.de

unread,

Oct 5, 2021, 1:51:07 AM10/5/21

to rabbitmq-users

We now output this line (and some other lines) at debug level instead of info level: https://github.com/rabbitmq/cluster-operator/pull/861