Cluster Operator: 401 unauthorized error with default credentials

[Ralph Otowo]

Hello,

I am attempting to create a RabbitMQ cluster using the attached manifest. I have defined a default_user and default_pass according to the example here.

The cluster gets created and I can access the management UI and login using the credentials I supplied within the additionalConfig section, however I am unable to create RabbitMQ CRD objects such as policies (example attached), and users.

I get the following 401 unauthorized error from the messaging-topology-operator pod:

{"level":"error","ts":1669705196.8308785,"msg":"Reconciler error","controller":"policy","controllerGroup":"rabbitmq.com","controllerKind":"Policy","policy":{"name":"high-availability","namespace":"rabbitmq"},"namespace":"rabbitmq","name":"high-availability","reconcileID":"827f20fc-976c-4f34-a816-f72ae6d034c1","error":"Error: API responded with a 401 Unauthorized","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/bitnami/blacksmith-sandox/rmq-messaging-topology-operator-1.8.0/pkg/mod/sigs.k8s.io/controlle...@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/bitnami/blacksmith-sandox/rmq-messaging-topology-operator-1.8.0/pkg/mod/sigs.k8s.io/controlle...@v0.12.3/pkg/internal/controller/controller.go:234"}

In a separate pod (one of the cluster pods), I get the following errors which implies that incorrect credentials are being used for pod to pod authentication.

2022-11-29 06:56:50.550034+00:00 [info] <0.733.0>  * rabbitmq_management
2022-11-29 06:56:50.550034+00:00 [info] <0.733.0>  * rabbitmq_web_dispatch
2022-11-29 06:56:50.550034+00:00 [info] <0.733.0>  * rabbitmq_management_agent
2022-11-29 06:58:35.126254+00:00 [warning] <0.1051.0> HTTP access denied: user 'default_user_gV8gfCqTL3bCDGqaiHB' - invalid credentials
2022-11-29 06:58:35.456578+00:00 [warning] <0.1053.0> HTTP access denied: user 'default_user_gV8gfCqTL3bCDGqaiHB' - invalid credentials
2022-11-29 06:58:36.106326+00:00 [warning] <0.1055.0> HTTP access denied: user 'default_user_gV8gfCqTL3bCDGqaiHB' - invalid credentials

If I remove the default_user and default_pass entries in the manifest file and recreate the cluster, I can create CRD resources without any error. Another user encountered the same issue as discussed in this thread where I replied, but I haven't come across a solution to this problem yet.

Any help is appreciated.

[Chunyi Lyu]

Hi Ralph,

This is a known limitation with the Topology Operator. The Topology Operator reads the k8s default user secret set in RabbitmqCluster status `status.binding` to authenticate with RabbitmqCluster. If the RabbitmqCluster is deployed with pre defined default user creds like your example, the provided creds will overwrite the default user credentials that's auto generated and stored in the default user k8s secret and therefore cause Messaging Topology Operator not be able to access the RabbitmqCluster. You can check the default user secret context to verify. The secret object name follows the pattern of "RabbitmqClusterName-default-user" and it's created in the same namespace.

To get around this issue, either you can stop setting the user creds (like what you did), or you can manually update the content of the default user kubernetes secret to the actual user credentials set by you in spec.rabbitmq.additionalConfig.

If you want to create RMQ (non-default) users via k8s, you can always use the Topology Operator for that: https://www.rabbitmq.com/kubernetes/operator/using-topology-operator.html#users-permissions

Thanks,

Chunyi

[Ralph Otowo]

Hi Chunyi,

I was hoping for a solution to create a kubernetes secret that can be used to pre-configure both RabbitMQCluster objects and the Messaging Topology Operator, especially one that doesn't require plain text secrets to be configured under the additionalConfig section. For now, I'll stick to not configuring any credentials.

I appreciate all your help.

Best,

Ralph