Internal server error authenticating to to Rabbit from management UI

[V Z]

Trying to login Rabbit this morning, and getting this (I scrubbed the real user id with ***userid***):

Got response code 500 with body {"error":"Internal Server Error","reason":"{error,badarg,\n [{ets,lookup,[rabbit_user,<<\"***userid***\">>],[]},\n {rabbit_misc,dirty_read,1,[]},\n {rabbit_auth_backend_internal,internal_check_user_login,2,[]},\n {rabbit_access_control,try_authenticate,3,[]},\n {rabbit_access_control,'-check_user_login/2-fun-0-',4,[]},\n {lists,foldl,3,[{file,\"lists.erl\"},{line,1262}]},\n {rabbit_mgmt_util,is_authorized,6,[]},\n {webmachine_resource,resource_call,3,[]}]}\n"}

Any ideas? Everything was fine yesterday. Thank in advance!

[Michael Klishin]

The only reason for ets:lookup/2 to throw a badarg is when the table it tries to use

does not exist. I cannot think of why `rabbit_user` can be missing. No client should be

able to connect if it doesn't.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

Which table, please?

This error comes up regardless whether the user is authenticated via LDAP or via internal database. Here is the screenshot when using the user from internal database.

[Michael Klishin]

The error comes from the internal backend which tries to query an ETS table named `rabbit_user` and

it throws a `badarg`. Which is not expected and I haven't seen before.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[V Z]

Tried to get status (see attached). No listeners were running. Tried to get the report but failed (see output attached). Tried 'rabbitmqctl start_app', which seems to have worked but still no listeners and still cannot login. Shall we attempt to restart erlang VM?

[V Z]

Stopped Erlang VM on one of the nodes (in a four node cluster), and the other 3 nodes came to life! 

However, could not restart that one node because it could not right to enabled plugins file. Apparently, node is out of disk space (25G). Disk alarm (1MB by default) was not triggered, or maybe it all happened too fast.

root@6ff88acb-571f-4a5d-8024-f589b4716701:/var/vcap/store/rabbitmq/mnesia/db/queues# ls -ld ADL1V56KEP9SGKTU2VV4UOYIO

drwxrwxr-x 2 vcap vcap 36864 May 6 06:12 ADL1V56KEP9SGKTU2VV4UOYIO

root@6ff88acb-571f-4a5d-8024-f589b4716701:/var/vcap/store/rabbitmq/mnesia/db/queues# du -sh ADL1V56KEP9SGKTU2VV4UOYIO
24G ADL1V56KEP9SGKTU2VV4UOYIO

Any idea how to figure out which queue that is and how to recover? Thanks!

[Michael Klishin]

Sadly, no. Queue names are hashed (not so much for security purposes

but rather because queue names can contain characters that can be unsafe as filesystem names).

Listing queues by their size in bytes would help but likely in your situation more internal database operations can fail.

We'd be interested in how this can be made more operations friendly.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.