erl_crash.dump generated while starting/stopping RabbitMQ service as non-root user

[César]

Hi all,

we have seen an intermittent issue (one out of three times), and we are not sure if it was a one-off or something worse.

We ran a sequence of stop + start + restart of RabbitMQ as non-root user. After that we got this:

# ls -lrth /var/lib/rabbitmq/
total 620K
-rw-r--r--. 1 root     root      33K Jan 26 01:15 rabbitmqadmin
drwxr-x---. 4 rabbitmq rabbitmq 4.0K Jan 26 09:59 mnesia
-rw-r-----. 1 rabbitmq rabbitmq 577K Jan 26 10:06 erl_crash.dump

In case it helps, this is the output of those three commands:

# /sbin/service rabbitmq-server stop
Stopping rabbitmq-server: RabbitMQ is not running
rabbitmq-server.

# /sbin/service rabbitmq-server start
Starting rabbitmq-server:
/etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied
rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

# /sbin/service rabbitmq-server restart
Restarting rabbitmq-server: RabbitMQ is not running
/etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied
rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

Have you seen this before? RabbitMQ seemed to be working after that, but the .dump file puzzled us. We are using RabbitMQ 3.5.4

[Michael Klishin]

This error message is what you are looking for.

Any abnormal VM terminations produce a crash dump,

even if the issue has nothing to do with VM state per se.

[César]

Hi Michael,

so that means the Erlang VM crashed during the process? Our concern is that an unprivileged user might be able to crash the VM by doing that.

[dfed...@pivotal.io]

Unprivileged user could start his own Erlang VM, which will crash (e.g. because there is node with same name already running) and write crash.dump. So it shouldn't affect running VM, but you still better check crash reason in .dump file.

[César]

According to the .dump file that was the reason, but is it expected that a start of a new VM is attempted? I would have thought that attempting to spawn a new VM should not occur if an unprivileged user runs the command.

[dfed...@pivotal.io]

You can restrict access to `erl` executable to restrict rights to start a new VM, but it's not a good practice because you could need to run it for other things. You can also restrict access to .dump log directory, so unprivileged users cannot write this log and cause a confusion.

[César]

So I ran the following script

su NON-ROOT-USER -c 'service rabbitmq-server stop; service rabbitmq-server start; service rabbitmq-server restart';   sleep 6; service rabbitmq-server status; ls -la /var/lib/rabbitmq

and after a few iterations the .dump file appeared. The service status command (which is in fact run as root) seems to create the .dump file, as the timestamps match. However, I still don't understand how that could result in the VM crashing.

Is it then likely that an unprivileged user might be able to create another RabbitMQ VM by running that set of commands?

Also, this is the output that I got the iteration prior to getting the .dump file and during the iteration in which it was created:

Stopping rabbitmq-server: RabbitMQ is not running

rabbitmq-server.

Starting rabbitmq-server: /etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied

rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

Restarting rabbitmq-server: RabbitMQ is not running

/etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied

rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

Status of node rabbit@ms1 ...

[{pid,38657},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.5.4"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.4"},

      {webmachine,"webmachine","1.10.3-rmq3.5.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.4-git680dba8"},

      {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.5.4"},

      {ssl,"Erlang/OTP SSL application","6.0"},

      {public_key,"Public key infrastructure","0.23"},

      {crypto,"CRYPTO","3.5"},

      {asn1,"The Erlang ASN1 compiler version 3.0.4","3.0.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.4"},

      {rabbit,"RabbitMQ","3.5.4"},

      {os_mon,"CPO  CXC 138 46","2.3.1"},

      {inets,"INETS  CXC 138 49","5.10.6"},

      {mnesia,"MNESIA  CXC 138 12","4.12.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.5.4"},

      {xmerl,"XML parser","1.3.7"},

      {sasl,"SASL  CXC 138 11","2.4.1"},

      {stdlib,"ERTS  CXC 138 10","2.4"},

      {kernel,"ERTS  CXC 138 10","3.2"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:2:2] [async-threads:64] [hipe] [kernel-poll:true]\n"},

 {memory,

     [{total,48535008},

      {connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,154744},

      {queue_procs,1238464},

      {queue_slave_procs,0},

      {plugins,802952},

      {other_proc,13799704},

      {mnesia,245064},

      {mgmt_db,713832},

      {msg_index,77968},

      {other_ets,1196016},

      {binary,1899360},

      {code,22672983},

      {atom,834473},

      {other_system,4899448}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},{stomp,61613,"::"},{'stomp/ssl',61614,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,1604293427},

 {disk_free_limit,50000000},

 {disk_free,28857769984},

 {file_descriptors,

     [{total_limit,924},{total_used,7},{sockets_limit,829},{sockets_used,5}]},

 {processes,[{limit,1048576},{used,395}]},

 {run_queue,0},

 {uptime,3456}]

total 52

drwxr-xr-x.  3 rabbitmq rabbitmq  4096 Jan 25 12:26 .

drwxr-xr-x. 32 root     root      4096 Jan 25 12:27 ..

-r--------.  1 rabbitmq rabbitmq    20 Jan 25 00:00 .erlang.cookie

drwxr-x---.  4 rabbitmq rabbitmq  4096 Jan 29 14:41 mnesia

-rw-r--r--.  1 root     root     33622 Jan 25 12:26 rabbitmqadmin

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Stopping rabbitmq-server: RabbitMQ is not running

rabbitmq-server.

Starting rabbitmq-server: /etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied

rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

Restarting rabbitmq-server: RabbitMQ is not running

/etc/init.d/rabbitmq-server: line 63: /var/log/rabbitmq/startup_log: Permission denied

rm: cannot remove `/var/run/rabbitmq/pid': Permission denied

Status of node rabbit@ms1 ...

[{pid,38657},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.5.4"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.4"},

      {webmachine,"webmachine","1.10.3-rmq3.5.4-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.4-git680dba8"},

      {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.5.4"},

      {ssl,"Erlang/OTP SSL application","6.0"},

      {public_key,"Public key infrastructure","0.23"},

      {crypto,"CRYPTO","3.5"},

      {asn1,"The Erlang ASN1 compiler version 3.0.4","3.0.4"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.4"},

      {rabbit,"RabbitMQ","3.5.4"},

      {os_mon,"CPO  CXC 138 46","2.3.1"},

      {inets,"INETS  CXC 138 49","5.10.6"},

      {mnesia,"MNESIA  CXC 138 12","4.12.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.5.4"},

      {xmerl,"XML parser","1.3.7"},

      {sasl,"SASL  CXC 138 11","2.4.1"},

      {stdlib,"ERTS  CXC 138 10","2.4"},

      {kernel,"ERTS  CXC 138 10","3.2"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:2:2] [async-threads:64] [hipe] [kernel-poll:true]\n"},

 {memory,

     [{total,48489232},

      {connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,154744},

      {queue_procs,1238464},

      {queue_slave_procs,0},

      {plugins,743880},

      {other_proc,13847856},

      {mnesia,245064},

      {mgmt_db,679464},

      {msg_index,77968},

      {other_ets,1196016},

      {binary,1899112},

      {code,22672983},

      {atom,834473},

      {other_system,4899208}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},{stomp,61613,"::"},{'stomp/ssl',61614,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,1604293427},

 {disk_free_limit,50000000},

 {disk_free,28857769984},

 {file_descriptors,

     [{total_limit,924},{total_used,7},{sockets_limit,829},{sockets_used,5}]},

 {processes,[{limit,1048576},{used,395}]},

 {run_queue,0},

 {uptime,3463}]

total 344

drwxr-xr-x.  3 rabbitmq rabbitmq   4096 Jan 29 15:39 .

drwxr-xr-x. 32 root     root       4096 Jan 25 12:27 ..

-r--------.  1 rabbitmq rabbitmq     20 Jan 25 00:00 .erlang.cookie

-rw-r-----.  1 rabbitmq rabbitmq 297560 Jan 29 15:39 erl_crash.dump

drwxr-x---.  4 rabbitmq rabbitmq   4096 Jan 29 14:41 mnesia

-rw-r--r--.  1 root     root      33622 Jan 25 12:26 rabbitmqadmin

[dfed...@pivotal.io]

In your case this is going on:

  User start script rabbitmq-server

  Since there is another user, it has different erlang cookie and cannot connect to running VM

  Script thinks that server is not running

  Script is trying to start server

  It start NEW VM.

  It has no access to PID file directory and crashes

  NEW VM writes crash.dump

So working VM cannot be affected by this user. 

What you can (and should) do to avoid confusion (and maybe security issues) is to set `rabbitmq-server` rights to not allow unprivileged users to run it (e.g. chmod o-x rabbitmq-server)

[César]

I see. I will try that, it should make things better.

Thanks for your assistance!

[César]

Hi again,

we ran the test during the weekend and unfortunately the .dump files are still there. It is interesting that if the service status is not run at the end then no vm crashes, so that seems to be making a difference!

[dfed...@pivotal.io]

Did you restricted executing access to rabbitmq-server?

[César]

Yes, I ran chmod o-x rabbitmq-server so that only rabbitmq-server user could run it. In fact, the script that I ran did not output anything at all when commands run as a non-root user were issued, so that change did make a difference there.