(no logger present) unexpected logger message: {log,erro

[guruprasad raorane]

HI ,

I am new to RabbitMQ as well as Erlang.  Facing issue with RabbitMQ. Details as below.

Setup 

     OS RHEL 7

     RabbitMQ 3.7.7  generic UNIX build.

     Erlang 21.0.1

RabbitMQ is flooded with high number messages using pertest utility. 

bin/runjava com.rabbitmq.perf.PerfTest --heartbeat-sender-threads 10 -b 30 -pmessages 80000 -producers 200 -consumers 4000 -o srno1.csv

Initially  everything looks to be fine. But after a while RabbitMQ stops accepting messages and console is flooded with below error. 

-----------------------------------------

no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.7.0>,{badarg,[{erl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.0.0>,pid=><0.7.0>,time=>1542326033336307}} 

--------------------------------------------

Post that even RabbitMQ restart or even system restart fails to bring back RabbitMQ.  Removal of RabbitMQ and deploy it again is only option. Every attempt to start RabbitMQ results in  no message routed/exchanged and console flooded with above listed error.

I tried searching on web. Found below link. 

https://github.com/erlang-lager/lager/issues/455

The issue looks to be same/similar.  But then looks like it is already addressed in 3.7.7.  Then why this issue seen on my setup(?). 

Since i am new to RabbitMQ as well as Erlang, requesting reply with details description/steps etc. Thanks in advance. 

Guruprasad. 

[Luke Bakken]

Hi Guruprasad,

I'm assuming you're starting RabbitMQ by running ./sbin/rabbitmq-server ... are you using any custom configuration?

PerfTest is running on a different server than RabbitMQ, correct?

I'm going to see if I can reproduce this using RabbitMQ 3.7.10 and Erlang 21.2.2. I will be running RabbitMQ on a FreeBSD server, and PerfTest on my Arch Linux workstation, connected via 1GiB ethernet.

Thanks -

Luke

[Luke Bakken]

Hello,

Some more questions -

Could you please describe the servers being used to run RabbitMQ and PerfTest? If I run PerfTest in my environment with those arguments, it quickly overwhelms RabbitMQ on my other home server.

What sort of test are you trying to do? Do you know what your expected message rate will be?

Thanks -

Luke

[guruprasad raorane]

Hi Luke,

Thanks a lot for looking into this. Please find my response to various questions asked so far.

 

I'm assuming you're starting RabbitMQ by running ./sbin/rabbitmq-server ... are you using any custom configuration?

  ---->  I am running RabbitMQ using ./sbin/rabbitmq-server .   No custom configurations.  In fact No config file is provided, so RabbitMQ is running  with default configuration,

 

PerfTest is running on a different server than RabbitMQ, correct?

Could you please describe the servers being used to run RabbitMQ and PerfTest? If I run PerfTest in my environment with those arguments, it quickly overwhelms RabbitMQ on my other home server.

   ---->  I am running  RabbimtMQ and perftest both on same RHEL box that must be equivalent to your home server.  So RabbitMQ getting overwhelmed is not really problem. Problem is that once RabbitMQ reaches to this state, in no case it can be bring back to normal state. RabbitMQ cannot be shutdown  using command “rabbitmqctl stop”.  So killing rabbitmq (kill – 9 erl_pid ) is only option.

 

What sort of test are you trying to do? Do you know what your expected message rate will be?

---->  I am doing this exercise to validate RabbitMQ. Obviously I am not expecting high msg rate on  such box. Only expectation is that  RabbitMQ should be able recover once msg publishing rate gets reduced/stopped.  Note that it would be perfectly fine if RabbitMQ returns error to publisher/ doesn’t respond under such load. But it should recover on its own or if shutdown command is triggered then it should shutdown in timely fashion.

 

As I mentioned above, RabbitMQ stops responding. It reaches to a state where even shutdown doesn’t stop RabbitMQ in timely fashion. Hence I have to kill RabbitMQ. Then restarting RabbitMQ results into error mentioned in this thread. 

[Luke Bakken]

Hi Guruprasad,

If you overwhelm your server with load it can't handle, especially by running PerfTest on the same machine, it's probably not going to recover gracefully. Give the same sort of a test with other server software and you will see similar behavior.

Note that, in my environment, when I subject RabbitMQ to high (but not unreasonable) load and then remove that load, RabbitMQ returns to normal and can be stopped normally.

I suggest evaluating RabbitMQ using a real-world scenario. Start by not running your benchmark tool on the same server as RabbitMQ. Then, monitor your system so you know where the limits are before they overwhelm it (https://www.rabbitmq.com/monitoring.html)

Thanks,

Luke