RabbitMQ stops from time to time with segfault

[Nilo Menezes]

Hi Guys,

I posted this as a bug, but it was closed and I was advised to post it here.

I don't have much more information, as RabbitMQ dies, but I will try to get a core dump.

I have a small RabbitMQ server with millions of messages. It all works well, but from time to time it simply stops. I saw a segfault being logged on dmesg, but besides that I don't have any extra information.
The machines had enougth ram and disk space.

I'm using RabbitMQ 3.6.9 with Erlang 19.3, on Ubuntu 16.04.2 LTS - Linux xcipp 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux.

rabbitmq.config:

[
    {kernel, [
      {inet_default_connect_options, [{nodelay, true}]},
      {inet_default_listen_options,  [{nodelay, true}]}
    ]},
    {rabbit, [
       {tcp_listeners, [{"192.168.1.224", 5672}]},
       {tcp_listen_options, [                         
                          {backlog,       4096},
                          {nodelay,       true},
                          {linger,        {true,0}},
                          {exit_on_close, false},
                          {sndbuf,        196608},
                          {recbuf,        196608}
                         ]},
       {vm_memory_high_watermark, 0.6},
       {vm_memory_high_watermark_paging_ratio, 0.30},
       {log_levels, [{channel, debug}, 
                  {connection, info}]}
    ]}
].

rabbitmq-env.conf

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+A 128 +t 10048576"

enabled_plugins

[rabbitmq_management].

/etc/sysctl.conf

net.ipv4.tcp_max_syn_backlog=2048
net.core.somaxconn = 1024

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63685
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63685
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[Nilo Menezes]

rabbitmqctl environment

 

Application environment of node rabbit@chipp...
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
 {asn1,[]},
 {compiler,[]},
 {cowboy,[]},
 {cowlib,[]},
 {crypto,[]},
 {inets,[]},
 {kernel,
     [{error_logger,tty},
      {inet_default_connect_options,[{nodelay,true}]},
      {inet_default_listen_options,[{nodelay,true}]},
      {inet_dist_listen_max,25672},
      {inet_dist_listen_min,25672}]},
 {mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@chipp"}]},
 {os_mon,
     [{start_cpu_sup,false},
      {start_disksup,false},
      {start_memsup,false},
      {start_os_sup,false}]},
 {public_key,[]},
 {rabbit,
     [{auth_backends,[rabbit_auth_backend_internal]},
      {auth_mechanisms,['PLAIN','AMQPLAIN']},
      {background_gc_enabled,false},
      {background_gc_target_interval,60000},
      {backing_queue_module,rabbit_priority_queue},
      {channel_max,0},
      {channel_operation_timeout,15000},
      {cluster_keepalive_interval,10000},
      {cluster_nodes,{[],disc}},
      {cluster_partition_handling,ignore},
      {collect_statistics,fine},
      {collect_statistics_interval,5000},
      {config_entry_decoder,
          [{cipher,aes_cbc256},
           {hash,sha512},
           {iterations,1000},
           {passphrase,undefined}]},
      {credit_flow_default_credit,{400,200}},
      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
      {default_user,<<"guest">>},
      {default_user_tags,[administrator]},
      {default_vhost,<<"/">>},
      {delegate_count,16},
      {disk_free_limit,50000000},
      {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
      {error_logger,{file,"/var/log/rabbitmq/rabbit@chipp.log"}},
      {fhc_read_buffering,false},
      {fhc_write_buffering,true},
      {frame_max,131072},
      {halt_on_upgrade_failure,true},
      {handshake_timeout,10000},
      {heartbeat,60},
      {hipe_compile,false},
      {hipe_modules,
          [rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
           rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
           rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
           rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
           rabbit_guid,rabbit_net,rabbit_amqqueue_process,
           rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
           delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
           rabbit_limiter,gb_trees,rabbit_queue_index,
           rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
           rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
           rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
           mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
           ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
      {lazy_queue_explicit_gc_run_operation_threshold,1000},
      {log_levels,[{channel,debug},{connection,info}]},
      {loopback_users,[<<"guest">>]},
      {memory_monitor_interval,2500},
      {mirroring_flow_control,true},
      {mirroring_sync_batch_size,4096},
      {mnesia_table_loading_retry_limit,10},
      {mnesia_table_loading_retry_timeout,30000},
      {msg_store_credit_disc_bound,{4000,800}},
      {msg_store_file_size_limit,16777216},
      {msg_store_index_module,rabbit_msg_store_ets_index},
      {msg_store_io_batch_size,4096},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {password_hashing_module,rabbit_password_hashing_sha256},
      {plugins_dir,
          "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.9/plugins"},
      {plugins_expand_dir,
          "/var/lib/rabbitmq/mnesia/rabbit@chipp-plugins-expand"},
      {queue_explicit_gc_run_operation_threshold,1000},
      {queue_index_embed_msgs_below,4096},
      {queue_index_max_journal_entries,32768},
      {reverse_dns_lookups,false},
      {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@chipp-sasl.log"}},
      {server_properties,[]},
      {ssl_allow_poodle_attack,false},
      {ssl_apps,[asn1,crypto,public_key,ssl]},
      {ssl_cert_login_from,distinguished_name},
      {ssl_handshake_timeout,5000},
      {ssl_listeners,[]},
      {ssl_options,[]},
      {tcp_listen_options,
          [{backlog,4096},
           {nodelay,true},
           {linger,{true,0}},
           {exit_on_close,false},
           {sndbuf,196608},
           {recbuf,196608}]},
      {tcp_listeners,[{"192.168.1.224",5672}]},
      {trace_vhosts,[]},
      {vm_memory_high_watermark,0.6},
      {vm_memory_high_watermark_paging_ratio,0.3}]},
 {rabbit_common,[]},
 {rabbitmq_management,
     [{cors_allow_origins,[]},
      {cors_max_age,1800},
      {http_log_dir,none},
      {listener,[{port,15672}]},
      {load_definitions,none},
      {management_db_cache_multiplier,5},
      {process_stats_gc_timeout,300000},
      {stats_event_max_backlog,250}]},
 {rabbitmq_management_agent,
     [{rates_mode,basic},
      {sample_retention_policies,
          [{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
           {basic,[{605,5},{3600,60}]},
           {detailed,[{605,5}]}]}]},
 {rabbitmq_web_dispatch,[]},
 {ranch,[]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {ssl,[]},
 {stdlib,[]},
 {syntax_tools,[]},
 {xmerl,[]}]

 rabbitmqctl status

Status of node rabbit@chipp...

[{pid,20764},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.6.9"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.9"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.9"},

      {rabbit,"RabbitMQ","3.6.9"},

      {amqp_client,"RabbitMQ AMQP Client","3.6.9"},

      {rabbit_common,

          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",

          "3.6.9"},

      {compiler,"ERTS  CXC 138 10","7.0.4"},

      {cowboy,"Small, fast, modular HTTP server.","1.0.4"},

      {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},

      {ssl,"Erlang/OTP SSL application","8.1.1"},

      {public_key,"Public key infrastructure","1.4"},

      {cowlib,"Support library for manipulating Web protocols.","1.0.2"},

      {crypto,"CRYPTO","3.7.3"},

      {mnesia,"MNESIA  CXC 138 12","4.14.3"},

      {os_mon,"CPO  CXC 138 46","2.4.2"},

      {xmerl,"XML parser","1.3.13"},

      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},

      {syntax_tools,"Syntax tools","2.1.1"},

      {inets,"INETS  CXC 138 49","6.3.6"},

      {sasl,"SASL  CXC 138 11","3.0.3"},

      {stdlib,"ERTS  CXC 138 10","3.3"},

      {kernel,"ERTS  CXC 138 10","5.2"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang/OTP 19 [erts-8.3] [source-d5c06c6] [64-bit] [smp:8:8] [async-threads:128] [hipe] [kernel-poll:true]\n"},

 {memory,

     [{total,1414497584},

      {connection_readers,11462752},

      {connection_writers,1866200},

      {connection_channels,25246672},

      {connection_other,22396568},

      {queue_procs,1011424240},

      {queue_slave_procs,0},

      {plugins,15718072},

      {other_proc,24336008},

      {mnesia,223200},

      {metrics,2163472},

      {mgmt_db,24265776},

      {msg_index,282584},

      {other_ets,5048568},

      {binary,231739096},

      {code,24747054},

      {atom,1033401},

      {other_system,14704561}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},{amqp,5672,"192.168.1.224"},{http,15672,"::"}]},

 {vm_memory_high_watermark,0.6},

 {vm_memory_limit,10082237644},

 {disk_free_limit,50000000},

 {disk_free,65727135744},

 {file_descriptors,

     [{total_limit,32668},

      {total_used,365},

      {sockets_limit,29399},

      {sockets_used,353}]},

 {processes,[{limit,1048576},{used,4034}]},

 {run_queue,0},

 {uptime,66222},

 {kernel,{net_ticktime,60}}]

[dfed...@pivotal.io]

Hi,

It's hard to tell from the information provided. Do you have some errors logged in RabbitMQ log files? Can you find an erl_crash.dump file after a crash?
Open files limit looks suspicious, but it could be different for RabbitMQ user, it is displayed on management overview page.

[Nilo Menezes]

These settings seems to be correct. The server runs sometimes a week without any problems.

I attached a screenshot of the admin interface with the limits.

It is really hard for me too. Nothing suspicius in the log :-( It just stops with the segmentation fault.

Best Regards,

Nilo Menezes

[Michael Klishin]

Can you nonetheless post an hour or two of log contents before the dmesg timestamp

as well as dmesg entries for 5-10 minutes before a segmentation fault is killed?

RabbitMQ does not have native code, so switching to a different Erlang/OTP version, such as 19.3.6 (or 19.3.5, if you are on 19.3.6)

is the first thing I'd recommend.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[dfed...@pivotal.io]

erl_crash.dump slogan is erl_child_setup closed. erl_child_setup is a process, that sets up an external port program, for example during os:cmd. The process can be killed by the kernel (e.g. OOM killer). Can you check in system logs if erl_child_setup was killed?

[Michael Klishin]

…os:cmd/1 is used to retrieve system-wide information such as total amount

of memory available, free disk space, and so on.

I wonder if there can be a kernel or SELinux limit that can be reached by such child

processes every so often?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gergely Fábián]

Hi all,

I've also experienced a similar issue with RabbitMQ in our environment.
This happens in our integration environment as part of a standard step that we do right after installing RabbitMQ and then reconfiguring it (does not happen for each build, just sometimes).
We restart RabbitMQ, and Rabbit while stopping generates segmentation faults, and also logs some errors.

What we can see in the syslog:

Jun 29 21:24:29 localhost systemd[1]: Stopping RabbitMQ broker...
Jun 29 21:24:30 localhost rabbitmqctl[15131]: Stopping and halting node 'aos-rabbit@app1' ...
Jun 29 21:24:30 localhost rabbitmq-server[9106]: Gracefully halting Erlang VM
Jun 29 21:24:30 localhost rabbitmq-server[9106]: Failed to read from erl_child_setup: 104
Jun 29 21:24:30 localhost kernel: [  963.813049] show_signal_msg: 21 callbacks suppressed
Jun 29 21:24:30 localhost kernel: [  963.813054] 2_scheduler[9374]: segfault at 290 ip 00000000004adc5c sp 00007f95e4cfdab0 error 6 in beam.smp[400000+24e000]

In the RabbitMQ logs I can see:

=INFO REPORT==== 29-Jun-2017::21:16:48 ===
Setting permissions for '...' in '/' to ......

=INFO REPORT==== 29-Jun-2017::21:24:30 ===
Stopping RabbitMQ

=INFO REPORT==== 29-Jun-2017::21:24:30 ===
stopped SSL Listener on 10.143.243.4:5671

=INFO REPORT==== 29-Jun-2017::21:24:30 ===
stopped TCP Listener on 10.143.243.4:5672

=INFO REPORT==== 29-Jun-2017::21:24:30 ===
Stopped RabbitMQ application

=INFO REPORT==== 29-Jun-2017::21:24:30 ===
Halting Erlang VM


In our scenario this always happens while shutting down RabbitMQ. I'm supposing that erl_child_setup may be gone faster than RabbitMQ.

Is there a fix/workaround for this kind of issue?

Regards,

Gergely

On Tuesday, June 13, 2017 at 5:50:58 PM UTC+2, Michael Klishin wrote:

…os:cmd/1 is used to retrieve system-wide information such as total amount

of memory available, free disk space, and so on.

I wonder if there can be a kernel or SELinux limit that can be reached by such child

processes every so often?

On Tue, Jun 13, 2017 at 6:10 PM, <dfed...@pivotal.io> wrote:

erl_crash.dump slogan is erl_child_setup closed. erl_child_setup is a process, that sets up an external port program, for example during os:cmd. The process can be killed by the kernel (e.g. OOM killer). Can you check in system logs if erl_child_setup was killed?

On Tuesday, 13 June 2017 14:46:55 UTC+1, Nilo Menezes wrote:

These settings seems to be correct. The server runs sometimes a week without any problems.

I attached a screenshot of the admin interface with the limits.

It is really hard for me too. Nothing suspicius in the log :-( It just stops with the segmentation fault.

Best Regards,

Nilo Menezes

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gergely Fábián]

The environment is:
Ubuntu 16.04
Erlang 1:19.3-1
RabbitMQ: 3.6.6-1

Please let me know what further data should I provide.

Regards,

Gergely

[Gergely Fábián]

One more very important detail, that I forgot to mention. Only when this error appears at RabbitMQ shutdown we experience a jump in load (to about 31, for 8 cores).
This causes system restart due to set loadavg limits.

ulimit is 1024 set for RabbitMQ currently.

Rabbit seems to be using file descriptors under that limit:

=INFO REPORT==== 29-Jun-2017::21:16:42 ===
Limiting to approx 924 file handles (829 sockets)

Regards,

Gergely

[Michael Klishin]

RabbitMQ does not allocate memory directly, the Erlang runtime does.

So try a different version, e.g. 19.3.6 (20 is not currently supported).

I'm not sure how the open file handle limit can correlate with OS load average. If a node fails to allocate a file descriptor during shutdown, it would wait for one to become

available and it should not incur any CPU usage per se. A shutdown can cause a lot of

concurrent disk flush operations. Again, it's not a particularly CPU intensive operation, even though with a large number of queues it can easily keep all of the cores busy for some time. Ultimately it is much more likely that more time will be spent doing I/O.

Speaking of node shutdown, there's another reason to move to 19.3.6: it fixes a bug that could prevent RabbitMQ from shutting down if there were any open client connections (open TCP sockets) at the time of shutdown.

[Gergely Fábián]

Thank you for the feedback.

Unfortunately this can be reproduced even after upgrading to RabbitMQ 3.6.10 and Erlang 19.3.6.

Regards,

Gergely

[Michael Klishin]

Starting with 3.6.11 you will be able to try OTP 20.

That said, you are running with a very low open file handle limit which can result in

all kinds of weird issues, including node's inability to open a file, which

is particularly bad when stopping the node

(from an earlier provided log: "Jun 29 21:24:29 localhost systemd[1]: Stopping RabbitMQ broker...").

Please bump it:

http://www.rabbitmq.com/networking.html

http://www.rabbitmq.com/production-checklist.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gergely Fábián]

We increased the limit of the file descriptors but we still can reproduce this with: RabbitMQ 3.6.10 on Erlang 19.3.6

The limits look the following:

$ sudo rabbitmqctl status
...
{file_descriptors,
[{total_limit,31900},
{total_used,7},
{sockets_limit,28708},
{sockets_used,5}]},

Is there any other option but trying Erlang 20 with RabbitMQ 3.6.11?

[gl...@pivotal.io]

Can you enable core dumps and share it when this happens next? We will also need the beam.smp binary that was used to run RabbitMQ.

* http://man7.org/linux/man-pages/man5/core.5.html

* https://stackoverflow.com/questions/6152232/how-to-generate-core-dump-file-in-ubuntu

[Gergely Fábián]

We turned on crash dumping, however when the segfault happens, there is no crash dump generated:

Sep  1 10:49:11 localhost systemd[1]: Stopping RabbitMQ broker...
Sep  1 10:49:12 localhost rabbitmqctl[25300]: Stopping and halting node 'aos-rabbit@...'
Sep  1 10:49:12 localhost rabbitmq-server[24847]: Gracefully halting Erlang VM
Sep  1 10:49:12 localhost kernel: [  714.344680] show_signal_msg: 21 callbacks suppressed
Sep  1 10:49:12 localhost kernel: [  714.344685] 2_scheduler[25130]: segfault at 2e8 ip 00000000004c7dae sp 00007f927963dab0 error 6 in beam.smp[400000+2cb000]
Sep  1 10:49:12 localhost rabbitmq-server[24847]: Failed to read from
Sep  1 10:49:12 localhost systemd[1]: rabbitmq-server.service: Main process exited, code=killed, status=11/SEGV
Sep  1 10:49:12 localhost systemd[1]: Stopped RabbitMQ broker.
Sep  1 10:49:12 localhost systemd[1]: rabbitmq-server.service: Unit entered failed state.
Sep  1 10:49:12 localhost systemd[1]: rabbitmq-server.service: Failed with result 'signal'.
Sep  1 10:49:14 localhost systemd[1]: Starting RabbitMQ broker...

Also, the load does not jump any more, and the VM is not restarted... It's a bit strange, but it seems turning on crash dumping solved our problems?

For a comparison that in general crash dumps are generated, a previous RabbitMQ shutdown:

Sep  1 10:42:19 localhost systemd[1]: Stopping RabbitMQ broker...
Sep  1 10:42:20 localhost rabbitmqctl[7913]: Stopping and halting node rabbit@...
Sep  1 10:42:20 localhost rabbitmq-server[6884]: Gracefully halting Erlang VM
Sep  1 10:42:20 localhost rabbitmq-server[6884]: erl_child_setup closed
Sep  1 10:42:20 localhost rabbitmq-server[6884]: #015
Sep  1 10:42:22 localhost rabbitmq-server[6884]: Crash dump is being written to: erl_crash.dump...done
Sep  1 10:42:22 localhost systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Sep  1 10:42:22 localhost systemd[1]: Stopped RabbitMQ broker.
Sep  1 10:42:22 localhost systemd[1]: rabbitmq-server.service: Unit entered failed state.
Sep  1 10:42:22 localhost systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Sep  1 10:42:23 localhost systemd[1]: Starting RabbitMQ broker...

Another question is why is a normal shutdown causing RabbitMQ enter into a failed state?

[Michael Klishin]

We reported at least two issues to the Erlang/OTP team that prevented nodes

from shutting down. Please move to 19.3.6.2. We are not aware of any Erlang VM shutdown issues

with it.

What segfaults is the Erlang VM. RabbitMQ doesn't contain any native code and does not allocate, reference

or release memory in its own code.

systemd service state can be a package issue, we keep going back and forth around a few

things around how the service is started. There's not enough information to tell for sure.

[Michael Klishin]

Here's the kind of script tweaks I was referring to:

https://github.com/rabbitmq/rabbitmq-server-release/commit/aa84ddcf36b67e0196b5cfa81bb34008bdf70a43.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad MacGregor]

With the current version of rabbitmq-server distributed with Ubuntu 18.04 (rabbitmq-server version 3.6.10), I saw 'sudo service rabbitmq-server stop' would take longer than a minute. I upgraded to the newest version (currently rabbitmq-server 3.7.8) using this walkthrough and now it restarts in about 6 seconds. I would recommend this for people seeing the same issue as me.

[Michael Klishin]

Thank you for sharing your findings, Brad.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.