Need help deciphering badmatch crashes

104 views
Skip to first unread message

Matt Fiocca

unread,
Jun 30, 2021, 11:52:01 AM6/30/21
to rabbitmq-users
I apologize if this was discussed already; I've tried digging for keywords without success.

Upon installing a fresh server, everything appears to work ok but I am getting these erlang crash reports every 10 seconds in the logs:

/var/log/rabbitmq/log/crash.log:

2021-06-30 14:42:16 =CRASH REPORT====
crasher:
initial call: rabbit_reader:init/3
pid: <0.17594.13>
registered_name: []
exception error: {
{badmatch,#{command => local,version => 2}},
[
{rabbit_net,socket_ends,2,[{file,"src/rabbit_net.erl"},{line,237}]},
{rabbit_net,connection_string,2,[{file,"src/rabbit_net.erl"},{line,209}]},
{rabbit_reader,start_connection,4,[{file,"src/rabbit_reader.erl"},{line,310}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}
]
}
ancestors: [<0.17592.13>,<0.13026.13>,<0.13025.13>,<0.13024.13>,<0.13022.13>,<0.13021.13>,rabbit_sup,<0.12394.13>]
message_queue_len: 0
messages: []
links: [<0.17592.13>,#Port<0.73211>]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 29
reductions: 501
neighbours:
2021-06-30 14:42:16 =SUPERVISOR REPORT====
Supervisor: {<0.17592.13>,rabbit_connection_sup}
Context: child_terminated
Reason: {
{badmatch,#{command => local,version => 2}},
[
{rabbit_net,socket_ends,2,[{file,"src/rabbit_net.erl"},{line,237}]},
{rabbit_net,connection_string,2,[{file,"src/rabbit_net.erl"},{line,209}]},
{rabbit_reader,start_connection,4,[{file,"src/rabbit_reader.erl"},{line,310}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}
]
}
Offender: [{pid,<0.17594.13>},{id,reader},{mfargs,{rabbit_reader,start_link,[<0.17593.13>,{acceptor,{0,0,0,0,0,0,0,0},5672}]}},{restart_type,intrinsic},{shutdown,30000},{child_type,worker}]
2021-06-30 14:42:16 =SUPERVISOR REPORT====
Supervisor: {<0.17592.13>,rabbit_connection_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [
{pid,<0.17594.13>},
{id,reader},
{mfargs,{rabbit_reader,start_link,[<0.17593.13>,{acceptor,{0,0,0,0,0,0,0,0},5672}]}},
{restart_type,intrinsic},
{shutdown,30000},
{child_type,worker}
]

Clients can connect and messages flow just fine, and i wouldn't have even known there was an issue had I not tailed the logs. I just need help in reading these reports so that I know what's going on. 

Environment: 
  • AWS EC2, r5a.large, ubuntu 20.04, 200GB EBS
  • cloudsmith quick start installation (https://www.rabbitmq.com/install-debian.html#apt-quick-start-cloudsmith)
  • RabbitMQ 3.8.18
  • Erlang configuration: Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
  • Enabled Plugins:
    rabbitmq_auth_mechanism_ssl
    rabbitmq_management
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
If anyone has seen this before, I'd really appreciate some insight.

Thank you

Matt Fiocca

unread,
Jun 30, 2021, 1:18:15 PM6/30/21
to rabbitmq-users
I forgot to mention that my server is behind an AWS Load Balancer, with proxy protocol enabled. These reports seem to line up with health checks. They go away when deregistering the node from the LB.

Matt Fiocca

unread,
Jun 30, 2021, 1:26:20 PM6/30/21
to rabbitmq-users
... logs disappear after disabling proxy protocol. 

Matt Fiocca

unread,
Jun 30, 2021, 1:42:09 PM6/30/21
to rabbitmq-users
ok, this all cleared up now. I had my ELB health checks configured to ping AMQP 5672. Proxy Protocol was configured for this port, so the health checks weren't being sent over correctly from the ELB. After changing the health check port to something like the management port (15672), everything is working fine now, including proxy protocol.
Reply all
Reply to author
Forward
0 new messages