RabbitMQ Crash wal_checksum_validation_failure

263 views

Skip to first unread message

rajiv chodisetti

unread,

Mar 8, 2022, 12:04:45 AM3/8/22

to rabbitmq-users

Hi Everyone,

We recently had a rabbitMQ crash with this error wal_checksum_validation_failure in one of our internal customers environment and here is the pod log for reference.

And to resolve this issue, I had to delete the PVC and recreate the rabbitMQ pods which resulted in loss of data.

So, can someone please let us know what scenario could lead this and the steps we can follow to mitigate this without loss of data and the possible ways to prevent this ? Thanks

RabbitMQ Operator Version: 1.5.0

Kubernetes Version: RKE2 v1.21.4+rke2r2

RabbitMQ is backed by: Longhorn PVC's

2022-03-03 14:43:48.233844+00:00 [warn] <0.130.0> cluster_formation.randomized_startup_delay_range.min and cluster_formation.randomized_startup_delay_range.max are deprecated
2022-03-03 14:43:49.753786+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
2022-03-03 14:43:49.753857+00:00 [info] <0.222.0> Feature flags: [x] implicit_default_bindings
2022-03-03 14:43:49.753888+00:00 [info] <0.222.0> Feature flags: [x] maintenance_mode_status
2022-03-03 14:43:49.753904+00:00 [info] <0.222.0> Feature flags: [x] quorum_queue
2022-03-03 14:43:49.754007+00:00 [info] <0.222.0> Feature flags: [x] stream_queue
2022-03-03 14:43:49.754030+00:00 [info] <0.222.0> Feature flags: [x] user_limits
2022-03-03 14:43:49.754044+00:00 [info] <0.222.0> Feature flags: [x] virtual_host_metadata
2022-03-03 14:43:49.754114+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
2022-03-03 14:43:50.162602+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
2022-03-03 14:43:50.162720+00:00 [noti] <0.222.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2022-03-03 14:43:50.176442+00:00 [noti] <0.222.0> Logging: configured log handlers are now ACTIVE
2022-03-03 14:43:52.994268+00:00 [info] <0.222.0> ra: starting system quorum_queues
2022-03-03 14:43:52.994391+00:00 [info] <0.222.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq
2022-03-03 14:43:53.023103+00:00 [info] <0.322.0> ra: meta data store initialised for system quorum_queues. 50 record(s) recovered
2022-03-03 14:43:53.023375+00:00 [noti] <0.327.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> crasher:
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> initial call: ra_log_wal:init/1
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> pid: <0.327.0>
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> registered_name: []
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> exception exit: wal_checksum_validation_failure
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> in function gen_batch_server:init_it/6 (src/gen_batch_server.erl, line 182)
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> ancestors: [<0.326.0>,ra_log_sup,<0.275.0>,ra_systems_sup,ra_sup,
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> <0.187.0>]
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> message_queue_len: 0
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> messages: []
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> links: [<0.326.0>]
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> dictionary: []
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> trap_exit: true
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> status: running
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> heap_size: 6772
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> stack_size: 29
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> reductions: 5188619
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0> neighbours:
2022-03-03 14:43:53.240295+00:00 [erro] <0.327.0>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> supervisor: {<0.326.0>,ra_log_wal_sup}
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> errorContext: start_error
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> reason: wal_checksum_validation_failure
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> offender: [{pid,undefined},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {id,ra_log_wal},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {mfargs,{ra_log_wal,start_link,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> [#{compute_checksums => true,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> dir =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> hibernate_after => undefined,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> max_batch_size => 4096,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> max_entries => undefined,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> max_size_bytes => 536870912,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> name => ra_log_wal,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> names =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> #{closed_mem_tbls =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> ra_log_closed_mem_tables,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> directory => ra_directory,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> directory_rev =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> ra_directory_reverse,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> log_ets => ra_log_ets,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> log_meta => ra_log_meta,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> log_sup => ra_log_sup,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> open_mem_tbls =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> ra_log_open_mem_tables,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> segment_writer =>
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> ra_log_segment_writer,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> server_sup => ra_server_sup_sup,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> wal => ra_log_wal,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> wal_sup => ra_log_wal_sup},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> segment_writer => ra_log_segment_writer,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> sync_method => datasync,
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> write_strategy => default}]}},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {restart_type,permanent},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {significant,false},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {shutdown,5000},
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0> {child_type,worker}]
2022-03-03 14:43:53.240038+00:00 [erro] <0.326.0>
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> supervisor: {local,ra_log_sup}
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> errorContext: start_error
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> reason: {shutdown,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {failed_to_start_child,ra_log_wal,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> wal_checksum_validation_failure}}
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> offender: [{pid,undefined},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {id,ra_log_wal_sup},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {mfargs,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {ra_log_wal_sup,start_link,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> [#{compute_checksums => true,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> dir =>
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> hibernate_after => undefined,max_batch_size => 4096,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> max_entries => undefined,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> max_size_bytes => 536870912,name => ra_log_wal,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> names =>
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> #{closed_mem_tbls => ra_log_closed_mem_tables,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> directory => ra_directory,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> directory_rev => ra_directory_reverse,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> log_ets => ra_log_ets,log_meta => ra_log_meta,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> log_sup => ra_log_sup,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> open_mem_tbls => ra_log_open_mem_tables,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> segment_writer => ra_log_segment_writer,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> server_sup => ra_server_sup_sup,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> wal => ra_log_wal,wal_sup => ra_log_wal_sup},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> segment_writer => ra_log_segment_writer,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> sync_method => datasync,
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> write_strategy => default}]}},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {restart_type,permanent},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {significant,false},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {shutdown,infinity},
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0> {child_type,supervisor}]
2022-03-03 14:43:53.240959+00:00 [erro] <0.286.0>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> supervisor: {<0.275.0>,ra_system_sup}
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> errorContext: start_error
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> reason: {shutdown,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {failed_to_start_child,ra_log_wal_sup,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {shutdown,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {failed_to_start_child,ra_log_wal,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_checksum_validation_failure}}}}
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> offender: [{pid,undefined},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {id,ra_log_sup},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {mfargs,{ra_log_sup,start_link,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> [#{data_dir =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> name => quorum_queues,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> names =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> #{closed_mem_tbls =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> ra_log_closed_mem_tables,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> directory => ra_directory,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> directory_rev =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> ra_directory_reverse,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> log_ets => ra_log_ets,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> log_meta => ra_log_meta,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> log_sup => ra_log_sup,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> open_mem_tbls =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> ra_log_open_mem_tables,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> segment_writer =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> ra_log_segment_writer,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> server_sup => ra_server_sup_sup,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal => ra_log_wal,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_sup => ra_log_wal_sup},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> segment_max_entries => 4096,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_compute_checksums => true,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_data_dir =>
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_max_batch_size => 4096,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_max_entries => undefined,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_max_size_bytes => 536870912,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_sync_method => datasync,
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> wal_write_strategy => default}]}},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {restart_type,permanent},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {significant,false},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {shutdown,infinity},
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0> {child_type,supervisor}]
2022-03-03 14:43:53.242041+00:00 [erro] <0.275.0>
2022-03-03 14:43:53.268030+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped

BOOT FAILED
===========
Exception during startup:

2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0>
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> BOOT FAILED
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> ===========
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> Exception during startup:
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0>
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",wal_max_batch_size => 4096,wal_max_entries => undefined,wal_max_size_bytes => 536870912,wal_sync_method => datasync,wal_write_strategy => default}]},permanent,false,infinity,supervisor,[ra_system_sup]}}}}
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0>
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> rabbit:run_prelaunch_second_phase/0, line 386
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> rabbit:start/2, line 847
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0> application_master:start_it_old/4, line 293
2022-03-03 14:43:53.268531+00:00 [erro] <0.222.0>
error:{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",wal_max_batch_size => 4096,wal_max_entries => undefined,wal_max_size_bytes => 536870912,wal_sync_method => datasync,wal_write_strategy => default}]},permanent,false,infinity,supervisor,[ra_system_sup]}}}}

rabbit:run_prelaunch_second_phase/0, line 386
rabbit:start/2, line 847
application_master:start_it_old/4, line 293

2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> crasher:
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> initial call: application_master:init/4
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> pid: <0.221.0>
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> registered_name: []
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> exception exit: {{badmatch,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {error,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {{shutdown,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {failed_to_start_child,ra_log_sup,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {shutdown,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {failed_to_start_child,ra_log_wal_sup,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {shutdown,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {failed_to_start_child,ra_log_wal,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_checksum_validation_failure}}}}}},
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {child,undefined,quorum_queues,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {ra_system_sup,start_link,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> [#{data_dir =>
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> name => quorum_queues,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> names =>
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> #{closed_mem_tbls => ra_log_closed_mem_tables,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> directory => ra_directory,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> directory_rev => ra_directory_reverse,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> log_ets => ra_log_ets,log_meta => ra_log_meta,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> log_sup => ra_log_sup,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> open_mem_tbls => ra_log_open_mem_tables,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> segment_writer => ra_log_segment_writer,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> server_sup => ra_server_sup_sup,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal => ra_log_wal,wal_sup => ra_log_wal_sup},
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> segment_max_entries => 4096,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_compute_checksums => true,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_data_dir =>
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_max_batch_size => 4096,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_max_entries => undefined,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_max_size_bytes => 536870912,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_sync_method => datasync,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> wal_write_strategy => default}]},
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> permanent,false,infinity,supervisor,
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> [ra_system_sup]}}}},
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> {rabbit,start,[normal,[]]}}
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> in function application_master:init/4 (application_master.erl, line 142)
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> ancestors: [<0.220.0>]
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> message_queue_len: 1
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> messages: [{'EXIT',<0.222.0>,normal}]
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> links: [<0.220.0>,<0.44.0>]
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> dictionary: []
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> trap_exit: true
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> status: running
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> heap_size: 1598
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> stack_size: 29
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> reductions: 156
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0> neighbours:
2022-03-03 14:43:54.270189+00:00 [erro] <0.221.0>
2022-03-03 14:43:54.271183+00:00 [noti] <0.44.0> Application rabbit exited with reason: {{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",wal_max_batch_size => 4096,wal_max_entries => undefined,wal_max_size_bytes => 536870912,wal_sync_method => datasync,wal_write_strategy => default}]},permanent,false,infinity,supervisor,[ra_system_sup]}}}},{rabbit,start,[normal,[]]}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => \"/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq\",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => \"/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq\",wal_max_batch_size => 4096,wal_max_entries => undefined,wal_max_size_bytes => 536870912,wal_sync_method => datasync,wal_write_strategy => default}]},permanent,false,infinity,supervisor,[ra_system_sup]}}}},{rabbit,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => "/var/lib/rabbitmq/mnesia/rab...@rabbitmq-server-0.rabbitmq-nodes.rabbitmq/quorum/rabbit@rabbitmq-server

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Reply all

Reply to author

Forward

0 new messages