RabbitMQ - BOOT FAILED

Saurav Bansal

unread,

Dec 28, 2021, 7:43:05 AM12/28/21

to rabbitmq-users

rabbitmq 11:53:44.82

rabbitmq 11:53:44.83 Welcome to the Bitnami rabbitmq container

rabbitmq 11:53:44.83 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq

rabbitmq 11:53:44.83 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues

rabbitmq 11:53:44.83

rabbitmq 11:53:44.84 INFO ==> ** Starting RabbitMQ setup **

rabbitmq 11:53:44.86 INFO ==> Validating settings in RABBITMQ_* env vars..

rabbitmq 11:53:44.88 INFO ==> Initializing RabbitMQ...

rabbitmq 11:53:44.92 INFO ==> Persisted data detected. Restoring...

rabbitmq 11:53:44.93 INFO ==> ** RabbitMQ setup finished! **

rabbitmq 11:53:44.95 INFO ==> ** Starting RabbitMQ **

2021-12-28 11:53:47.777776+00:00 [info] <0.226.0> Feature flags: list of feature flags found:

2021-12-28 11:53:47.790072+00:00 [info] <0.226.0> Feature flags: [x] implicit_default_bindings

2021-12-28 11:53:47.790207+00:00 [info] <0.226.0> Feature flags: [x] maintenance_mode_status

2021-12-28 11:53:47.790242+00:00 [info] <0.226.0> Feature flags: [x] quorum_queue

2021-12-28 11:53:47.790259+00:00 [info] <0.226.0> Feature flags: [ ] stream_queue

2021-12-28 11:53:47.790352+00:00 [info] <0.226.0> Feature flags: [x] user_limits

2021-12-28 11:53:47.790374+00:00 [info] <0.226.0> Feature flags: [x] virtual_host_metadata

2021-12-28 11:53:47.790390+00:00 [info] <0.226.0> Feature flags: feature flag states written to disk: yes

2021-12-28 11:53:48.126226+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped

2021-12-28 11:53:48.126330+00:00 [noti] <0.226.0> Logging: switching to configured handler(s); following messages may not be visible in this log output

2021-12-28 11:53:48.140860+00:00 [noti] <0.226.0> Logging: configured log handlers are now ACTIVE

2021-12-28 11:53:50.768349+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped

2021-12-28 11:53:50.770574+00:00 [info] <0.226.0> Feature flags: list of feature flags found:

2021-12-28 11:53:50.770667+00:00 [info] <0.226.0> Feature flags: [x] drop_unroutable_metric

2021-12-28 11:53:50.770740+00:00 [info] <0.226.0> Feature flags: [x] empty_basic_get_metric

2021-12-28 11:53:50.770788+00:00 [info] <0.226.0> Feature flags: [x] implicit_default_bindings

2021-12-28 11:53:50.770834+00:00 [info] <0.226.0> Feature flags: [x] maintenance_mode_status

2021-12-28 11:53:50.770922+00:00 [info] <0.226.0> Feature flags: [x] quorum_queue

2021-12-28 11:53:50.770960+00:00 [info] <0.226.0> Feature flags: [ ] stream_queue

2021-12-28 11:53:50.770985+00:00 [info] <0.226.0> Feature flags: [x] user_limits

2021-12-28 11:53:50.771014+00:00 [info] <0.226.0> Feature flags: [x] virtual_host_metadata

2021-12-28 11:53:50.771070+00:00 [info] <0.226.0> Feature flags: feature flag states written to disk: yes

2021-12-28 11:53:50.849298+00:00 [info] <0.226.0> ra: starting system quorum_queues

2021-12-28 11:53:50.849473+00:00 [info] <0.226.0> starting Ra system: quorum_queues in directory: /bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local

2021-12-28 11:53:50.865644+00:00 [info] <0.348.0> ra: meta data store initialised for system quorum_queues. 8 record(s) recovered

2021-12-28 11:53:50.866070+00:00 [noti] <0.358.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> crasher:

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> initial call: ra_log_wal:init/1

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> pid: <0.358.0>

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> registered_name: []

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> exception exit: wal_checksum_validation_failure

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> in function gen_batch_server:init_it/6 (src/gen_batch_server.erl, line 182)

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> ancestors: [<0.357.0>,ra_log_sup,<0.327.0>,ra_systems_sup,ra_sup,

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> <0.191.0>]

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> message_queue_len: 0

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> messages: []

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> links: [<0.357.0>]

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> dictionary: []

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> trap_exit: true

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> status: running

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> heap_size: 6772

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> stack_size: 29

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> reductions: 298640

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> neighbours:

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> supervisor: {<0.357.0>,ra_log_wal_sup}

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> errorContext: start_error

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> reason: wal_checksum_validation_failure

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> offender: [{pid,undefined},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {id,ra_log_wal},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {mfargs,{ra_log_wal,start_link,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> [#{compute_checksums => true,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> dir =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> hibernate_after => undefined,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> max_batch_size => 4096,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> max_entries => undefined,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> max_size_bytes => 536870912,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> name => ra_log_wal,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> names =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> #{closed_mem_tbls =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> ra_log_closed_mem_tables,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> directory => ra_directory,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> directory_rev =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> ra_directory_reverse,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> log_ets => ra_log_ets,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> log_meta => ra_log_meta,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> log_sup => ra_log_sup,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> open_mem_tbls =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> ra_log_open_mem_tables,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> segment_writer =>

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> ra_log_segment_writer,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> server_sup => ra_server_sup_sup,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> wal => ra_log_wal,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> wal_sup => ra_log_wal_sup},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> segment_writer => ra_log_segment_writer,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> sync_method => datasync,

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> write_strategy => default}]}},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {restart_type,permanent},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {significant,false},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {shutdown,5000},

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0> {child_type,worker}]

2021-12-28 11:53:50.936792+00:00 [erro] <0.357.0>

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> supervisor: {local,ra_log_sup}

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> errorContext: start_error

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> reason: {shutdown,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {failed_to_start_child,ra_log_wal,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> wal_checksum_validation_failure}}

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> offender: [{pid,undefined},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {id,ra_log_wal_sup},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {mfargs,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {ra_log_wal_sup,start_link,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> [#{compute_checksums => true,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> dir =>

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> hibernate_after => undefined,max_batch_size => 4096,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> max_entries => undefined,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> max_size_bytes => 536870912,name => ra_log_wal,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> names =>

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> #{closed_mem_tbls => ra_log_closed_mem_tables,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> directory => ra_directory,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> directory_rev => ra_directory_reverse,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> log_ets => ra_log_ets,log_meta => ra_log_meta,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> log_sup => ra_log_sup,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> open_mem_tbls => ra_log_open_mem_tables,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> segment_writer => ra_log_segment_writer,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> server_sup => ra_server_sup_sup,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> wal => ra_log_wal,wal_sup => ra_log_wal_sup},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> segment_writer => ra_log_segment_writer,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> sync_method => datasync,

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> write_strategy => default}]}},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {restart_type,permanent},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {significant,false},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {shutdown,infinity},

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0> {child_type,supervisor}]

2021-12-28 11:53:50.937746+00:00 [erro] <0.331.0>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> supervisor: {<0.327.0>,ra_system_sup}

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> errorContext: start_error

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> reason: {shutdown,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {failed_to_start_child,ra_log_wal_sup,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {shutdown,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {failed_to_start_child,ra_log_wal,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_checksum_validation_failure}}}}

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> offender: [{pid,undefined},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {id,ra_log_sup},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {mfargs,{ra_log_sup,start_link,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> [#{data_dir =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> name => quorum_queues,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> names =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> #{closed_mem_tbls =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> ra_log_closed_mem_tables,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> directory => ra_directory,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> directory_rev =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> ra_directory_reverse,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> log_ets => ra_log_ets,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> log_meta => ra_log_meta,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> log_sup => ra_log_sup,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> open_mem_tbls =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> ra_log_open_mem_tables,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> segment_writer =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> ra_log_segment_writer,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> server_sup => ra_server_sup_sup,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal => ra_log_wal,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_sup => ra_log_wal_sup},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> segment_max_entries => 4096,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_compute_checksums => true,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_data_dir =>

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_max_batch_size => 4096,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_max_entries => undefined,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_max_size_bytes => 536870912,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_sync_method => datasync,

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> wal_write_strategy => default}]}},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {restart_type,permanent},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {significant,false},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {shutdown,infinity},

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0> {child_type,supervisor}]

2021-12-28 11:53:50.939198+00:00 [erro] <0.327.0>

2021-12-28 11:53:50.950083+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0>

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> BOOT FAILED

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> ===========

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> Exception during startup:

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0>

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,wal_checksum_validation_failure}}}}}},{child,undefined,quorum_queues,{ra_system_sup,start_link,[#{data_dir => "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",name => quorum_queues,names => #{closed_mem_tbls => ra_log_closed_mem_tables,directory => ra_directory,directory_rev => ra_directory_reverse,log_ets => ra_log_ets,log_meta => ra_log_meta,log_sup => ra_log_sup,open_mem_tbls => ra_log_open_mem_tables,segment_writer => ra_log_segment_writer,server_sup => ra_server_sup_sup,wal => ra_log_wal,wal_sup => ra_log_wal_sup},segment_max_entries => 4096,wal_compute_checksums => true,wal_data_dir => "/bitnami/rabbitmq/mnesia/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local/quorum/rab...@rabbitmq-2.rabbitmq-headless.jx-production.svc.cluster.local",wal_max_batch_size => 4096,wal_max_entries => undefined,wal_max_size_bytes => 536870912,wal_sync_method => datasync,wal_write_strategy => default}]},permanent,false,infinity,supervisor,[ra_system_sup]}}}}

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0>

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> rabbit:run_prelaunch_second_phase/0, line 386

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> rabbit:start/2, line 847

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0> application_master:start_it_old/4, line 293

2021-12-28 11:53:50.950640+00:00 [erro] <0.226.0>

BOOT FAILED

Wes Peng

unread,

Dec 28, 2021, 8:24:15 AM12/28/21

to rabbitm...@googlegroups.com

As the starting info shows, you may want to submit an issue on their support community.

rabbitmq 11:53:44.83 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues

Thanks

Luke Bakken

unread,

Dec 28, 2021, 10:28:15 AM12/28/21

to rabbitmq-users

Hello,

Please help us help you. Pasting a stack trace into a message without any other information is not very helpful.

What version of RabbitMQ and Erlang are you using?
What were you doing when this exception happened? Was this a normal reboot of the container, did you bring up a new container, etc?
Is this the first time you have seen the wal_checksum_validation_failure issue?

Thanks -

Luke

On Tuesday, December 28, 2021 at 4:43:05 AM UTC-8 saurav...@galaxycard.in wrote:

rabbitmq 11:53:44.82

rabbitmq 11:53:44.83 Welcome to the Bitnami rabbitmq container

rabbitmq 11:53:44.83 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq

rabbitmq 11:53:44.83 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues

2021-12-28 11:53:50.937228+00:00 [erro] <0.358.0> exception exit: wal_checksum_validation_failure

Saurav Bansal

unread,

Dec 28, 2021, 1:32:01 PM12/28/21

to rabbitmq-users

Hi,

What version of RabbitMQ and Erlang are you using?

- RabbitMQ 3.9.8
- Erlang 24.1.2

What were you doing when this exception happened? Was this a normal reboot of the container, did you bring up a new container, etc?

- rabbitMQ service is defined in kubernetes cluster. Above is the helm chart of the cluster service. Kubernates pods getting down now and then but cluster restarts them automatically. For last 1 day whenever any pod gets restart it stuck with above logs. For RabbitMQ we are using GCP with Kube Cluster, pd-ssd PV. We are using rabbitMQ with quorum queues.

Is this the first time you have seen the wal_checksum_validation_failure issue?

- This is the 1st time that I have seen this message.

Thanks,

Saaurav Bansal

Reply all

Reply to author

Forward