RabbitMQ fails to start "no match of right hand value undefined in rabbit_priority_queue:enable/0"

[Roman O]

Hi,

RabbitMQ 3.7.4 running withing docker container (run from official docker image) fails to start  with the below error:

The issue happens randomly. 

2019-07-03 17:08:17.859 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.634 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.635 [info] <0.33.0> Application recon started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.635 [info] <0.33.0> Application jsx started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.649 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.649 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.936 [info] <0.33.0> Application inets started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.937 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.937 [info] <0.33.0> Application crypto started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:19.937 [info] <0.33.0> Application public_key started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:20.172 [info] <0.33.0> Application ssl started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:20.178 [info] <0.33.0> Application ranch started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:20.178 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:20.178 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:20.211 [info] <0.187.0>
 Starting RabbitMQ 3.7.4 on Erlang 20.3.4
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/
epic_rabbitmq.0.ovt2eij5rr0r@im-pla-slave-1    |
  ##  ##
  ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>
epic_rabbitmq.0.ovt2eij5rr0r@im-pla-slave-1    |
              Starting broker...
2019-07-03 17:08:20.231 [info] <0.187.0>
 node           : rabbit@rabbit-im-pla-slave-1
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/paasrabbitmq/paasrabbitmq.conf
 cookie hash    : Sqvm1ja/eeUKv+fi/FfMKg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1
2019-07-03 17:08:24.188 [info] <0.199.0> Memory high watermark set to 12867 MiB (13492759756 bytes) of 32169 MiB (33731899392 bytes) total
2019-07-03 17:08:24.197 [info] <0.201.0> Enabling free disk space monitoring
2019-07-03 17:08:24.197 [info] <0.201.0> Disk free limit set to 50MB
2019-07-03 17:08:24.208 [info] <0.204.0> FHC read buffering:  OFF
2019-07-03 17:08:24.208 [info] <0.204.0> FHC write buffering: ON
2019-07-03 17:08:24.208 [info] <0.203.0> Limiting to approx 65436 file handles (58890 sockets)
2019-07-03 17:08:24.209 [info] <0.187.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2019-07-03 17:08:24.209 [info] <0.187.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2019-07-03 17:08:24.210 [info] <0.187.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2019-07-03 17:08:24.210 [info] <0.187.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-07-03 17:08:24.210 [info] <0.187.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2019-07-03 17:08:24.210 [info] <0.187.0> All discovered existing cluster peers:
2019-07-03 17:08:24.210 [info] <0.187.0> Discovered no peer nodes to cluster with
2019-07-03 17:08:24.213 [info] <0.33.0> Application mnesia exited with reason: stopped
2019-07-03 17:08:24.700 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:08:25.845 [info] <0.187.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-03 17:08:25.879 [info] <0.187.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-03 17:08:26.072 [info] <0.187.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-03 17:08:26.072 [info] <0.187.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2019-07-03 17:08:26.073 [error] <0.186.0> CRASH REPORT Process <0.186.0> with 0 neighbours exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74 in application_master:init/4 line 134
2019-07-03 17:08:26.074 [info] <0.33.0> Application rabbit exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,\"src/rabbit_priority_queue.erl\"},{line,74}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,793}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,273}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,"src/rab
epic_rabbitmq.0.ovt2eij5rr0r@im-pla-slave-1    |
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

After restarting the container the following errors pop:

2019-07-03 17:09:21.171 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:09:25.718 [info] <0.5.0> upgrades: Backing up mnesia dir to "/var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup"
2019-07-03 17:09:25.731 [info] <0.5.0> upgrades: Mnesia dir backed up to "/var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup"
2019-07-03 17:09:25.739 [info] <0.5.0> mnesia upgrades: 39 to apply
2019-07-03 17:09:25.828 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:09:25.836 [info] <0.5.0> mnesia upgrades: Applying rabbit_upgrade_functions:topic_trie_node
2019-07-03 17:09:25.836 [error] <0.5.0>
Error description:
    init:start_em/1
    rabbit:start_it/1 line 445
    rabbit:'-boot/0-fun-0-'/0 line 296
    rabbit_upgrade:primary_upgrade/2 line 218
    rabbit_upgrade:apply_upgrades/3 line 287
    rabbit_upgrade:'-apply_upgrades/3-lc$^0/1-0-'/2 line 287
    rabbit_upgrade:apply_upgrade/2 line 294
    rabbit_upgrade_functions:create/2 line 626
error:{badmatch,{aborted,{already_exists,rabbit_topic_trie_node}}}
Log file(s) (may contain more information):
   <stdout>
epic_rabbitmq.0.d93wu4coniaw@im-pla-slave-1    |
BOOT FAILED
===========
epic_rabbitmq.0.d93wu4coniaw@im-pla-slave-1    |
Error description:
    init:start_em/1
    rabbit:start_it/1 line 445
    rabbit:'-boot/0-fun-0-'/0 line 296
    rabbit_upgrade:primary_upgrade/2 line 218
    rabbit_upgrade:apply_upgrades/3 line 287
    rabbit_upgrade:'-apply_upgrades/3-lc$^0/1-0-'/2 line 287
    rabbit_upgrade:apply_upgrade/2 line 294
    rabbit_upgrade_functions:create/2 line 626
error:{badmatch,{aborted,{already_exists,rabbit_topic_trie_node}}}
Log file(s) (may contain more information):
   <stdout>
epic_rabbitmq.0.d93wu4coniaw@im-pla-slave-1    |
{"init terminating in do_boot",{badmatch,{aborted,{already_exists,rabbit_topic_trie_node}}}}
init terminating in do_boot ({badmatch,{aborted,{already_exists,rabbit_topic_trie_node}}})
epic_rabbitmq.0.d93wu4coniaw@im-pla-slave-1    |
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

After 3rd restart the following error pop and it pops again on subsequent restarts unless rabbitmq data is deleted and then RabbitMQ starts successfully (unless the the first error happens again and so on)

2019-07-03 17:10:17.641 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-im-pla-slave-1'
2019-07-03 17:10:23.036 [error] <0.5.0> Found lock file at /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1/schema_upgrade_lock.
            Either previous upgrade is in progress or has failed.
            Database backup path: /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup
2019-07-03 17:10:23.037 [error] <0.5.0>
Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 445
    rabbit:'-boot/0-fun-0-'/0 line 296
    rabbit_upgrade:maybe_upgrade_mnesia/0 line 148
    rabbit_upgrade:ensure_backup_taken/0 line 106
throw:{error,previous_upgrade_failed}
Log file(s) (may contain more information):
   <stdout>
epic_rabbitmq.0.jh3ntyb6rtkd@im-pla-slave-1    |
BOOT FAILED
===========
epic_rabbitmq.0.jh3ntyb6rtkd@im-pla-slave-1    |
Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 445
    rabbit:'-boot/0-fun-0-'/0 line 296
    rabbit_upgrade:maybe_upgrade_mnesia/0 line 148
    rabbit_upgrade:ensure_backup_taken/0 line 106
throw:{error,previous_upgrade_failed}
Log file(s) (may contain more information):
   <stdout>
epic_rabbitmq.0.jh3ntyb6rtkd@im-pla-slave-1    |
init terminating in do_boot ({error,previous_upgrade_failed})
{"init terminating in do_boot",{error,previous_upgrade_failed}}
epic_rabbitmq.0.jh3ntyb6rtkd@im-pla-slave-1    |
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | /paas-docker-entrypoint.sh: line 3:     6 Terminated              sleep 18000
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | 2019-07-03 17:11:17.056 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-im-pla-slave-1'
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | 2019-07-03 17:11:23.230 [error] <0.5.0> Found lock file at /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1/schema_upgrade_lock.
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |             Either previous upgrade is in progress or has failed.
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |             Database backup path: /var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | 2019-07-03 17:11:23.231 [error] <0.5.0>
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | Error description:
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     init:do_boot/3
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     init:start_em/1
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     rabbit:start_it/1 line 445
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     rabbit:'-boot/0-fun-0-'/0 line 296
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     rabbit_upgrade:maybe_upgrade_mnesia/0 line 148
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |     rabbit_upgrade:ensure_backup_taken/0 line 106
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | throw:{error,previous_upgrade_failed}
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    | Log file(s) (may contain more information):
epic_rabbitmq.0.kksmf8yyn22m@im-pla-slave-1    |    <stdout>

[Roman O]

Is there an environment issue or rabbitmq bug?

[Luke Bakken]

Hi Roman,

Please try to reproduce this using the latest version of RabbitMQ, 3.7.15

Thanks -

Luke

[Roman O]

Hi, the same issue is reproduced in latest 3.7.16:

 Starting RabbitMQ 3.7.16 on Erlang 22.0.5
 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/


  ##  ##
  ##  ##      RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/

  ######  ##
  ##########  Logs: <stdout>

              Starting broker...
2019-07-09 08:42:06.664 [info] <0.222.0>
 node           : rabbit@rabbit-vm-centos74
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/paasrabbitmq/paasrabbitmq.conf
 cookie hash    : Sqvm1ja/eeUKv+fi/FfMKg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-centos74
2019-07-09 08:42:09.950 [info] <0.222.0> Running boot step pre_boot defined by app rabbit
2019-07-09 08:42:09.950 [info] <0.222.0> Running boot step rabbit_core_metrics defined by app rabbit
2019-07-09 08:42:09.951 [info] <0.222.0> Running boot step rabbit_alarm defined by app rabbit
2019-07-09 08:42:09.972 [info] <0.234.0> Memory high watermark set to 12867 MiB (13492905574 bytes) of 32169 MiB (33732263936 bytes) total
2019-07-09 08:42:10.009 [info] <0.236.0> Enabling free disk space monitoring
2019-07-09 08:42:10.009 [info] <0.236.0> Disk free limit set to 50MB
2019-07-09 08:42:10.024 [info] <0.222.0> Running boot step code_server_cache defined by app rabbit
2019-07-09 08:42:10.025 [info] <0.222.0> Running boot step file_handle_cache defined by app rabbit
2019-07-09 08:42:10.025 [info] <0.240.0> FHC read buffering:  OFF
2019-07-09 08:42:10.025 [info] <0.240.0> FHC write buffering: ON
2019-07-09 08:42:10.025 [info] <0.239.0> Limiting to approx 65436 file handles (58890 sockets)
2019-07-09 08:42:10.029 [info] <0.222.0> Running boot step worker_pool defined by app rabbit
2019-07-09 08:42:10.029 [info] <0.223.0> Will use 2 processes for default worker pool
2019-07-09 08:42:10.029 [info] <0.223.0> Starting worker pool 'worker_pool' with 2 processes in it
2019-07-09 08:42:10.029 [info] <0.222.0> Running boot step database defined by app rabbit
2019-07-09 08:42:10.030 [info] <0.222.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-centos74 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2019-07-09 08:42:10.030 [info] <0.222.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2019-07-09 08:42:10.030 [info] <0.222.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2019-07-09 08:42:10.030 [info] <0.222.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-07-09 08:42:10.030 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2019-07-09 08:42:10.030 [info] <0.222.0> All discovered existing cluster peers:
2019-07-09 08:42:10.030 [info] <0.222.0> Discovered no peer nodes to cluster with
2019-07-09 08:42:10.035 [info] <0.43.0> Application mnesia exited with reason: stopped
2019-07-09 08:42:11.061 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.114 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.301 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.301 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2019-07-09 08:42:11.301 [info] <0.222.0> Running boot step database_sync defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step codec_correctness_check defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step external_infrastructure defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_registry defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_queue_location_random defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_event defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_priority_queue defined by app rabbit
2019-07-09 08:42:11.305 [error] <0.221.0> CRASH REPORT Process <0.221.0> with 0 neighbours exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74 in application_master:init/4 line 138
2019-07-09 08:42:11.305 [info] <0.43.0> Application rabbit exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,\"src/rabbit_priority_queue.erl\"},{line,74}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,55}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,52}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,28}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,913}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,

"src/rab

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

[Luke Bakken]

Hi Roman,

This is the line causing the error:

https://github.com/rabbitmq/rabbitmq-server/blob/v3.7.16/src/rabbit_priority_queue.erl#L74

You have a corrupted installation or bad configuration.

How exactly are you installing RabbitMQ? Can you please attach your configuration file(s)? Are you using any RABBITMQ_ environment variables?

Thanks -

Luke

[Roman O]

RabbitMQ is deployed as docker service using official docker image

Below is the configuration:

cluster_partition_handling = pause_minority
listeners.tcp.default = 5672
default_pass = paas
default_user = root
hipe_compile =  false
log.default.level = debug

Dockerfile:

FROM rabbitmq:3.7.16

COPY custom-docker-entrypoint.sh /custom-docker-entrypoint.sh


RUN mkdir -p /etc/rabbitmqconf
COPY rabbitmqconf.conf /etc/rabbitmqconf/rabbitmqconf.conf
COPY enabled-plugins.conf /etc/rabbitmqconf/enabled-plugins.conf
RUN chmod -R 777 /etc/rabbitmqconf


ENTRYPOINT ["/custom-docker-entrypoint.sh"]

custom-docker-entrypoint.sh:

sleep 18000 # 30 mins to allow for smooth restart
exec docker-entrypoint.sh rabbitmq-server

Sleep is killed after the below commands are run:

echo -e \$RABBITMQ_ERLANG_COOKIE > /var/lib/rabbitmq/.erlang.cookie

chmod 600 /var/lib/rabbitmq/.erlang.cookie

Below is the environment variables in docker service descriptor:

    environment:              
      - RABBITMQ_LOG_BASE=/var/log/rabbitmq        
      - RABBITMQ_CONFIG_FILE=/etc/rabbitmqconf/rabbitmqconf
      - RABBITMQ_ENABLED_PLUGINS_FILE=/etc/rabbitmqconf/enabled-plugins.conf
      - RABBITMQ_ERLANG_COOKIE=LQKLAGNROLGHUYELNBVY
      - RABBITMQ_DEFAULT_USER=root
      - RABBITMQ_DEFAULT_PASS=paas

Thanks in advance.

[Luke Bakken]

Hi Roman,

The following line echoes the literal string $RABBITMQ_ERLANG_COOKIE to the /var/lib/rabbitmq/.erlang.cookie file, is that really what you intended? The /var/lib/rabbitmq/.erlang.cookie file is read directly and no shell expansion is done.

echo -e \$RABBITMQ_ERLANG_COOKIE > /var/lib/rabbitmq/.erlang.cookie

The above should probably be this:

echo -e LQKLAGNROLGHUYELNBVY > /var/lib/rabbitmq/.erlang.cookie

I'm assuming the following is what is saved to /etc/rabbitmqconf/rabbitmqconf - is that correct?

cluster_partition_handling = pause_minority
listeners.tcp.default = 5672
default_pass = paas
default_user = root
hipe_compile =  false
log.default.level = debug

Please provide the contents of /etc/rabbitmqconf/enabled-plugins.conf

Here's what would lead to the fastest resolution - provide a script that runs all the necessary docker commands to reproduce your issue. I am not very familar with Docker and do not have the time to piece together something that reproduces your issue.

I can restart RabbitMQ 3.7.16 over and over using our generic unix package without seeing this issue.

Thanks,

Luke

[Roman O]

Hi, thanks a lot for your attention. It's important to not that the issue happens randomly. Once i delete the rabbitmq data in volumes the issue is usually gone.

Below is the script you asked for:

docker pull warrior7089/rabbitmq:1.0
docker swarm init
docker stack deploy -c docker-compose.yml rab
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') bash -c 'echo -e $RABBITMQ_ERLANG_COOKIE > /var/lib/rabbitmq/.erlang.cookie'
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') bash -c 'chmod 600 /var/lib/rabbitmq/.erlang.cookie'
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') pkill sleep
sleep 5
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') rabbitmq-plugins enable rabbitmq_management

warrior7089/rabbitmq:1.0 image Dockerfile is Below:

FROM rabbitmq:3.7.16


RUN apt-get update && apt-get -y install iputils-ping


COPY custom-docker-entrypoint.sh /custom-docker-entrypoint.sh


RUN mkdir -p /etc/customrabbitmq
COPY customrabbitmq.conf /etc/customrabbitmq/customrabbitmq.conf
RUN chmod -R 777 /etc/customrabbitmq


ENTRYPOINT ["/custom-docker-entrypoint.sh"]

custom-docker-entrypoint.sh

#!/bin/bash

sleep 18000 # 30 mins to allow for smooth restart
exec docker-entrypoint.sh rabbitmq-server

customrabbitmq.conf

cluster_partition_handling = pause_minority
listeners.tcp.default = 5672

default_pass = root
default_user = pass
hipe_compile =  false

docker-compose.yml

version: '3'
services:
  rabbitmq:
    hostname: "{{.Node.Hostname}}"
    deploy:
      mode: global
      restart_policy:
         delay: 10s
    image: docker.io/warrior7089/rabbitmq:1.0
    healthcheck:
        test: ["CMD-SHELL", "rabbitmqctl node_health_check"]        
        interval: 1m
        timeout: 30s
        retries: 10
    networks:
      - default
    ports:
      - 15672:15672
      - 5672:5672
    volumes:
      - rabbitmq-data-etc:/etc/rabbitmq
      - rabbitmq-data-var:/var/lib/rabbitmq    
    environment:              
      - RABBITMQ_LOG_BASE=/var/log/rabbitmq        
      - RABBITMQ_CONFIG_FILE=/etc/customrabbitmq/customrabbitmq.conf
      - RABBITMQ_ERLANG_COOKIE=ATKLALLROLGHWFVLNTBY
      - RABBITMQ_DEFAULT_USER=root
      - RABBITMQ_DEFAULT_PASS=pass
    logging:
      driver: "json-file"


volumes:
  rabbitmq-data-etc:
  rabbitmq-data-var:

[Roman O]

small correction 

customrabbitmq.conf contents are:

cluster_partition_handling = pause_minority 

listeners.tcp.default = 5672

default_pass = pass
default_user = root 
hipe_compile =  false 

[Michael Klishin]

There is evidence of a 3.6-to-3.7 upgrades with priority queues enabled. Priority queue migration is currently unsupported

(we might address that in one of the final 3.7 releases).

You can use Blue/Green deployment strategy for this upgrade or delete all priority queues before upgrading.

On Thursday, July 4, 2019 at 11:21:44 AM UTC+3, Roman O wrote:

[Roman O]

Thanks. Michael.

However, i don't understand how it is related. 

This issue happens when RabbitMQ 3.7.4 (and 3.7.16) is deployed from scratch with completely empty data folder.

No upgrade is performed. 

Regards

[Luke Bakken]

Hi Roman,

When I use the files you provide, I can't disable the 30 minute sleep even if I modify custom-docker-entrypoint.sh -

lbakken@shostakovich ~/issues/rabbitmq-users/docker-restart-error-Dpl6uR3-LYQ

$ cat custom-docker-entrypoint.sh 

#!/bin/sh

set -o errexit

set -o nounset

exec docker-entrypoint.sh rabbitmq-server

This is what I see in the running container:

$ docker exec -it b19f2112d91d bash

root@shostakovich:/# ps auxww

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root         1  0.3  0.0  20120  3388 ?        Ss   15:31   0:00 /bin/bash /custom-docker-entrypoint.sh

root         6  0.0  0.0   6276   880 ?        S    15:31   0:00 sleep 18000

root       124  0.0  0.0   8268    88 ?        S    15:31   0:00 /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon

root       161 13.0  0.0  20252  3740 pts/0    Ss   15:32   0:00 bash

root       168  0.0  0.0  36144  3320 pts/0    R+   15:32   0:00 ps auxww

root@shostakovich:/# cat custom-docker-entrypoint.sh 

#!/bin/bash

sleep 18000 # 30 mins to allow for smooth restart

exec docker-entrypoint.sh 

It appears that the COPY line in the Dockerfile isn't actually copying the local custom-docker-entrypoint.sh file.

I really don't have time to debug issues like these. Please remove this 30 minute sleep from your Docker image and re-publish it. In addition, provide the command you are using to restart RabbitMQ that might show the issue you report.

I have added all of the files to the following GitHub repository: https://github.com/lukebakken/rabbitmq-users-Dpl6uR3-LYQ

Please let me know your GitHub username and I can add you as a collaborator and we can fix these issues.

If we could reproduce this issue without using docker swarm or compose, that would be even better.

Thanks -
Luke

[Michael Klishin]

If this is a blank node, why do the logs have messages pointing at a schema database upgrade?

> 2019-07-03 17:09:25.718 [info] <0.5.0> upgrades: Backing up mnesia dir to "/var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup"
>
> 2019-07-03 17:09:25.731 [info] <0.5.0> upgrades: Mnesia dir backed up to "/var/lib/rabbitmq/mnesia/rabbit@rabbit-im-pla-slave-1-upgrade-backup"
> 2019-07-03 17:09:25.739 [info] <0.5.0> mnesia upgrades: 39 to apply
> 2019-07-03 17:09:25.828 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-im-pla-slave-1'

> 2019-07-03 17:09:25.836 [info] <0.5.0> mnesia upgrades: Applying rabbit_upgrade_functions:topic_trie_node

[Roman O]

I don't know. 

I guess, that some data is written by RabbitMQ to Mnesia during the initialization.

The 1st error: CRASH REPORT Process <0.221.0> with 0 neighbours exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74 inapplication_master:init/4 line 138 

happens when the node is blank.

Subsequent (2nd and 3rd) error messages and RabbitMQ failures pop after docker swarm auto restarts RabbitMQ container which crashed with the 1st error.

Subsequent restarts (4th and so on) result in the same error (3rd) which vanishes only after RabbitMQ data is deleted from docker volumes. It happened once that even after data deletion, the whole crash sequence happened again.

Regards.

[Roman O]

The issue happened again.

I deleted the contents of rabbitmq-data-etc and rabbitmq-data-var contents and restarted RabbitMQ.

RabbitMQ started successfully then. 

I watched  rabbitmq-data-etc and rabbitmq-data-var contents during the initialization with watch command and noticed that data is written to RabbitMQ data folders during the initialization.

Below is the log of successful initialization of blank node after deletion of RabbitMQ data folder contents, maybe it may provide some useful input:

2019-07-21 08:31:45.742 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.531 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.531 [info] <0.33.0> Application jsx started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.531 [info] <0.33.0> Application crypto started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.532 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.532 [info] <0.33.0> Application recon started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.543 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.833 [info] <0.33.0> Application inets started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.833 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:47.833 [info] <0.33.0> Application public_key started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:48.116 [info] <0.33.0> Application ssl started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:48.125 [info] <0.33.0> Application ranch started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:48.125 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:48.125 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:31:48.164 [info] <0.186.0>

 Starting RabbitMQ 3.7.4 on Erlang 20.3.4
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/

epic_rabbitmq.0.xo727yfxbzle@vm-3.2-cwalech    |

  ##  ##
  ##  ##      RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>

epic_rabbitmq.0.xo727yfxbzle@vm-3.2-cwalech    |
              Starting broker...
2019-07-21 08:31:48.171 [info] <0.186.0>
 node           : rabbit@rabbit-vm-3
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/customrabbitmq/customrabbitmq.conf
 cookie hash    : Sqvm1ja/eeUKv+fi/FfMKg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-3
2019-07-21 08:31:58.000 [info] <0.198.0> Memory high watermark set to 12867 MiB (13492759756 bytes) of 32169 MiB (33731899392 bytes) total
2019-07-21 08:31:58.127 [info] <0.200.0> Enabling free disk space monitoring
2019-07-21 08:31:58.127 [info] <0.200.0> Disk free limit set to 50MB
2019-07-21 08:31:58.172 [info] <0.202.0> Limiting to approx 65436 file handles (58890 sockets)
2019-07-21 08:31:58.172 [info] <0.203.0> FHC read buffering:  OFF
2019-07-21 08:31:58.172 [info] <0.203.0> FHC write buffering: ON
2019-07-21 08:31:58.223 [info] <0.186.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-3 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2019-07-21 08:31:58.223 [info] <0.186.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2019-07-21 08:31:58.223 [info] <0.186.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2019-07-21 08:31:58.224 [info] <0.186.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-07-21 08:31:58.224 [info] <0.186.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2019-07-21 08:31:58.224 [info] <0.186.0> All discovered existing cluster peers:
2019-07-21 08:31:58.224 [info] <0.186.0> Discovered no peer nodes to cluster with
2019-07-21 08:31:58.229 [info] <0.33.0> Application mnesia exited with reason: stopped
2019-07-21 08:31:59.484 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:01.087 [info] <0.186.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-21 08:32:01.245 [info] <0.186.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-21 08:32:01.530 [info] <0.186.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-21 08:32:01.530 [info] <0.186.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2019-07-21 08:32:01.532 [info] <0.186.0> Priority queues enabled, real BQ is rabbit_variable_queue
2019-07-21 08:32:01.569 [info] <0.373.0> Starting rabbit_node_monitor
2019-07-21 08:32:01.684 [info] <0.186.0> message_store upgrades: 1 to apply
2019-07-21 08:32:01.685 [info] <0.186.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2019-07-21 08:32:01.685 [info] <0.186.0> message_store upgrades: No durable queues found. Skipping message store migration
2019-07-21 08:32:01.685 [info] <0.186.0> message_store upgrades: Removing the old message store data
2019-07-21 08:32:01.733 [info] <0.186.0> message_store upgrades: All upgrades applied successfully
2019-07-21 08:32:01.874 [info] <0.186.0> Adding vhost '/'
2019-07-21 08:32:01.923 [info] <0.407.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2019-07-21 08:32:02.191 [info] <0.407.0> Starting message stores for vhost '/'
2019-07-21 08:32:02.191 [info] <0.411.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2019-07-21 08:32:02.276 [info] <0.407.0> Started message store of type transient for vhost '/'
2019-07-21 08:32:02.277 [info] <0.414.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2019-07-21 08:32:02.303 [warning] <0.414.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2019-07-21 08:32:02.329 [info] <0.407.0> Started message store of type persistent for vhost '/'
2019-07-21 08:32:02.331 [info] <0.186.0> Creating user 'admin'
2019-07-21 08:32:02.335 [info] <0.186.0> Setting user tags for user 'admin' to [administrator]
2019-07-21 08:32:02.339 [info] <0.186.0> Setting permissions for 'admin' in '/' to '.*', '.*', '.*'
2019-07-21 08:32:02.361 [info] <0.452.0> started TCP Listener on [::]:5672
2019-07-21 08:32:02.478 [info] <0.186.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbit-vm-3'
2019-07-21 08:32:02.524 [info] <0.186.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbit-vm-3'
2019-07-21 08:32:02.525 [info] <0.33.0> Application rabbit started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:04.376 [info] <0.5.0> Server startup complete; 0 plugins started.
 completed with 0 plugins.
epic_rabbitmq.0.xo727yfxbzle@vm-3.2-cwalech    |
Unable to update config for app rabbit from a .conf file:  The app is already running.
2019-07-21 08:32:19.122 [info] <0.476.0> accepting AMQP connection <0.476.0> (10.0.0.54:36750 -> 10.0.11.115:5672)
2019-07-21 08:32:19.633 [info] <0.476.0> connection <0.476.0> (10.0.0.54:36750 -> 10.0.11.115:5672): user 'admin' authenticated and granted access to vhost '/'
2019-07-21 08:32:27.836 [info] <0.193.0> Management plugin: using rates mode 'basic'
2019-07-21 08:32:27.836 [info] <0.33.0> Application cowlib started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.836 [info] <0.33.0> Application amqp_client started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.837 [info] <0.33.0> Application cowboy started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.837 [info] <0.33.0> Application rabbitmq_web_dispatch started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.861 [info] <0.33.0> Application rabbitmq_management_agent started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.895 [info] <0.555.0> Management plugin started. Port: 15672
2019-07-21 08:32:27.895 [info] <0.661.0> Statistics database started.
2019-07-21 08:32:27.895 [info] <0.33.0> Application rabbitmq_management started on node 'rabbit@rabbit-vm-3'
2019-07-21 08:32:27.896 [info] <0.193.0> Plugins changed; enabled [rabbitmq_management_agent,cowlib,cowboy,rabbitmq_web_dispatch,amqp_client,rabbitmq_management]

On Thursday, July 18, 2019 at 4:05:31 AM UTC+3, Michael Klishin wrote:

[Roman O]

Hi, Luke. 

Thanks a lot for your time.

I forked you repo, pushed the relevant changes and tested that it works on linux (centos) and windows (10). You can find it here, added you as a collaborator.

Sleep is needed so erlang cookie would be overridden before starting RabbitMQ server. COPY line in the Dockerfile  copies the custom-docker-entrypoint.sh. You can find it inside the container after the service is deployed.  Docker swarm is not a must, however i tried to replicate the way i use RabbitMQ.

It's possible that in your case pkill sleep was run on still not running container .

I added sleep 30 command after creating the docker service and before running commands that follow to avoid the issues you had.

Below is successful sequence of manual commands which start RabbitMQ according to the commands i provided and assuming that docker-compose.yml is present in the current folder where you run docker commands.

docker stack deploy -c docker-compose.yml rab

Creating service rab_rabbitmq



docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES



docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                            PORTS                                NAMES
a2c4d9958b0c        warrior7089/rabbitmq:1.0   "/custom-docker-entr…"   6 seconds ago       Up 2 seconds (health: starting)   4369/tcp, 5671-5672/tcp, 25672/tcp   rab_rabbitmq.ow63mfie6xq9g53xkk7gy6vs5.xxfgad7cb6rswtck7r3gn47el


docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') bash -c 'echo -e $RABBITMQ_ERLANG_COOKIE > /var/lib/rabbitmq/.erlang.cookie'

docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') bash -c 'chmod 600 /var/lib/rabbitmq/.erlang.cookie'
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') pkill sleep
sleep 5
docker exec -i $(docker ps -q -f 'name=rab_rabbitmq') rabbitmq-plugins enable rabbitmq_management

Enabling plugins on node rabbit@linuxkit-00155d7ff100:
rabbitmq_management
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@linuxkit-00155d7ff100...
Plugin configuration unchanged.


docker service logs -f rab_rabbitmq


rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | /custom-docker-entrypoint.sh: line 3:     6 Terminated              sleep 18000
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:47.977 [info] <0.262.0>
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  Starting RabbitMQ 3.7.16 on Erlang 22.0.5
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  Copyright (C) 2007-2019 Pivotal Software, Inc.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  Licensed under the MPL.  See https://www.rabbitmq.com/
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |   ##  ##
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |   ##  ##      RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |   ######  ##
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |   ##########  Logs: <stdout>
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |               Starting broker...
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:47.979 [info] <0.262.0>
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  node           : rabbit@linuxkit-00155d7ff100
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  home dir       : /var/lib/rabbitmq
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  config file(s) : /etc/customrabbitmq/customrabbitmq.conf
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  cookie hash    : MmiXGCVZJ68rk89VycE1/w==
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  log(s)         : <stdout>
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@linuxkit-00155d7ff100
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.600 [info] <0.262.0> Running boot step pre_boot defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.600 [info] <0.262.0> Running boot step rabbit_core_metrics defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.601 [info] <0.262.0> Running boot step rabbit_alarm defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.607 [info] <0.310.0> Memory high watermark set to 792 MiB (830613094 bytes) of 1980 MiB (2076532736 bytes) total
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.613 [info] <0.312.0> Enabling free disk space monitoring
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.613 [info] <0.312.0> Disk free limit set to 50MB
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.617 [info] <0.262.0> Running boot step code_server_cache defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.262.0> Running boot step file_handle_cache defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.316.0> FHC read buffering:  OFFrab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.316.0> FHC write buffering: ON
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.315.0> Limiting to approx 1048476 file handles (943626 sockets)
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.262.0> Running boot step worker_pool defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.618 [info] <0.264.0> Will use 2 processes for default worker pool
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.619 [info] <0.264.0> Starting worker pool 'worker_pool' with 2 processes in it
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.619 [info] <0.262.0> Running boot step database defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.622 [info] <0.262.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.695 [info] <0.262.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.695 [info] <0.262.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.695 [info] <0.262.0> Running boot step database_sync defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step codec_correctness_check defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step external_infrastructure defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_registry defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_queue_location_random defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_event defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.696 [info] <0.262.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_priority_queue defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Priority queues enabled, real BQ is rabbit_variable_queue
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step kernel_ready defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.697 [info] <0.262.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.700 [info] <0.262.0> Running boot step guid_generator defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.710 [info] <0.262.0> Running boot step rabbit_node_monitor defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.711 [info] <0.340.0> Starting rabbit_node_monitor
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.711 [info] <0.262.0> Running boot step delegate_sup defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.711 [info] <0.262.0> Running boot step rabbit_memory_monitor defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.712 [info] <0.262.0> Running boot step core_initialized defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.712 [info] <0.262.0> Running boot step upgrade_queues defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.750 [info] <0.262.0> Running boot step rabbit_connection_tracking defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.751 [info] <0.262.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.751 [info] <0.262.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.751 [info] <0.262.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.752 [info] <0.262.0> Running boot step rabbit_policies defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step rabbit_policy defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step rabbit_vhost_limit defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Management plugin: using rates mode 'basic'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.753 [info] <0.262.0> Running boot step recovery defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.756 [info] <0.373.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@linuxkit-00155d7ff100/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.764 [info] <0.373.0> Starting message stores for vhost '/'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.764 [info] <0.377.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.765 [info] <0.373.0> Started message store of type transient for vhost '/'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.766 [info] <0.380.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.766 [warning] <0.380.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.768 [info] <0.373.0> Started message store of type persistent for vhost '/'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.771 [info] <0.262.0> Running boot step load_definitions defined by app rabbitmq_management
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.771 [info] <0.262.0> Running boot step empty_db_check defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.771 [info] <0.262.0> Running boot step rabbit_looking_glass defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.772 [info] <0.262.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.772 [info] <0.262.0> Running boot step background_gc defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.773 [info] <0.262.0> Running boot step connection_tracking defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.773 [info] <0.262.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@linuxkit-00155d7ff100'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.773 [info] <0.262.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@linuxkit-00155d7ff100'
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.774 [info] <0.262.0> Running boot step routing_ready defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.774 [info] <0.262.0> Running boot step pre_flight defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.774 [info] <0.262.0> Running boot step notify_cluster defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.774 [info] <0.262.0> Running boot step networking defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.777 [warning] <0.403.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.778 [info] <0.417.0> started TCP listener on [::]:5672
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.779 [info] <0.262.0> Running boot step direct_client defined by app rabbit
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.820 [info] <0.476.0> Management plugin: HTTP (non-TLS) listener started on port 15672
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.821 [info] <0.582.0> Statistics database started.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.822 [info] <0.581.0> Starting worker pool 'management_worker_pool' with 3 processes in it
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | 2019-07-21 11:45:48.928 [info] <0.8.0> Server startup complete; 3 plugins started.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  * rabbitmq_management
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  * rabbitmq_management_agent
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  * rabbitmq_web_dispatch
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |  completed with 3 plugins.
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    |
rab_rabbitmq.0.xxfgad7cb6rs@linuxkit-00155d7ff100    | Unable to update config for app rabbit from a .conf file:  The app is already running.

[Roman O]

Any updates on this? You can clearly see that the original error no match of right hand value undefined in rabbit_priority_queue:enable happens when Node database directory is empty.

Schema database upgrade you mentioned happens in subsequent RabbitMQ restarts 

On Thursday, July 18, 2019 at 4:05:31 AM UTC+3, Michael Klishin wrote:

If this is a blank node, why do the logs have messages pointing at a schema database upgrade?

[Luke Bakken]

Hi Roman,

I have been away this past week. Please remember that support is provided for free and may take a while. If I have time to investigate this week I will.

Thanks,

Luke

[Roman O]

Hi, Luke

It's ok. It was my response to Michael Klishin remark. 

I provided you with all the scripts and added you as a collaborator. So would be happy to get your help when and if you can. 

Regards,

Roman.

[Michael Klishin]

I cannot think of a scenario where a blank node would fail to enable priority queues. This is the first time I see this brought up on this list.

Please provide a set of steps to reproduce with 3.7.17. We will not be investigating with 3.7.4 (it is over 10 patch releases behind, sorry)

[Roman O]

Luke asked me to use latest 3.7.16 and the same error happened. You can see it. I've shared the log in the post from 9th of July

Attached it again with relevant colored lines 

Issue happens randomly. Posting again the steps for reproduction in the next post

Starting RabbitMQ 3.7.16 on Erlang 22.0.5

 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/


  ##  ##
  ##  ##      RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/

  ######  ##
  ##########  Logs: <stdout>

              Starting broker...

2019-07-09 08:42:06.664 [info] <0.222.0> 
 node           : rabbit@rabbit-vm-

centos74
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/paasrabbitmq/paasrabbitmq.conf
 cookie hash    : Sqvm1ja/eeUKv+fi/FfMKg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-centos74
2019-07-09 08:42:09.950 [info] <0.222.0> Running boot step pre_boot defined by app rabbit
2019-07-09 08:42:09.950 [info] <0.222.0> Running boot step rabbit_core_metrics defined by app rabbit
2019-07-09 08:42:09.951 [info] <0.222.0> Running boot step rabbit_alarm defined by app rabbit
2019-07-09 08:42:09.972 [info] <0.234.0> Memory high watermark set to 12867 MiB (13492905574 bytes) of 32169 MiB (33732263936 bytes) total
2019-07-09 08:42:10.009 [info] <0.236.0> Enabling free disk space monitoring
2019-07-09 08:42:10.009 [info] <0.236.0> Disk free limit set to 50MB
2019-07-09 08:42:10.024 [info] <0.222.0> Running boot step code_server_cache defined by app rabbit
2019-07-09 08:42:10.025 [info] <0.222.0> Running boot step file_handle_cache defined by app rabbit

2019-07-09 08:42:10.025 [info] <0.240.0> FHC read buffering:  OFF
2019-07-09 08:42:10.025 [info] <0.240.0> FHC write buffering: ON

2019-07-09 08:42:10.025 [info] <0.239.0> Limiting to approx 65436 file handles (58890 sockets)
2019-07-09 08:42:10.029 [info] <0.222.0> Running boot step worker_pool defined by app rabbit
2019-07-09 08:42:10.029 [info] <0.223.0> Will use 2 processes for default worker pool
2019-07-09 08:42:10.029 [info] <0.223.0> Starting worker pool 'worker_pool' with 2 processes in it
2019-07-09 08:42:10.029 [info] <0.222.0> Running boot step database defined by app rabbit
2019-07-09 08:42:10.030 [info] <0.222.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-vm-centos74 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2019-07-09 08:42:10.030 [info] <0.222.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2019-07-09 08:42:10.030 [info] <0.222.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2019-07-09 08:42:10.030 [info] <0.222.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-07-09 08:42:10.030 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2019-07-09 08:42:10.030 [info] <0.222.0> All discovered existing cluster peers: 
2019-07-09 08:42:10.030 [info] <0.222.0> Discovered no peer nodes to cluster with
2019-07-09 08:42:10.035 [info] <0.43.0> Application mnesia exited with reason: stopped
2019-07-09 08:42:11.061 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.114 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.301 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-07-09 08:42:11.301 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2019-07-09 08:42:11.301 [info] <0.222.0> Running boot step database_sync defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step codec_correctness_check defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step external_infrastructure defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_registry defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_queue_location_random defined by app rabbit
2019-07-09 08:42:11.302 [info] <0.222.0> Running boot step rabbit_event defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2019-07-09 08:42:11.303 [info] <0.222.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2019-07-09 08:42:11.304 [info] <0.222.0> Running boot step rabbit_priority_queue defined by app rabbit
2019-07-09 08:42:11.305 [error] <0.221.0> CRASH REPORT Process <0.221.0> with 0 neighbours exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74 in application_master:init/4 line 138
2019-07-09 08:42:11.305 [info] <0.43.0> Application rabbit exited with reason: no match of right hand value undefined in rabbit_priority_queue:enable/0 line 74
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,\"src/rabbit_priority_queue.erl\"},{line,74}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,55}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,52}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,28}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,913}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,undefined},[{rabbit_priority_queue,enable,0,[{file,

"src/rab

[Roman O]

You can clone the repo here and run run-docker.sh inside the cloned repo folder. The script would simulate the way i use RabbitMQ.

To inspect logs of RabbitMQ run 

docker service logs -f rab_rabbitmq

To remind, the issue happens randomly and mostly (~95% cases) goes up just fine.