rabbit_peer_discovery_k8s - Pods crashes with error "no match of right hand value {error,enoent}"

[Srinivasa Kartheek]

Hello

I am trying to create a rabbitmq cluster inside a GKE kubernetes cluster using stateful set and a headless service (named rabbitmq). The pod management policy of this stateful set is 'OrderedReady'. The pod initialization fails with error message "no match of right hand value {error,enoent}".

2019-07-03 17:13:23.006 [debug] <0.115.0> Lager installed handler error_logger_lager_h into error_logger

2019-07-03 17:13:23.006 [debug] <0.118.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event

2019-07-03 17:13:23.006 [debug] <0.121.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event

2019-07-03 17:13:23.006 [debug] <0.124.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event

2019-07-03 17:13:23.006 [debug] <0.127.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event

2019-07-03 17:13:23.006 [debug] <0.130.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event

2019-07-03 17:13:23.006 [debug] <0.133.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event

2019-07-03 17:13:23.006 [debug] <0.136.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event

2019-07-03 17:13:23.006 [debug] <0.139.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event

2019-07-03 17:13:23.006 [debug] <0.142.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event

2019-07-03 17:13:23.464 [debug] <0.109.0> Lager installed handler lager_backend_throttle into lager_event

2019-07-03 17:13:24.405 [info] <0.216.0> 

 Starting RabbitMQ 3.7.15 on Erlang 22.0.4

 Copyright (C) 2007-2019 Pivotal Software, Inc.

 Licensed under the MPL.  See https://www.rabbitmq.com/

  ##  ##

  ##  ##      RabbitMQ 3.7.15. Copyright (C) 2007-2019 Pivotal Software, Inc.

  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/

  ######  ##

  ##########  Logs: <stdout>

              Starting broker...

2019-07-03 17:13:24.411 [info] <0.216.0> 

 node           : rab...@rabbitmq-0.rabbitmq.test294.svc.cluster.local

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.conf

 cookie hash    : XhdCf8zpVJeJ0EHyaxszPg==

 log(s)         : <stdout>

 database dir   : /var/lib/rabbitmq/mnesia/rab...@rabbitmq-0.rabbitmq.test294.svc.cluster.local

2019-07-03 17:13:28.533 [info] <0.216.0> Running boot step pre_boot defined by app rabbit

2019-07-03 17:13:28.533 [info] <0.216.0> Running boot step rabbit_core_metrics defined by app rabbit

2019-07-03 17:13:28.535 [debug] <0.216.0> Applying MFA: M = rabbit_sup, F = start_child, A = [rabbit_metrics]

2019-07-03 17:13:28.536 [info] <0.216.0> Running boot step rabbit_alarm defined by app rabbit

2019-07-03 17:13:28.536 [debug] <0.216.0> Applying MFA: M = rabbit_alarm, F = start, A = []

2019-07-03 17:13:28.538 [debug] <0.221.0> Supervisor rabbit_alarm_sup started rabbit_alarm:start_link() at pid <0.222.0>

2019-07-03 17:13:28.552 [info] <0.224.0> Memory high watermark set to 1479 MiB (1551451750 bytes) of 3698 MiB (3878629376 bytes) total

2019-07-03 17:13:28.553 [debug] <0.223.0> Supervisor vm_memory_monitor_sup started vm_memory_monitor:start_link(0.4, #Fun<rabbit_alarm.0.13465474>, #Fun<rabbit_alarm.1.13465474>) at pid <0.224.0>

2019-07-03 17:13:28.563 [info] <0.226.0> Enabling free disk space monitoring

2019-07-03 17:13:28.563 [info] <0.226.0> Disk free limit set to 50MB

2019-07-03 17:13:28.571 [debug] <0.225.0> Supervisor rabbit_disk_monitor_sup started rabbit_disk_monitor:start_link(50000000) at pid <0.226.0>

2019-07-03 17:13:28.571 [info] <0.216.0> Running boot step code_server_cache defined by app rabbit

2019-07-03 17:13:28.571 [debug] <0.216.0> Applying MFA: M = rabbit_sup, F = start_child, A = [code_server_cache]

2019-07-03 17:13:28.571 [info] <0.216.0> Running boot step file_handle_cache defined by app rabbit

2019-07-03 17:13:28.571 [debug] <0.216.0> Applying MFA: M = rabbit, F = start_fhc, A = []

2019-07-03 17:13:28.572 [info] <0.229.0> Limiting to approx 1048476 file handles (943626 sockets)

2019-07-03 17:13:28.574 [debug] <0.228.0> Supervisor file_handle_cache_sup started file_handle_cache:start_link(fun rabbit_alarm:set_alarm/1, fun rabbit_alarm:clear_alarm/1) at pid <0.229.0>

2019-07-03 17:13:28.574 [info] <0.230.0> FHC read buffering:  OFF

2019-07-03 17:13:28.575 [info] <0.230.0> FHC write buffering: ON

2019-07-03 17:13:28.578 [info] <0.216.0> Running boot step worker_pool defined by app rabbit

2019-07-03 17:13:28.579 [debug] <0.216.0> Applying MFA: M = rabbit_sup, F = start_supervisor_child, A = [worker_pool_sup]

2019-07-03 17:13:28.580 [info] <0.216.0> Running boot step database defined by app rabbit

2019-07-03 17:13:28.580 [debug] <0.216.0> Applying MFA: M = rabbit_mnesia, F = init, A = []

2019-07-03 17:13:28.583 [info] <0.216.0> Node database directory at /var/lib/rabbitmq/mnesia/rab...@rabbitmq-0.rabbitmq.test294.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch...

2019-07-03 17:13:28.583 [info] <0.216.0> Configured peer discovery backend: rabbit_peer_discovery_k8s

2019-07-03 17:13:28.583 [debug] <0.216.0> Peer discovery backend supports initialisation.

2019-07-03 17:13:28.583 [debug] <0.216.0> Peer discovery Kubernetes: initialising...

2019-07-03 17:13:28.584 [debug] <0.216.0> HTTP client proxy is not configured

2019-07-03 17:13:28.584 [debug] <0.216.0> Peer discovery backend initialisation succeeded.

2019-07-03 17:13:28.585 [info] <0.216.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s

2019-07-03 17:13:28.585 [info] <0.216.0> Peer discovery backend does not support locking, falling back to randomized delay

2019-07-03 17:13:28.586 [info] <0.216.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.

2019-07-03 17:13:28.588 [error] <0.215.0> CRASH REPORT Process <0.215.0> with 0 neighbours exited with reason: no match of right hand value {error,enoent} in rabbit_peer_discovery_k8s:make_request/0 line 109 in application_master:init/4 line 138

2019-07-03 17:13:28.589 [info] <0.43.0> Application rabbit exited with reason: no match of right hand value {error,enoent} in rabbit_peer_discovery_k8s:make_request/0 line 109

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,{error,enoent}},[{rabbit_peer_discovery_k8s,make_request,0,

{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{badmatch,{error,enoent}},[{rabbit_peer_discovery_k8s,make_request,0,[{file,\"src/rabbit_peer_discovery_k8s.erl\"},{line,109}]},{rabbit_peer_discovery_k8s,list_nodes,0,[{file,\"src/rabbit_peer_discovery_k8s.erl\"},{line,55}]},{rabbit_peer_discovery,discover_cluster_nodes,0,[{file,\"src/rabbit_peer_discovery.erl\"},{line,120}]},{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,55}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,52}]}]}}}}}"}

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Even after enabling debug mode, I couldn't understand what is wrong with my setup. Will appreciate any lead on this one. Please find the attached 'rabbitmq.conf', 'statefulset.yml'

Thanks

[Srinivasa Kartheek]

I was able to find the problem, the service account credentials are not mounted in my pods in statefulset even though the service account configuration has 'automountServiceAccountToken' set to true. I had to set it in pod template in my stateful set as well.