RabbitMQ v3.9.13/Erlang 24.2.1 - Command Timeout when attempting to read disk free on Windows

331 views
Skip to first unread message

Damian TagDev

unread,
Feb 10, 2022, 5:56:54 AM2/10/22
to rabbitmq-users
Hi,

My machine is running Windows 10 v21H2 - OS Build 19044.1466, Erlang 24.2.1, and RabbitMQ v3.9.13.

I've noticed the following error appearing in the RabbitMQ logs daily.
Command timed out: 'powershell.exe -NoLogo -NoProfile -NonInteractive -Command (Get-PSDrive c).Free'

If I run the command in powershell it returns with a result instantly.

The management API is showing free disk space correctly - so not a huge problem.

Here is the full log from startup up to the error:
2022-02-10 09:07:14.005000+00:00 [notice] <0.226.0> Logging: configured log handlers are now ACTIVE
2022-02-10 09:07:21.982000+00:00 [info] <0.226.0> ra: starting system quorum_queues
2022-02-10 09:07:21.982000+00:00 [info] <0.226.0> starting Ra system: quorum_queues in directory: c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/quorum/rabbit@MachineName
2022-02-10 09:07:21.982000+00:00 [info] <0.300.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2022-02-10 09:07:21.982000+00:00 [notice] <0.305.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2022-02-10 09:07:22.013000+00:00 [info] <0.305.0> wal: preallocation may not be supported by the file system falling back to fsync instead of fdatasync
2022-02-10 09:07:22.013000+00:00 [info] <0.226.0> ra: starting system coordination
2022-02-10 09:07:22.013000+00:00 [info] <0.226.0> starting Ra system: coordination in directory: c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/coordination/rabbit@MachineName
2022-02-10 09:07:22.013000+00:00 [info] <0.351.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2022-02-10 09:07:22.013000+00:00 [notice] <0.356.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2022-02-10 09:07:22.029000+00:00 [info] <0.356.0> wal: preallocation may not be supported by the file system falling back to fsync instead of fdatasync
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  Starting RabbitMQ 3.9.13 on Erlang 24.2.1 [jit]
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  node           : rabbit@MachineName
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  home dir       : c:/WINDOWS/system32/config/systemprofile
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  config file(s) : c:/Users/Username/AppData/Roaming/RabbitMQ/advanced.config
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  cookie hash    : aQzBFMHtpzhf5MCOY77zYw==
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  log(s)         : <stdout>
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>                 : c:/Users/Username/AppData/Roaming/RabbitMQ/log/rab...@MachineName.log
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>                 : c:/Users/Username/AppData/Roaming/RabbitMQ/log/rabbit@MachineName_upgrade.log
2022-02-10 09:07:22.029000+00:00 [info] <0.226.0>  database dir   : c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags: list of feature flags found:
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] drop_unroutable_metric
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] empty_basic_get_metric
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] implicit_default_bindings
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] maintenance_mode_status
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] quorum_queue
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] stream_queue
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] user_limits
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags:   [x] virtual_host_metadata
2022-02-10 09:07:22.419000+00:00 [info] <0.226.0> Feature flags: feature flag states written to disk: yes
2022-02-10 09:07:22.747000+00:00 [info] <0.226.0> Running boot step pre_boot defined by app rabbit
2022-02-10 09:07:22.747000+00:00 [info] <0.226.0> Running boot step rabbit_global_counters defined by app rabbit
2022-02-10 09:07:22.747000+00:00 [info] <0.226.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2022-02-10 09:07:22.747000+00:00 [info] <0.226.0> Running boot step rabbit_core_metrics defined by app rabbit
2022-02-10 09:07:22.747000+00:00 [info] <0.226.0> Running boot step rabbit_alarm defined by app rabbit
2022-02-10 09:07:22.747000+00:00 [info] <0.366.0> Memory high watermark set to 6470 MiB (6784412876 bytes) of 16175 MiB (16961032192 bytes) total
2022-02-10 09:07:24.600000+00:00 [info] <0.368.0> Enabling free disk space monitoring
2022-02-10 09:07:24.600000+00:00 [info] <0.368.0> Disk free limit set to 50MB
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Running boot step code_server_cache defined by app rabbit
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Running boot step file_handle_cache defined by app rabbit
2022-02-10 09:07:25.246000+00:00 [info] <0.375.0> Limiting to approx 65439 file handles (58893 sockets)
2022-02-10 09:07:25.246000+00:00 [info] <0.376.0> FHC read buffering: OFF
2022-02-10 09:07:25.246000+00:00 [info] <0.376.0> FHC write buffering: ON
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Running boot step worker_pool defined by app rabbit
2022-02-10 09:07:25.246000+00:00 [info] <0.358.0> Will use 4 processes for default worker pool
2022-02-10 09:07:25.246000+00:00 [info] <0.358.0> Starting worker pool 'worker_pool' with 4 processes in it
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Running boot step database defined by app rabbit
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Successfully synced tables from a peer
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-02-10 09:07:25.246000+00:00 [info] <0.226.0> Successfully synced tables from a peer
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Successfully synced tables from a peer
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step database_sync defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step feature_flags defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step codec_correctness_check defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step external_infrastructure defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step rabbit_registry defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step rabbit_queue_location_random defined by app rabbit
2022-02-10 09:07:25.261000+00:00 [info] <0.226.0> Running boot step rabbit_event defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_priority_queue defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Priority queues enabled, real BQ is rabbit_variable_queue
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step kernel_ready defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.398.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step guid_generator defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_node_monitor defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.402.0> Starting rabbit_node_monitor
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step delegate_sup defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step rabbit_memory_monitor defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step core_initialized defined by app rabbit
2022-02-10 09:07:25.277000+00:00 [info] <0.226.0> Running boot step upgrade_queues defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step channel_tracking defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@MachineName'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@MachineName'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step connection_tracking defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@MachineName'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@MachineName'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@MachineName'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_policies defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_policy defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_vhost_limit defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Management plugin: using rates mode 'basic'
2022-02-10 09:07:25.292000+00:00 [info] <0.226.0> Running boot step recovery defined by app rabbit
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.3668.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.3271.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.3468.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.676.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.473.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1271.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2674.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1674.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1871.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1474.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.673.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2071.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.3077.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1868.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.3465.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2268.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1268.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.1471.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2877.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2874.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.670.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.667.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.888.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2671.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2471.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2265.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.727867240.1590427651.700>}} from <0.226.0> to <0.2468.0> in an old incarnation (1644395467) of this node (1644484026)
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [error] <0.226.0>
2022-02-10 09:07:25.292000+00:00 [info] <0.440.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/8XJR85D7DFAU77VJK6IPHY6IE' for vhost 'Alpha Corp_641b4cfa-5216-4af6-aa60-9385bbf6125d' exists
2022-02-10 09:07:25.292000+00:00 [info] <0.440.0> Starting message stores for vhost 'Alpha Corp_641b4cfa-5216-4af6-aa60-9385bbf6125d'
2022-02-10 09:07:25.292000+00:00 [info] <0.444.0> Message store "8XJR85D7DFAU77VJK6IPHY6IE/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.308000+00:00 [info] <0.440.0> Started message store of type transient for vhost 'Alpha Corp_641b4cfa-5216-4af6-aa60-9385bbf6125d'
2022-02-10 09:07:25.308000+00:00 [info] <0.448.0> Message store "8XJR85D7DFAU77VJK6IPHY6IE/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.308000+00:00 [info] <0.440.0> Started message store of type persistent for vhost 'Alpha Corp_641b4cfa-5216-4af6-aa60-9385bbf6125d'
2022-02-10 09:07:25.308000+00:00 [info] <0.440.0> Recovering 1 queues of type rabbit_classic_queue took 20ms
2022-02-10 09:07:25.308000+00:00 [info] <0.440.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.308000+00:00 [info] <0.440.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.324000+00:00 [info] <0.634.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/6SIH8QYGXT7UWY1TWBSTSBS3Q' for vhost 'Sprint Review 73_bb42f586-d77b-4ca4-94a7-5a6593a5aacf' exists
2022-02-10 09:07:25.339000+00:00 [info] <0.634.0> Starting message stores for vhost 'Sprint Review 73_bb42f586-d77b-4ca4-94a7-5a6593a5aacf'
2022-02-10 09:07:25.339000+00:00 [info] <0.638.0> Message store "6SIH8QYGXT7UWY1TWBSTSBS3Q/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.339000+00:00 [info] <0.634.0> Started message store of type transient for vhost 'Sprint Review 73_bb42f586-d77b-4ca4-94a7-5a6593a5aacf'
2022-02-10 09:07:25.339000+00:00 [info] <0.642.0> Message store "6SIH8QYGXT7UWY1TWBSTSBS3Q/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.339000+00:00 [info] <0.634.0> Started message store of type persistent for vhost 'Sprint Review 73_bb42f586-d77b-4ca4-94a7-5a6593a5aacf'
2022-02-10 09:07:25.355000+00:00 [info] <0.634.0> Recovering 4 queues of type rabbit_classic_queue took 21ms
2022-02-10 09:07:25.355000+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.355000+00:00 [info] <0.634.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.386000+00:00 [info] <0.855.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/2Z32JV915377W9BZBUR1Z8S9S' for vhost 'Preprod Test_3b65d483-8598-40e6-bd58-59fff63c98bc' exists
2022-02-10 09:07:25.386000+00:00 [info] <0.855.0> Starting message stores for vhost 'Preprod Test_3b65d483-8598-40e6-bd58-59fff63c98bc'
2022-02-10 09:07:25.386000+00:00 [info] <0.859.0> Message store "2Z32JV915377W9BZBUR1Z8S9S/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.386000+00:00 [info] <0.855.0> Started message store of type transient for vhost 'Preprod Test_3b65d483-8598-40e6-bd58-59fff63c98bc'
2022-02-10 09:07:25.386000+00:00 [info] <0.863.0> Message store "2Z32JV915377W9BZBUR1Z8S9S/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.402000+00:00 [info] <0.855.0> Started message store of type persistent for vhost 'Preprod Test_3b65d483-8598-40e6-bd58-59fff63c98bc'
2022-02-10 09:07:25.402000+00:00 [info] <0.855.0> Recovering 1 queues of type rabbit_classic_queue took 18ms
2022-02-10 09:07:25.402000+00:00 [info] <0.855.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.402000+00:00 [info] <0.855.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.417000+00:00 [info] <0.1049.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2022-02-10 09:07:25.417000+00:00 [info] <0.1049.0> Starting message stores for vhost '/'
2022-02-10 09:07:25.417000+00:00 [info] <0.1053.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.417000+00:00 [info] <0.1049.0> Started message store of type transient for vhost '/'
2022-02-10 09:07:25.417000+00:00 [info] <0.1057.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.433000+00:00 [info] <0.1049.0> Started message store of type persistent for vhost '/'
2022-02-10 09:07:25.433000+00:00 [info] <0.1049.0> Recovering 0 queues of type rabbit_classic_queue took 14ms
2022-02-10 09:07:25.433000+00:00 [info] <0.1049.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.433000+00:00 [info] <0.1049.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.433000+00:00 [info] <0.1234.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/7WNH5HJD6DV4XPSZA6SIZ4VQS' for vhost 'Test Organisation 06_5e24b6c8-ee75-4ddb-a27c-eeda7f34bc97' exists
2022-02-10 09:07:25.449000+00:00 [info] <0.1234.0> Starting message stores for vhost 'Test Organisation 06_5e24b6c8-ee75-4ddb-a27c-eeda7f34bc97'
2022-02-10 09:07:25.449000+00:00 [info] <0.1238.0> Message store "7WNH5HJD6DV4XPSZA6SIZ4VQS/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.449000+00:00 [info] <0.1234.0> Started message store of type transient for vhost 'Test Organisation 06_5e24b6c8-ee75-4ddb-a27c-eeda7f34bc97'
2022-02-10 09:07:25.449000+00:00 [info] <0.1242.0> Message store "7WNH5HJD6DV4XPSZA6SIZ4VQS/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.449000+00:00 [info] <0.1234.0> Started message store of type persistent for vhost 'Test Organisation 06_5e24b6c8-ee75-4ddb-a27c-eeda7f34bc97'
2022-02-10 09:07:25.464000+00:00 [info] <0.1234.0> Recovering 2 queues of type rabbit_classic_queue took 29ms
2022-02-10 09:07:25.464000+00:00 [info] <0.1234.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.464000+00:00 [info] <0.1234.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.480000+00:00 [info] <0.1437.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/EXOVE1LTRTSZGQWZ526AC42DH' for vhost 'Test Organisation 10_4b78224a-4aeb-48fe-8401-b8b138c43c9d' exists
2022-02-10 09:07:25.496000+00:00 [info] <0.1437.0> Starting message stores for vhost 'Test Organisation 10_4b78224a-4aeb-48fe-8401-b8b138c43c9d'
2022-02-10 09:07:25.496000+00:00 [info] <0.1441.0> Message store "EXOVE1LTRTSZGQWZ526AC42DH/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.496000+00:00 [info] <0.1437.0> Started message store of type transient for vhost 'Test Organisation 10_4b78224a-4aeb-48fe-8401-b8b138c43c9d'
2022-02-10 09:07:25.496000+00:00 [info] <0.1445.0> Message store "EXOVE1LTRTSZGQWZ526AC42DH/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.496000+00:00 [info] <0.1437.0> Started message store of type persistent for vhost 'Test Organisation 10_4b78224a-4aeb-48fe-8401-b8b138c43c9d'
2022-02-10 09:07:25.527000+00:00 [info] <0.1437.0> Recovering 2 queues of type rabbit_classic_queue took 31ms
2022-02-10 09:07:25.527000+00:00 [info] <0.1437.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.527000+00:00 [info] <0.1437.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.527000+00:00 [info] <0.1640.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/B4QGNTUYYXS3X1WU202GKQFXA' for vhost 'Ben Bates CRM_c2d55910-5e15-49bd-b8c6-33f3c2a048a3' exists
2022-02-10 09:07:25.542000+00:00 [info] <0.1640.0> Starting message stores for vhost 'Ben Bates CRM_c2d55910-5e15-49bd-b8c6-33f3c2a048a3'
2022-02-10 09:07:25.542000+00:00 [info] <0.1644.0> Message store "B4QGNTUYYXS3X1WU202GKQFXA/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.542000+00:00 [info] <0.1640.0> Started message store of type transient for vhost 'Ben Bates CRM_c2d55910-5e15-49bd-b8c6-33f3c2a048a3'
2022-02-10 09:07:25.542000+00:00 [info] <0.1648.0> Message store "B4QGNTUYYXS3X1WU202GKQFXA/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.542000+00:00 [info] <0.1640.0> Started message store of type persistent for vhost 'Ben Bates CRM_c2d55910-5e15-49bd-b8c6-33f3c2a048a3'
2022-02-10 09:07:25.558000+00:00 [info] <0.1640.0> Recovering 1 queues of type rabbit_classic_queue took 19ms
2022-02-10 09:07:25.558000+00:00 [info] <0.1640.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.558000+00:00 [info] <0.1640.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.574000+00:00 [info] <0.1834.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/8K1QASHCSCTCB0QMJL9CJE9D7' for vhost 'Test Organisation 08_2c70d7fa-a503-4c89-887f-5b4541e79f80' exists
2022-02-10 09:07:25.574000+00:00 [info] <0.1834.0> Starting message stores for vhost 'Test Organisation 08_2c70d7fa-a503-4c89-887f-5b4541e79f80'
2022-02-10 09:07:25.574000+00:00 [info] <0.1838.0> Message store "8K1QASHCSCTCB0QMJL9CJE9D7/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.574000+00:00 [info] <0.1834.0> Started message store of type transient for vhost 'Test Organisation 08_2c70d7fa-a503-4c89-887f-5b4541e79f80'
2022-02-10 09:07:25.574000+00:00 [info] <0.1842.0> Message store "8K1QASHCSCTCB0QMJL9CJE9D7/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.574000+00:00 [info] <0.1834.0> Started message store of type persistent for vhost 'Test Organisation 08_2c70d7fa-a503-4c89-887f-5b4541e79f80'
2022-02-10 09:07:25.589000+00:00 [info] <0.1834.0> Recovering 2 queues of type rabbit_classic_queue took 26ms
2022-02-10 09:07:25.589000+00:00 [info] <0.1834.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.589000+00:00 [info] <0.1834.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.605000+00:00 [info] <0.2037.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/A3BA4PLZT18AW5P11CSPMZMQ7' for vhost 'Gamma Inc_f0fb9f26-37bb-46cf-b96a-43b7d6f631f4' exists
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Starting message stores for vhost 'Gamma Inc_f0fb9f26-37bb-46cf-b96a-43b7d6f631f4'
2022-02-10 09:07:25.621000+00:00 [info] <0.2041.0> Message store "A3BA4PLZT18AW5P11CSPMZMQ7/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Started message store of type transient for vhost 'Gamma Inc_f0fb9f26-37bb-46cf-b96a-43b7d6f631f4'
2022-02-10 09:07:25.621000+00:00 [info] <0.2045.0> Message store "A3BA4PLZT18AW5P11CSPMZMQ7/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Started message store of type persistent for vhost 'Gamma Inc_f0fb9f26-37bb-46cf-b96a-43b7d6f631f4'
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Recovering 1 queues of type rabbit_classic_queue took 14ms
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.621000+00:00 [info] <0.2037.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.636000+00:00 [info] <0.2231.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/4U8D4QO6F4U0X11KEDIR3LVER' for vhost 'Test Organisation 05_c20fde3f-1239-43ea-a8b8-fec38749b900' exists
2022-02-10 09:07:25.636000+00:00 [info] <0.2231.0> Starting message stores for vhost 'Test Organisation 05_c20fde3f-1239-43ea-a8b8-fec38749b900'
2022-02-10 09:07:25.636000+00:00 [info] <0.2235.0> Message store "4U8D4QO6F4U0X11KEDIR3LVER/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.652000+00:00 [info] <0.2231.0> Started message store of type transient for vhost 'Test Organisation 05_c20fde3f-1239-43ea-a8b8-fec38749b900'
2022-02-10 09:07:25.652000+00:00 [info] <0.2239.0> Message store "4U8D4QO6F4U0X11KEDIR3LVER/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.652000+00:00 [info] <0.2231.0> Started message store of type persistent for vhost 'Test Organisation 05_c20fde3f-1239-43ea-a8b8-fec38749b900'
2022-02-10 09:07:25.667000+00:00 [info] <0.2231.0> Recovering 2 queues of type rabbit_classic_queue took 29ms
2022-02-10 09:07:25.667000+00:00 [info] <0.2231.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.667000+00:00 [info] <0.2231.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.683000+00:00 [info] <0.2434.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/9ECP0X5P5E8L2FMKMMUMP6UMA' for vhost 'Test Organisation 04_0b39eb46-dba3-480e-a577-01311f4060df' exists
2022-02-10 09:07:25.683000+00:00 [info] <0.2434.0> Starting message stores for vhost 'Test Organisation 04_0b39eb46-dba3-480e-a577-01311f4060df'
2022-02-10 09:07:25.683000+00:00 [info] <0.2438.0> Message store "9ECP0X5P5E8L2FMKMMUMP6UMA/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.699000+00:00 [info] <0.2434.0> Started message store of type transient for vhost 'Test Organisation 04_0b39eb46-dba3-480e-a577-01311f4060df'
2022-02-10 09:07:25.699000+00:00 [info] <0.2442.0> Message store "9ECP0X5P5E8L2FMKMMUMP6UMA/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.699000+00:00 [info] <0.2434.0> Started message store of type persistent for vhost 'Test Organisation 04_0b39eb46-dba3-480e-a577-01311f4060df'
2022-02-10 09:07:25.714000+00:00 [info] <0.2434.0> Recovering 2 queues of type rabbit_classic_queue took 27ms
2022-02-10 09:07:25.714000+00:00 [info] <0.2434.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.714000+00:00 [info] <0.2434.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.730000+00:00 [info] <0.2637.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/1KPVY7GONUJVQ2H4UCWYVUXTK' for vhost 'Test Organisation 07_55c9cfb4-851d-496e-83e8-37c9bb3dae10' exists
2022-02-10 09:07:25.730000+00:00 [info] <0.2637.0> Starting message stores for vhost 'Test Organisation 07_55c9cfb4-851d-496e-83e8-37c9bb3dae10'
2022-02-10 09:07:25.730000+00:00 [info] <0.2641.0> Message store "1KPVY7GONUJVQ2H4UCWYVUXTK/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.730000+00:00 [info] <0.2637.0> Started message store of type transient for vhost 'Test Organisation 07_55c9cfb4-851d-496e-83e8-37c9bb3dae10'
2022-02-10 09:07:25.730000+00:00 [info] <0.2645.0> Message store "1KPVY7GONUJVQ2H4UCWYVUXTK/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.746000+00:00 [info] <0.2637.0> Started message store of type persistent for vhost 'Test Organisation 07_55c9cfb4-851d-496e-83e8-37c9bb3dae10'
2022-02-10 09:07:25.761000+00:00 [info] <0.2637.0> Recovering 2 queues of type rabbit_classic_queue took 31ms
2022-02-10 09:07:25.761000+00:00 [info] <0.2637.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.761000+00:00 [info] <0.2637.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.771000+00:00 [info] <0.2840.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/37M4BGH5SDCUBQTN6BFV5P5DT' for vhost 'Test Organisation 09_a842c83e-6ceb-4de2-84b9-096f9985b90f' exists
2022-02-10 09:07:25.771000+00:00 [info] <0.2840.0> Starting message stores for vhost 'Test Organisation 09_a842c83e-6ceb-4de2-84b9-096f9985b90f'
2022-02-10 09:07:25.771000+00:00 [info] <0.2844.0> Message store "37M4BGH5SDCUBQTN6BFV5P5DT/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.787000+00:00 [info] <0.2840.0> Started message store of type transient for vhost 'Test Organisation 09_a842c83e-6ceb-4de2-84b9-096f9985b90f'
2022-02-10 09:07:25.787000+00:00 [info] <0.2848.0> Message store "37M4BGH5SDCUBQTN6BFV5P5DT/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.787000+00:00 [info] <0.2840.0> Started message store of type persistent for vhost 'Test Organisation 09_a842c83e-6ceb-4de2-84b9-096f9985b90f'
2022-02-10 09:07:25.803000+00:00 [info] <0.2840.0> Recovering 2 queues of type rabbit_classic_queue took 27ms
2022-02-10 09:07:25.803000+00:00 [info] <0.2840.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.803000+00:00 [info] <0.2840.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.818000+00:00 [info] <0.3043.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/9QB6V9XM1K5FK7AO1TQU4MV0Q' for vhost 'System' exists
2022-02-10 09:07:25.818000+00:00 [info] <0.3043.0> Starting message stores for vhost 'System'
2022-02-10 09:07:25.818000+00:00 [info] <0.3047.0> Message store "9QB6V9XM1K5FK7AO1TQU4MV0Q/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.818000+00:00 [info] <0.3043.0> Started message store of type transient for vhost 'System'
2022-02-10 09:07:25.818000+00:00 [info] <0.3051.0> Message store "9QB6V9XM1K5FK7AO1TQU4MV0Q/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.834000+00:00 [info] <0.3043.0> Started message store of type persistent for vhost 'System'
2022-02-10 09:07:25.834000+00:00 [info] <0.3043.0> Recovering 1 queues of type rabbit_classic_queue took 14ms
2022-02-10 09:07:25.834000+00:00 [info] <0.3043.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.834000+00:00 [info] <0.3043.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.849000+00:00 [info] <0.3237.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/A6P3THTTGXBWE19ONLCKIYZLU' for vhost 'Test Organisation 03_abacd4da-e5c7-49a0-bfe2-4eb5bc0bb6ce' exists
2022-02-10 09:07:25.849000+00:00 [info] <0.3237.0> Starting message stores for vhost 'Test Organisation 03_abacd4da-e5c7-49a0-bfe2-4eb5bc0bb6ce'
2022-02-10 09:07:25.849000+00:00 [info] <0.3241.0> Message store "A6P3THTTGXBWE19ONLCKIYZLU/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.849000+00:00 [info] <0.3237.0> Started message store of type transient for vhost 'Test Organisation 03_abacd4da-e5c7-49a0-bfe2-4eb5bc0bb6ce'
2022-02-10 09:07:25.849000+00:00 [info] <0.3245.0> Message store "A6P3THTTGXBWE19ONLCKIYZLU/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.865000+00:00 [info] <0.3237.0> Started message store of type persistent for vhost 'Test Organisation 03_abacd4da-e5c7-49a0-bfe2-4eb5bc0bb6ce'
2022-02-10 09:07:25.881000+00:00 [info] <0.3237.0> Recovering 2 queues of type rabbit_classic_queue took 28ms
2022-02-10 09:07:25.881000+00:00 [info] <0.3237.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.881000+00:00 [info] <0.3237.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.881000+00:00 [info] <0.3440.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/CZC6PJ7PXS6HAIMRR0F3DKE9T' for vhost 'Beta Ltd_c1985a4e-6021-4389-a2eb-598ea7a44dec' exists
2022-02-10 09:07:25.896000+00:00 [info] <0.3440.0> Starting message stores for vhost 'Beta Ltd_c1985a4e-6021-4389-a2eb-598ea7a44dec'
2022-02-10 09:07:25.896000+00:00 [info] <0.3444.0> Message store "CZC6PJ7PXS6HAIMRR0F3DKE9T/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.896000+00:00 [info] <0.3440.0> Started message store of type transient for vhost 'Beta Ltd_c1985a4e-6021-4389-a2eb-598ea7a44dec'
2022-02-10 09:07:25.896000+00:00 [info] <0.3448.0> Message store "CZC6PJ7PXS6HAIMRR0F3DKE9T/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.896000+00:00 [info] <0.3440.0> Started message store of type persistent for vhost 'Beta Ltd_c1985a4e-6021-4389-a2eb-598ea7a44dec'
2022-02-10 09:07:25.912000+00:00 [info] <0.3440.0> Recovering 1 queues of type rabbit_classic_queue took 16ms
2022-02-10 09:07:25.912000+00:00 [info] <0.3440.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.912000+00:00 [info] <0.3440.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.912000+00:00 [info] <0.3634.0> Making sure data directory 'c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia/msg_stores/vhosts/CXS58YHQT6FQ5YFXAOJ3UW4LQ' for vhost 'Message Sequence Org_1742f98b-fce3-48c4-a1a3-317855a04f4a' exists
2022-02-10 09:07:25.928000+00:00 [info] <0.3634.0> Starting message stores for vhost 'Message Sequence Org_1742f98b-fce3-48c4-a1a3-317855a04f4a'
2022-02-10 09:07:25.928000+00:00 [info] <0.3638.0> Message store "CXS58YHQT6FQ5YFXAOJ3UW4LQ/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.928000+00:00 [info] <0.3634.0> Started message store of type transient for vhost 'Message Sequence Org_1742f98b-fce3-48c4-a1a3-317855a04f4a'
2022-02-10 09:07:25.928000+00:00 [info] <0.3642.0> Message store "CXS58YHQT6FQ5YFXAOJ3UW4LQ/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-02-10 09:07:25.943000+00:00 [info] <0.3634.0> Started message store of type persistent for vhost 'Message Sequence Org_1742f98b-fce3-48c4-a1a3-317855a04f4a'
2022-02-10 09:07:25.943000+00:00 [info] <0.3634.0> Recovering 1 queues of type rabbit_classic_queue took 20ms
2022-02-10 09:07:25.943000+00:00 [info] <0.3634.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2022-02-10 09:07:25.943000+00:00 [info] <0.3634.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step empty_db_check defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Will not seed default virtual host and user: have definitions to load...
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step rabbit_looking_glass defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step background_gc defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step routing_ready defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step pre_flight defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step notify_cluster defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step networking defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step definition_import_worker_pool defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.358.0> Starting worker pool 'definition_import_pool' with 12 processes in it
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step cluster_name defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step direct_client defined by app rabbit
2022-02-10 09:07:25.953000+00:00 [info] <0.226.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2022-02-10 09:07:25.953000+00:00 [info] <0.3845.0> Resetting node maintenance status
2022-02-10 09:07:26.088000+00:00 [info] <0.3904.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2022-02-10 09:07:26.088000+00:00 [info] <0.3932.0> Statistics database started.
2022-02-10 09:07:26.088000+00:00 [info] <0.3931.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2022-02-10 09:07:26.104000+00:00 [info] <0.3946.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2022-02-10 09:07:26.104000+00:00 [info] <0.3845.0> Ready to start client connection listeners
2022-02-10 09:07:26.104000+00:00 [info] <0.3990.0> started TCP listener on [::]:5672
2022-02-10 09:07:26.104000+00:00 [info] <0.4008.0> started TCP listener on 0.0.0.0:5672
2022-02-10 09:07:26.198000+00:00 [info] <0.3845.0> Server startup complete; 4 plugins started.
2022-02-10 09:07:26.198000+00:00 [info] <0.3845.0>  * rabbitmq_prometheus
2022-02-10 09:07:26.198000+00:00 [info] <0.3845.0>  * rabbitmq_management
2022-02-10 09:07:26.198000+00:00 [info] <0.3845.0>  * rabbitmq_web_dispatch
2022-02-10 09:07:26.198000+00:00 [info] <0.3845.0>  * rabbitmq_management_agent
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> Command timed out: 'powershell.exe -NoLogo -NoProfile -NonInteractive -Command (Get-PSDrive c).Free'
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> ** Generic server rabbit_disk_monitor terminating
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> ** Last message in was update
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> ** When Server state == {state,"c:/Users/Username/AppData/Roaming/RabbitMQ/db/rabbit@MachineName-mnesia",
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>                                50000000,162253606912,100,10000,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>                                #Ref<0.727867240.1590427650.89021>,false,true,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>                                10,120000}
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> ** Reason for termination ==
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0> ** {could_not_determine_disk_free,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>        [{rabbit_disk_monitor,get_disk_free,2,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>             [{file,"rabbit_disk_monitor.erl"},{line,257}]},
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>         {rabbit_disk_monitor,internal_update,1,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>             [{file,"rabbit_disk_monitor.erl"},{line,216}]},
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>         {rabbit_disk_monitor,handle_info,2,
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>             [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>   crasher:
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     initial call: rabbit_disk_monitor:init/1
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     pid: <0.368.0>
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     registered_name: rabbit_disk_monitor
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     exception exit: could_not_determine_disk_free
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>       in function  rabbit_disk_monitor:get_disk_free/2 (rabbit_disk_monitor.erl, line 257)
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>       in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 216)
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>       in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 771)
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.226.0>]
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     message_queue_len: 0
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     messages: []
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     links: [<0.367.0>]
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     dictionary: []
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     trap_exit: false
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     status: running
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     heap_size: 10958
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     stack_size: 29
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>     reductions: 35904
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>   neighbours:
2022-02-10 09:26:03.126000+00:00 [error] <0.368.0>
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>     supervisor: {local,rabbit_disk_monitor_sup}
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>     errorContext: child_terminated
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>     reason: could_not_determine_disk_free
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>     offender: [{pid,<0.368.0>},
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>                {id,rabbit_disk_monitor},
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>                {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>                {restart_type,{transient,1}},
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>                {shutdown,300000},
2022-02-10 09:26:03.126000+00:00 [error] <0.367.0>                {child_type,worker}]
2022-02-10 09:26:04.594000+00:00 [info] <0.4794.0> Enabling free disk space monitoring
2022-02-10 09:26:04.594000+00:00 [info] <0.4794.0> Disk free limit set to 50MB

Luke Bakken

unread,
Feb 11, 2022, 1:39:48 PM2/11/22
to rabbitmq-users
Hello,

Going forward please attach log files to your messages to this group rather than pasting an enormous amount of text.

Unless this happens continuously (every minute or so) I wouldn't be too concerned. Apparently starting up and running powershell takes a bit more time than we allow (5 seconds). We may have to adjust that. Is RabbitMQ running on a low-powered VM or under a lot of load, that sort of thing?

Thanks for the report -
Luke

Damian TagDev

unread,
Feb 18, 2022, 6:19:02 AM2/18/22
to rabbitm...@googlegroups.com
Hi Luke,

It doesn't happen continuously, but consistently everyday, usually a short while after logging in to my laptop. This is running on real hardware - albeit a laptop, 6 core (12 thread) i7 8850H, nvme 512GB ssd, 16GB RAM. I wouldn't say it's under load as such but the laptop is 3 years old. I can't see anything significant in the Windows Event logs that would correspond with the time of the error to suggest it was under load.

Is the powershell timeout configurable? If so I could try a few different times to see what works.

Kind Regards
Damian

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/WwRMEs08MOo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e6d60f01-6c01-45ff-864f-d0a6a25e6028n%40googlegroups.com.

Luke Bakken

unread,
Feb 18, 2022, 11:32:53 AM2/18/22
to rabbitmq-users
Hi Damian,

Thanks for the additional information. There isn't a timeout that can be adjusted at this point in time. I have this on my very long list of items to look into.

Have a nice weekend!
Luke

Luke Bakken

unread,
Feb 18, 2022, 12:14:45 PM2/18/22
to rabbitmq-users

PS I did open an issue here if you'd like to follow it - https://github.com/rabbitmq/rabbitmq-server/issues/4140
Reply all
Reply to author
Forward
0 new messages