RabbitMQ 3.11.4 could not start on k8s
897 views
Skip to first unread message
Leanid Kopats
Dec 13, 2022, 3:26:36 AM12/13/22
to rabbitmq-users
Hi all!
I have app deployed to azure k8s with RabbitMQ.
After upgrade rabbit version from 3.10.8 to 3.11.4, I got the following error:
BOOT FAILED
===========
Exception during startup:
exit:{error,not_json}
rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 45
rabbit_boot_steps:run_step/2, line 48
rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
rabbit_boot_steps:run_boot_steps/1, line 20
rabbit:start/2, line 924
application_master:start_it_old/4, line 293
But upgrading from 3.10.8 to 3.10.12 is fine
So, does anyone getting the same error? How it can be overcome?
Michal Kuratczyk
Dec 13, 2022, 4:59:14 AM12/13/22
to rabbitm...@googlegroups.com
Hi,
To overcome this error, fix your definitions file and make it a valid JSON. :)
In 3.11 we changed the JSON parser. The new one is much faster but also more strict. Almost certainly your definitions file is invalid in a subtle way and
the old parser accepted it but the new one (rightfully) doesn't.
Best,
This email message, including attachments, may contain information that is proprietary, confidential, privileged and/or exempt from disclosure. Please hold it in confidence to protect privilege and confidentiality. Please be advised that the sender of this message is a licensee of SDVentures and shall not be construed or deemed to be an employee, agent, partner, associate or joint venturer of SDVentures. If you are not the intended recipient, then please notify the sender and delete this message. Any viewing, copying, publishing, disclosure, distribution of this information, or the taking of any action in reliance on the contents of this message by unintended recipients is prohibited. This message cannot be guaranteed to be secure or error-free.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7bbcd465-b5c7-4896-8f6c-b37ba4ff5894n%40googlegroups.com.
Michał
RabbitMQ teamLeanid Kopats
Dec 13, 2022, 5:27:10 AM12/13/22
to rabbitmq-users
Ok, we will try. Thank you for your time!
Himanshu Swami
May 8, 2023, 4:44:03 PM5/8/23
to rabbitmq-users
Hi Michal,
This is my definitions.json and for the life of me I cannot figure out what is wrong with it. I have run it through a few validators online and none of them report any issues. I have redacted password_hash and global_parameters["cluster_name"].value for obvious reasons but otherwise the file is what we have been using until 3.10.8:
{
"users": [{
"name": "guest",
"password_hash": "<hash>",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}],
"vhosts": [{
"name": "/"
}],
"permissions": [{
"user": "guest",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}],
"topic_permissions": [],
"parameters": [],
"global_parameters": [{
"name": "cluster_name",
"value": "<cluster_name>"
}],
"policies": [],
"queues": [{
"name": "Audit",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Batch",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Communication",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Computes",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "DataSyncCQRS",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Printing",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "ReferenceData",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Report",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Security",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Submission",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
}
],
"exchanges": [{
"name": "sacs_event_bus",
"vhost": "/",
"type": "direct",
"durable": false,
"auto_delete": false,
"internal": false,
"arguments": {}
}]
}
{
"users": [{
"name": "guest",
"password_hash": "<hash>",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}],
"vhosts": [{
"name": "/"
}],
"permissions": [{
"user": "guest",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}],
"topic_permissions": [],
"parameters": [],
"global_parameters": [{
"name": "cluster_name",
"value": "<cluster_name>"
}],
"policies": [],
"queues": [{
"name": "Audit",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Batch",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Communication",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Computes",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "DataSyncCQRS",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Printing",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "ReferenceData",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Report",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Security",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "Submission",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
}
],
"exchanges": [{
"name": "sacs_event_bus",
"vhost": "/",
"type": "direct",
"durable": false,
"auto_delete": false,
"internal": false,
"arguments": {}
}]
}
Himanshu Swami
May 8, 2023, 4:45:04 PM5/8/23
to rabbitmq-users
Hey Leanid, did you ever figure out what was wrong with your JSON file? Grasping at straws here ...
Himanshu Swami
May 8, 2023, 6:01:05 PM5/8/23
to rabbitmq-users
This is the log output:
2023-05-08 14:59:58 2023-05-08 21:59:58.751082+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-05-08 14:59:58 2023-05-08 21:59:58.757154+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-05-08 14:59:58 2023-05-08 21:59:58.772500+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE
2023-05-08 15:00:01 2023-05-08 22:00:01.460500+00:00 [info] <0.230.0> ra: starting system quorum_queues
2023-05-08 15:00:01 2023-05-08 22:00:01.460604+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/quorum/rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.461903+00:00 [info] <0.301.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-05-08 15:00:01 2023-05-08 22:00:01.463133+00:00 [info] <0.304.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-05-08 15:00:01 2023-05-08 22:00:01.463290+00:00 [notice] <0.313.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-05-08 15:00:01 2023-05-08 22:00:01.468730+00:00 [info] <0.230.0> ra: starting system coordination
2023-05-08 15:00:01 2023-05-08 22:00:01.468770+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/coordination/rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.469829+00:00 [info] <0.346.0> ra system 'coordination' running pre init for 0 registered servers
2023-05-08 15:00:01 2023-05-08 22:00:01.470839+00:00 [info] <0.347.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2023-05-08 15:00:01 2023-05-08 22:00:01.471012+00:00 [notice] <0.352.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0>
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Starting RabbitMQ 3.11.15 on Erlang 25.3.1 [jit]
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-05-08 15:00:01
2023-05-08 15:00:01 ## ## RabbitMQ 3.11.15
2023-05-08 15:00:01 ## ##
2023-05-08 15:00:01 ########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-05-08 15:00:01 ###### ##
2023-05-08 15:00:01 ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-05-08 15:00:01
2023-05-08 15:00:01 Erlang: 25.3.1 [jit]
2023-05-08 15:00:01 TLS Library: OpenSSL - OpenSSL 3.0.8 7 Feb 2023
2023-05-08 15:00:01 Release series support status: supported
2023-05-08 15:00:01
2023-05-08 15:00:01 Doc guides: https://rabbitmq.com/documentation.html
2023-05-08 15:00:01 Support: https://rabbitmq.com/contact.html
2023-05-08 15:00:01 Tutorials: https://rabbitmq.com/getstarted.html
2023-05-08 15:00:01 Monitoring: https://rabbitmq.com/monitoring.html
2023-05-08 15:00:01
2023-05-08 15:00:01 Logs: /var/log/rabbitmq/rabbit@rabbitmq_upgrade.log
2023-05-08 15:00:01 <stdout>
2023-05-08 15:00:01
2023-05-08 15:00:01 Config file(s): /etc/rabbitmq/rabbitmq.conf
2023-05-08 15:00:01 /etc/rabbitmq/conf.d/10-defaults.conf
2023-05-08 15:00:01
2023-05-08 15:00:01 Starting broker...2023-05-08 22:00:01.478099+00:00 [info] <0.230.0>
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> node : rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> home dir : /var/lib/rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> config file(s) : /etc/rabbitmq/rabbitmq.conf
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/10-defaults.conf
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> cookie hash : nXtfVjK1VEJ4m2oA5EUvNw==
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> log(s) : /var/log/rabbitmq/rabbit@rabbitmq_upgrade.log
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> : <stdout>
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
2023-05-08 15:00:02 2023-05-08 22:00:02.066337+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066430+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066737+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066835+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.067535+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.071279+00:00 [info] <0.367.0> Memory high watermark set to 793 MiB (831635456 bytes) of 1982 MiB (2079088640 bytes) total
2023-05-08 15:00:02 2023-05-08 22:00:02.074118+00:00 [info] <0.369.0> Enabling free disk space monitoring (disk free space: 59586543616, total memory: 2079088640)
2023-05-08 15:00:02 2023-05-08 22:00:02.074174+00:00 [info] <0.369.0> Disk free limit set to 50MB
2023-05-08 15:00:02 2023-05-08 22:00:02.075645+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.075731+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.075925+00:00 [info] <0.372.0> Limiting to approx 1048479 file handles (943629 sockets)
2023-05-08 15:00:02 2023-05-08 22:00:02.076025+00:00 [info] <0.373.0> FHC read buffering: OFF
2023-05-08 15:00:02 2023-05-08 22:00:02.076061+00:00 [info] <0.373.0> FHC write buffering: ON
2023-05-08 15:00:02 2023-05-08 22:00:02.076449+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.076567+00:00 [info] <0.354.0> Will use 12 processes for default worker pool
2023-05-08 15:00:02 2023-05-08 22:00:02.076608+00:00 [info] <0.354.0> Starting worker pool 'worker_pool' with 12 processes in it
2023-05-08 15:00:02 2023-05-08 22:00:02.077099+00:00 [info] <0.230.0> Running boot step database defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.078508+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-08 15:00:02 2023-05-08 22:00:02.081267+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-08 15:00:02 2023-05-08 22:00:02.093228+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-08 15:00:02 2023-05-08 22:00:02.093341+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-08 15:00:02 2023-05-08 22:00:02.093371+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2023-05-08 15:00:02 2023-05-08 22:00:02.093511+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.093615+00:00 [info] <0.394.0> Setting up a table for connection tracking on this node: tracked_connection
2023-05-08 15:00:02 2023-05-08 22:00:02.093657+00:00 [info] <0.394.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2023-05-08 15:00:02 2023-05-08 22:00:02.093706+00:00 [info] <0.394.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2023-05-08 15:00:02 2023-05-08 22:00:02.093777+00:00 [info] <0.394.0> Setting up a table for channel tracking on this node: tracked_channel
2023-05-08 15:00:02 2023-05-08 22:00:02.093811+00:00 [info] <0.394.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2023-05-08 15:00:02 2023-05-08 22:00:02.093861+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.093981+00:00 [info] <0.230.0> Running boot step database_sync defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094080+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094239+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094314+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094355+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094459+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094643+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094741+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094820+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094855+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094882+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094915+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094949+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094980+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095082+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095209+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095325+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095409+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue
2023-05-08 15:00:02 2023-05-08 22:00:02.095471+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095549+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095631+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095690+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095712+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095882+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.096674+00:00 [info] <0.403.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2023-05-08 15:00:02 2023-05-08 22:00:02.096803+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.100188+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.100417+00:00 [info] <0.407.0> Starting rabbit_node_monitor
2023-05-08 15:00:02 2023-05-08 22:00:02.100632+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101143+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101408+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101525+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101553+00:00 [info] <0.230.0> Running boot step upgrade_queues defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108762+00:00 [info] <0.230.0> Running boot step channel_tracking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108836+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108889+00:00 [info] <0.230.0> Running boot step connection_tracking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108937+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108992+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109075+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109154+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109494+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109735+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109794+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109828+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109939+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.110122+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.110219+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2023-05-08 15:00:02 2023-05-08 22:00:02.110298+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2023-05-08 15:00:02 2023-05-08 22:00:02.110340+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'
2023-05-08 15:00:02 2023-05-08 22:00:02.110830+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111758+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111803+00:00 [info] <0.230.0> Will not seed default virtual host and user: have definitions to load...
2023-05-08 15:00:02 2023-05-08 22:00:02.111828+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111882+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111906+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112047+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112243+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112330+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112388+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112416+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112443+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112502+00:00 [info] <0.354.0> Starting worker pool 'definition_import_pool' with 12 processes in it
2023-05-08 15:00:02 2023-05-08 22:00:02.113361+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113410+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113505+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113548+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
2023-05-08 15:00:02 2023-05-08 22:00:02.113733+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2023-05-08 15:00:02 2023-05-08 22:00:02.113867+00:00 [info] <0.230.0> Applying definitions from regular file at /etc/rabbitmq/definitions.json
2023-05-08 15:00:02 2023-05-08 22:00:02.115866+00:00 [info] <0.230.0> Applying definitions from file at '/etc/rabbitmq/definitions.json'
2023-05-08 15:00:02 2023-05-08 22:00:02.115944+00:00 [info] <0.230.0> Asked to import definitions. Acting user: rmq-internal
2023-05-08 15:00:02 2023-05-08 22:00:02.118959+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-05-08 15:00:02
2023-05-08 15:00:02 BOOT FAILED
2023-05-08 15:00:02 ===========
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> BOOT FAILED
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> ===========
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> Exception during startup:
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> exit:{error,not_json}
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 45
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:run_step/2, line 48
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:run_boot_steps/1, line 20
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit:start/2, line 958
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> application_master:start_it_old/4, line 293
2023-05-08 15:00:02 Exception during startup:
2023-05-08 15:00:02
2023-05-08 15:00:02 exit:{error,not_json}
2023-05-08 15:00:02
2023-05-08 15:00:02 rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 45
2023-05-08 15:00:02 rabbit_boot_steps:run_step/2, line 48
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2023-05-08 15:00:02 rabbit_boot_steps:run_boot_steps/1, line 20
2023-05-08 15:00:02 rabbit:start/2, line 958
2023-05-08 15:00:02 application_master:start_it_old/4, line 293
2023-05-08 15:00:02
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> crasher:
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> initial call: application_master:init/4
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> pid: <0.229.0>
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> registered_name: []
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> exception exit: {{error,not_json},{rabbit,start,[normal,[]]}}
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> in function application_master:init/4 (application_master.erl, line 142)
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> ancestors: [<0.228.0>]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> message_queue_len: 1
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> messages: [{'EXIT',<0.230.0>,normal}]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> links: [<0.228.0>,<0.44.0>]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> dictionary: []
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> trap_exit: true
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> status: running
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> heap_size: 2586
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> stack_size: 28
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> reductions: 176
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> neighbours:
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0>
2023-05-08 15:00:03 2023-05-08 22:00:03.121353+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{error,not_json},{rabbit,start,[normal,[]]}}
2023-05-08 15:00:03 2023-05-08 22:00:03.121943+00:00 [info] <0.398.0> Management plugin: to stop collect_statistics.
2023-05-08 15:00:04 Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{error,not_json},{rabbit,start,[normal,[]]}}})
2023-05-08 15:00:04
2023-05-08 15:00:04 {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{error,not_json},{rabbit,start,[normal,[]]}}}"}
2023-05-08 15:00:04 Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
2023-05-08 14:59:58 2023-05-08 21:59:58.751082+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-05-08 14:59:58 2023-05-08 21:59:58.757154+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-05-08 14:59:58 2023-05-08 21:59:58.772500+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE
2023-05-08 15:00:01 2023-05-08 22:00:01.460500+00:00 [info] <0.230.0> ra: starting system quorum_queues
2023-05-08 15:00:01 2023-05-08 22:00:01.460604+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/quorum/rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.461903+00:00 [info] <0.301.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-05-08 15:00:01 2023-05-08 22:00:01.463133+00:00 [info] <0.304.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-05-08 15:00:01 2023-05-08 22:00:01.463290+00:00 [notice] <0.313.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-05-08 15:00:01 2023-05-08 22:00:01.468730+00:00 [info] <0.230.0> ra: starting system coordination
2023-05-08 15:00:01 2023-05-08 22:00:01.468770+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq/coordination/rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.469829+00:00 [info] <0.346.0> ra system 'coordination' running pre init for 0 registered servers
2023-05-08 15:00:01 2023-05-08 22:00:01.470839+00:00 [info] <0.347.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2023-05-08 15:00:01 2023-05-08 22:00:01.471012+00:00 [notice] <0.352.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0>
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Starting RabbitMQ 3.11.15 on Erlang 25.3.1 [jit]
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-05-08 15:00:01 2023-05-08 22:00:01.475162+00:00 [info] <0.230.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-05-08 15:00:01
2023-05-08 15:00:01 ## ## RabbitMQ 3.11.15
2023-05-08 15:00:01 ## ##
2023-05-08 15:00:01 ########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-05-08 15:00:01 ###### ##
2023-05-08 15:00:01 ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-05-08 15:00:01
2023-05-08 15:00:01 Erlang: 25.3.1 [jit]
2023-05-08 15:00:01 TLS Library: OpenSSL - OpenSSL 3.0.8 7 Feb 2023
2023-05-08 15:00:01 Release series support status: supported
2023-05-08 15:00:01
2023-05-08 15:00:01 Doc guides: https://rabbitmq.com/documentation.html
2023-05-08 15:00:01 Support: https://rabbitmq.com/contact.html
2023-05-08 15:00:01 Tutorials: https://rabbitmq.com/getstarted.html
2023-05-08 15:00:01 Monitoring: https://rabbitmq.com/monitoring.html
2023-05-08 15:00:01
2023-05-08 15:00:01 Logs: /var/log/rabbitmq/rabbit@rabbitmq_upgrade.log
2023-05-08 15:00:01 <stdout>
2023-05-08 15:00:01
2023-05-08 15:00:01 Config file(s): /etc/rabbitmq/rabbitmq.conf
2023-05-08 15:00:01 /etc/rabbitmq/conf.d/10-defaults.conf
2023-05-08 15:00:01
2023-05-08 15:00:01 Starting broker...2023-05-08 22:00:01.478099+00:00 [info] <0.230.0>
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> node : rabbit@rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> home dir : /var/lib/rabbitmq
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> config file(s) : /etc/rabbitmq/rabbitmq.conf
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/10-defaults.conf
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> cookie hash : nXtfVjK1VEJ4m2oA5EUvNw==
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> log(s) : /var/log/rabbitmq/rabbit@rabbitmq_upgrade.log
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> : <stdout>
2023-05-08 15:00:01 2023-05-08 22:00:01.478099+00:00 [info] <0.230.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
2023-05-08 15:00:02 2023-05-08 22:00:02.066337+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066430+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066737+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.066835+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.067535+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.071279+00:00 [info] <0.367.0> Memory high watermark set to 793 MiB (831635456 bytes) of 1982 MiB (2079088640 bytes) total
2023-05-08 15:00:02 2023-05-08 22:00:02.074118+00:00 [info] <0.369.0> Enabling free disk space monitoring (disk free space: 59586543616, total memory: 2079088640)
2023-05-08 15:00:02 2023-05-08 22:00:02.074174+00:00 [info] <0.369.0> Disk free limit set to 50MB
2023-05-08 15:00:02 2023-05-08 22:00:02.075645+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.075731+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.075925+00:00 [info] <0.372.0> Limiting to approx 1048479 file handles (943629 sockets)
2023-05-08 15:00:02 2023-05-08 22:00:02.076025+00:00 [info] <0.373.0> FHC read buffering: OFF
2023-05-08 15:00:02 2023-05-08 22:00:02.076061+00:00 [info] <0.373.0> FHC write buffering: ON
2023-05-08 15:00:02 2023-05-08 22:00:02.076449+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.076567+00:00 [info] <0.354.0> Will use 12 processes for default worker pool
2023-05-08 15:00:02 2023-05-08 22:00:02.076608+00:00 [info] <0.354.0> Starting worker pool 'worker_pool' with 12 processes in it
2023-05-08 15:00:02 2023-05-08 22:00:02.077099+00:00 [info] <0.230.0> Running boot step database defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.078508+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-08 15:00:02 2023-05-08 22:00:02.081267+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-08 15:00:02 2023-05-08 22:00:02.093228+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-08 15:00:02 2023-05-08 22:00:02.093341+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-08 15:00:02 2023-05-08 22:00:02.093371+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2023-05-08 15:00:02 2023-05-08 22:00:02.093511+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.093615+00:00 [info] <0.394.0> Setting up a table for connection tracking on this node: tracked_connection
2023-05-08 15:00:02 2023-05-08 22:00:02.093657+00:00 [info] <0.394.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2023-05-08 15:00:02 2023-05-08 22:00:02.093706+00:00 [info] <0.394.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2023-05-08 15:00:02 2023-05-08 22:00:02.093777+00:00 [info] <0.394.0> Setting up a table for channel tracking on this node: tracked_channel
2023-05-08 15:00:02 2023-05-08 22:00:02.093811+00:00 [info] <0.394.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2023-05-08 15:00:02 2023-05-08 22:00:02.093861+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.093981+00:00 [info] <0.230.0> Running boot step database_sync defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094080+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094239+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094314+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094355+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094459+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094643+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094741+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094820+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094855+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094882+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094915+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094949+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.094980+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095082+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095209+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095325+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095409+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue
2023-05-08 15:00:02 2023-05-08 22:00:02.095471+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095549+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095631+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095690+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095712+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.095882+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.096674+00:00 [info] <0.403.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2023-05-08 15:00:02 2023-05-08 22:00:02.096803+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.100188+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.100417+00:00 [info] <0.407.0> Starting rabbit_node_monitor
2023-05-08 15:00:02 2023-05-08 22:00:02.100632+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101143+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101408+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101525+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.101553+00:00 [info] <0.230.0> Running boot step upgrade_queues defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108762+00:00 [info] <0.230.0> Running boot step channel_tracking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108836+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108889+00:00 [info] <0.230.0> Running boot step connection_tracking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108937+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.108992+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109075+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109154+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109494+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109735+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109794+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109828+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.109939+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.110122+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.110219+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2023-05-08 15:00:02 2023-05-08 22:00:02.110298+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2023-05-08 15:00:02 2023-05-08 22:00:02.110340+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'
2023-05-08 15:00:02 2023-05-08 22:00:02.110830+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111758+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111803+00:00 [info] <0.230.0> Will not seed default virtual host and user: have definitions to load...
2023-05-08 15:00:02 2023-05-08 22:00:02.111828+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111882+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.111906+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112047+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112243+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112330+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112388+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112416+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112443+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.112502+00:00 [info] <0.354.0> Starting worker pool 'definition_import_pool' with 12 processes in it
2023-05-08 15:00:02 2023-05-08 22:00:02.113361+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113410+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113505+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
2023-05-08 15:00:02 2023-05-08 22:00:02.113548+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
2023-05-08 15:00:02 2023-05-08 22:00:02.113733+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2023-05-08 15:00:02 2023-05-08 22:00:02.113867+00:00 [info] <0.230.0> Applying definitions from regular file at /etc/rabbitmq/definitions.json
2023-05-08 15:00:02 2023-05-08 22:00:02.115866+00:00 [info] <0.230.0> Applying definitions from file at '/etc/rabbitmq/definitions.json'
2023-05-08 15:00:02 2023-05-08 22:00:02.115944+00:00 [info] <0.230.0> Asked to import definitions. Acting user: rmq-internal
2023-05-08 15:00:02 2023-05-08 22:00:02.118959+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-05-08 15:00:02
2023-05-08 15:00:02 BOOT FAILED
2023-05-08 15:00:02 ===========
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> BOOT FAILED
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> ===========
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> Exception during startup:
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> exit:{error,not_json}
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 45
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:run_step/2, line 48
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit_boot_steps:run_boot_steps/1, line 20
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> rabbit:start/2, line 958
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0> application_master:start_it_old/4, line 293
2023-05-08 15:00:02 Exception during startup:
2023-05-08 15:00:02
2023-05-08 15:00:02 exit:{error,not_json}
2023-05-08 15:00:02
2023-05-08 15:00:02 rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 45
2023-05-08 15:00:02 rabbit_boot_steps:run_step/2, line 48
2023-05-08 15:00:02 2023-05-08 22:00:02.119226+00:00 [error] <0.230.0>
2023-05-08 15:00:02 rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 19
2023-05-08 15:00:02 rabbit_boot_steps:run_boot_steps/1, line 20
2023-05-08 15:00:02 rabbit:start/2, line 958
2023-05-08 15:00:02 application_master:start_it_old/4, line 293
2023-05-08 15:00:02
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> crasher:
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> initial call: application_master:init/4
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> pid: <0.229.0>
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> registered_name: []
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> exception exit: {{error,not_json},{rabbit,start,[normal,[]]}}
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> in function application_master:init/4 (application_master.erl, line 142)
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> ancestors: [<0.228.0>]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> message_queue_len: 1
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> messages: [{'EXIT',<0.230.0>,normal}]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> links: [<0.228.0>,<0.44.0>]
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> dictionary: []
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> trap_exit: true
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> status: running
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> heap_size: 2586
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> stack_size: 28
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> reductions: 176
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0> neighbours:
2023-05-08 15:00:03 2023-05-08 22:00:03.121003+00:00 [error] <0.229.0>
2023-05-08 15:00:03 2023-05-08 22:00:03.121353+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{error,not_json},{rabbit,start,[normal,[]]}}
2023-05-08 15:00:03 2023-05-08 22:00:03.121943+00:00 [info] <0.398.0> Management plugin: to stop collect_statistics.
2023-05-08 15:00:04 Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{error,not_json},{rabbit,start,[normal,[]]}}})
2023-05-08 15:00:04
2023-05-08 15:00:04 {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{error,not_json},{rabbit,start,[normal,[]]}}}"}
2023-05-08 15:00:04 Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
Leanid Kopats
May 9, 2023, 2:49:43 AM5/9/23
to rabbitmq-users
Hi Himanshu!
Honestly, I don't know exactly what was wrong with my json. New DevOps guy just came and fix it :)
if I remember correctly, there was some tabulation misses - extra tabs either absent tabs in lines
Michal Kuratczyk
May 9, 2023, 4:24:25 AM5/9/23
to rabbitm...@googlegroups.com
I copy-pasted your file, put a valid hash in password_hash and I can import it just fine. I guess it can be some strange characters or encoding
issue that gets lost when you copy-paste it into an email and/or when I copy-paste it from here.
If you still can't find it, provide a repo with the exact files and steps that fail for you.
Best,
This email message, including attachments, may contain information that is proprietary, confidential, privileged and/or exempt from disclosure. If you are not the intended recipient, then please notify the sender and delete this message. Any viewing, copying, publishing, disclosure, distribution of this information, or the taking of any action in reliance on the contents of this message by unintended recipients is prohibited. This message cannot be guaranteed to be secure or error-free.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/39c08f04-4bd2-45ca-9b25-6ad608129862n%40googlegroups.com.
Michał
RabbitMQ team
Luke Bakken
May 10, 2023, 11:08:52 AM5/10/23
to rabbitmq-users
Hello,
Luke Bakken
May 10, 2023, 11:10:30 AM5/10/23
to rabbitmq-users
Note the first two bytes (efbb, the BOM) and the CRLF line endings (0d0a)
C:\Users\bakkenl\Downloads
> xxd -l 128 .\definitions.json
00000000: efbb bf7b 0d0a 2020 2272 6162 6269 745f ...{.. "rabbit_
00000010: 7665 7273 696f 6e22 3a20 2233 2e38 2e30 version": "3.8.0
00000020: 222c 0d0a 2020 2275 7365 7273 223a 205b ",.. "users": [
00000030: 7b0d 0a20 2020 2022 6e61 6d65 223a 2022 {.. "name": "
00000040: 6775 6573 7422 2c0d 0a20 2020 2022 7061 guest",.. "pa
00000050: 7373 776f 7264 5f68 6173 6822 3a20 223c ssword_hash": "<
00000060: 6861 7368 3e22 2c0d 0a20 2020 2022 6861 hash>",.. "ha
00000070: 7368 696e 675f 616c 676f 7269 7468 6d22 shing_algorithm"
> xxd -l 128 .\definitions.json
00000000: efbb bf7b 0d0a 2020 2272 6162 6269 745f ...{.. "rabbit_
00000010: 7665 7273 696f 6e22 3a20 2233 2e38 2e30 version": "3.8.0
00000020: 222c 0d0a 2020 2275 7365 7273 223a 205b ",.. "users": [
00000030: 7b0d 0a20 2020 2022 6e61 6d65 223a 2022 {.. "name": "
00000040: 6775 6573 7422 2c0d 0a20 2020 2022 7061 guest",.. "pa
00000050: 7373 776f 7264 5f68 6173 6822 3a20 223c ssword_hash": "<
00000060: 6861 7368 3e22 2c0d 0a20 2020 2022 6861 hash>",.. "ha
00000070: 7368 696e 675f 616c 676f 7269 7468 6d22 shing_algorithm"
Himanshu Swami
May 10, 2023, 5:07:57 PM5/10/23
to rabbitmq-users
THANK YOU SO MUCH! I was about to download erlang and figure out how to run that parser on this file ... so appreciate the assist HUGELY!
So key takeaway - use Unix-style line-endings and ensure there is no BOM.
So key takeaway - use Unix-style line-endings and ensure there is no BOM.
Reply all
Reply to author
Forward
0 new messages