Vhost corruption after restart
439 views
Skip to first unread message
Daniel Krebs
Sep 20, 2021, 11:07:35 AM9/20/21
to rabbitmq-users
Hi list,
we have now experienced for the 2nd time a vhost corruption after restarting our RabbitMQ broker. We're currently running 3.8.22, but essentially the same thing has already happened in July on 3.8.15. I am running a bit out of ideas where to look for solutions, so I hope someone can help.
I have assembled all the available logs and exact package information of our current setup here: https://gist.github.com/daniel-k/e78d3ba808b9422df326b91824ce5bf4
We basically run 2 vhost: "dev" and "live". Luckily, this issue happened both times in the dev vhost, but that might have been pure luck. In order to repair it, I couldn't help but remove the whole vhost and recreate it based on a backup of the definitions.
While both vhost are more or less identical w.r.t. to queues, exchanges and the messages passing through them, the "dev" vhost has some additional queues with a TTL set. I've read about some bug related with TTL here on the list recently, so I thought it might be worth mentioning.
I have tried to reproduce the issue locally with TTL queues on a Docker setup (same Ubuntu/RabbitMQ/Erlang versions) and publishing at high rates while restarting the broker constantly, but to no avail so far.
Any help would be very much appreciated!
Best regards,
Daniel Krebs
jo...@cloudamqp.com
Sep 21, 2021, 11:01:52 AM9/21/21
to rabbitmq-users
Hi,
Ouch! OK, from the log line:
2021-09-17 09:25:13.381 [error] <0.823.0> Failed to start message store of type msg_store_persistent for vhost 'xxx-dev': {{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{no_pub,no_del,ack},
it looks a lot like https://github.com/rabbitmq/rabbitmq-server/discussions/3272, in which we (CloudAMQP) couldn't replicate the bug (3.8.21 and higher+)
Looking at the data it is related in your case to the vhost "enlyze-dev" and the exchange "gippy-data", ttl used for appliance uuid 788acd91-2e5d-4d45-87df-c892fca08482 might help in tracking this down.
Your rabbitmq.config and "rabbitmqctl environment" might be helpful.
(NB: You do not need to fully delete the vhost, take a backup of the data, use strace to find which .idx file it cannot parse)
/Johan
jo...@cloudamqp.com
Sep 21, 2021, 11:25:27 AM9/21/21
to rabbitmq-users
Reading a bit more in the logs the dictionary for the crashing processes is dumped out containing the queue it is trying to read:
dictionary:
[{fhc_age_tree,{1,{{-576460732282115816,#Ref<0.2538068118.3083075586.223024>},true,nil,nil}}},{worker_pool_worker,true},{worker_pool_name,worker_pool},{rand_seed,{#{jump
=> #Fun<rand.3.47293030>,max => 288230376151711743,next
=> #Fun<rand.5.47293030>,type =>
exsplus},[21664875678697632|122503214363267052]}},{guid,{{3384895795,3739523404,3470788915,3792953416},67}},{{#Ref<0.2538068118.3083075586.223024>,fhc_handle},{handle,{file_descriptor,prim_file,#{handle
=> #Ref<0.2538068118.3083206663.218189>,owner =>
<0.438.0>,r_ahead_size => 0,r_buffer =>
#Ref<0.2538068118.3083206658.223027>}},#Ref<0.2538068118.3083075586.223024>,1599486,false,0,infinity,[],<<>>,0,0,0,0,0,false,"/var/lib/rabbitmq/mnesia/rabbit@broker1/msg_stores/vhosts/CLSVYG882Q3XWCZY90GLHUM8M/queues/1DKY2QW1LRQPSRDFB2FLQRPAM/journal.jif",[write,binary,raw,read],[{write_buffer,infinity}],true,true,-576460732282115816}},{segment_entry_count,2048},{{"/var/lib/rabbitmq/mnesia/rabbit@broker1/msg_stores/vhosts/CLSVYG882Q3XWCZY90GLHUM8M/queues/1DKY2QW1LRQPSRDFB2FLQRPAM/journal.jif",fhc_file},{file,1,true}},{{xtype_to_module,topic},rabbit_exchange_type_topic}]
I'd take a look at the queue "1DKY2QW1LRQPSRDFB2FLQRPAM" (you can find its name by looking in the file "/var/lib/rabbitmq/mnesia/rabbit@broker1/msg_stores/vhosts/CLSVYG882Q3XWCZY90GLHUM8M/queues/1DKY2QW1LRQPSRDFB2FLQRPAM/.queue_name") and see if there is anything special about it.
/Johan
Loïc Hoguin
Sep 21, 2021, 11:51:09 AM9/21/21
to rabbitm...@googlegroups.com
Hello,
If your versions are 3.8.15 and 3.8.22 then it isn’t the bug introduced in 3.8.17 and fixed in 3.8.21, but your problem looks very similar to it.
I would be interested in having the full logs from before the shutdown until the crashes start. The queue data would also be useful.
Thanks.
Cheers,
--
Loïc Hoguin
From: <rabbitm...@googlegroups.com> on behalf of Daniel Krebs <d.k...@enlyze.com>
Reply to: "rabbitm...@googlegroups.com" <rabbitm...@googlegroups.com>
Date: Monday 20 September 2021 at 17:07
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [Suspected Spam] [rabbitmq-users] Vhost corruption after restart
Hi list,
we have now experienced for the 2nd time a vhost corruption after restarting our RabbitMQ broker. We're currently running 3.8.22, but essentially the same thing has already happened in July on 3.8.15. I am running a bit out of ideas where to look for solutions, so I hope someone can help.
I have assembled all the available logs and exact package information of our current setup here: https://gist.github.com/daniel-k/e78d3ba808b9422df326b91824ce5bf4
We basically run 2 vhost: "dev" and "live". Luckily, this issue happened both times in the dev vhost, but that might have been pure luck. In order to repair it, I couldn't help but remove the whole vhost and recreate it based on a backup of the definitions.
While both vhost are more or less identical w.r.t. to queues, exchanges and the messages passing through them, the "dev" vhost has some additional queues with a TTL set. I've read about some bug related with TTL here on the list recently, so I thought it might be worth mentioning.
I have tried to reproduce the issue locally with TTL queues on a Docker setup (same Ubuntu/RabbitMQ/Erlang versions) and publishing at high rates while restarting the broker constantly, but to no avail so far.
Any help would be very much appreciated!
Best regards,
Daniel Krebs
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit
https://groups.google.com/d/msgid/rabbitmq-users/78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%40googlegroups.com.
Daniel Krebs
Sep 22, 2021, 8:30:15 AM9/22/21
to rabbitm...@googlegroups.com
I just noticed that I somehow mixed up the RabbitMQ version when the
first corruption happened. It was 3.8.19 and not on 3.8.15.
It's well possible that the queue that was corrupted has been created
while we were on 3.8.19 (which is one of the affected version IIUC).
Would it be possible that "the problem" was still present in its data
structures and still blew up now on 3.8.22?
I have sent you more complete logs and the queue data off-list as
requested. Hope it helps to pin down the issue :)
Cheers,
Daniel
On 21.09.21 17:51, Loïc Hoguin wrote:
> Hello,
>
> If your versions are 3.8.15 and 3.8.22 then it isn’t the bug introduced
> in 3.8.17 and fixed in 3.8.21, but your problem looks very similar to it.
>
> I would be interested in having the full logs from before the shutdown
> until the crashes start. The queue data would also be useful.
>
> Thanks.
>
> Cheers,
>
> --
>
> Loïc Hoguin
>
> *From: *<rabbitm...@googlegroups.com> on behalf of Daniel Krebs
> <d.k...@enlyze.com>
> *Reply to: *"rabbitm...@googlegroups.com"
> <rabbitm...@googlegroups.com>
> *Date: *Monday 20 September 2021 at 17:07
> *To: *rabbitmq-users <rabbitm...@googlegroups.com>
> *Subject: *[Suspected Spam] [rabbitmq-users] Vhost corruption after restart
> <mailto:rabbitmq-user...@googlegroups.com>.
> <https://groups.google.com/d/msgid/rabbitmq-users/1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%40vmware.com?utm_medium=email&utm_source=footer>.
first corruption happened. It was 3.8.19 and not on 3.8.15.
It's well possible that the queue that was corrupted has been created
while we were on 3.8.19 (which is one of the affected version IIUC).
Would it be possible that "the problem" was still present in its data
structures and still blew up now on 3.8.22?
I have sent you more complete logs and the queue data off-list as
requested. Hope it helps to pin down the issue :)
Cheers,
Daniel
On 21.09.21 17:51, Loïc Hoguin wrote:
> Hello,
>
> If your versions are 3.8.15 and 3.8.22 then it isn’t the bug introduced
> in 3.8.17 and fixed in 3.8.21, but your problem looks very similar to it.
>
> I would be interested in having the full logs from before the shutdown
> until the crashes start. The queue data would also be useful.
>
> Thanks.
>
> Cheers,
>
> --
>
> Loïc Hoguin
>
> <d.k...@enlyze.com>
> *Reply to: *"rabbitm...@googlegroups.com"
> <rabbitm...@googlegroups.com>
> *Date: *Monday 20 September 2021 at 17:07
> *To: *rabbitmq-users <rabbitm...@googlegroups.com>
> *Subject: *[Suspected Spam] [rabbitmq-users] Vhost corruption after restart
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/rabbitmq-users/78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%40googlegroups.com
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Clhoguin%40vmware.com%7Cd7564b3dcd134025637e08d97c48644e%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637677472633309807%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Hwi6adzu8cQD7GY4jg3%2FBGXaZzgWHatd%2Bjbf4Ozt1VA%3D&reserved=0>.
> https://groups.google.com/d/msgid/rabbitmq-users/78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%40googlegroups.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/rabbitmq-users/1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%40vmware.com
> <https://groups.google.com/d/msgid/rabbitmq-users/1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%40vmware.com?utm_medium=email&utm_source=footer>.
Daniel Krebs
Sep 22, 2021, 8:37:40 AM9/22/21
to rabbitm...@googlegroups.com
Hi Johan,
thanks for looking into our issue :)
Here is rabbitmq.config:
# only allow TLS-encrypted conncections
listeners.tcp = none
listeners.ssl.default = 5671
ssl_options.certfile = /etc/letsencrypt/live/XXXX/cert.pem
ssl_options.cacertfile = /etc/letsencrypt/live/XXXX/chain.pem
ssl_options.keyfile = /etc/letsencrypt/live/XXXX/privkey.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
# the default user name is 'guest' and can only connect via localhost
default_pass = XXXXXXXXXX
# default is 50MB, that seems a little bit too aggressive
disk_free_limit.absolute = 200MB
# increase memory watermark to make better use of existing RAM
vm_memory_high_watermark.relative = 0.7
And "rabbitmqctl environment" as well:
[{amqp10_client,[]},
{amqp10_common,[]},
{amqp_client,
[{gen_server_call_timeout,130000},
{prefer_ipv6,false},
{ssl_options,[]},
{writer_gc_threshold,1000000000}]},
{asn1,[]},
{aten,
[{detection_threshold,0.99},
{heartbeat_interval,100},
{poll_interval,5000},
{scaling_factor,1.5}]},
{base64url,[]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{credentials_obfuscation,[{enabled,true}]},
{crypto,[{fips_mode,false},{rand_cache_size,896}]},
{cuttlefish,[]},
{gen_batch_server,[]},
{goldrush,[]},
{inets,[]},
{jose,[{crypto_fallback,true}]},
{jsx,[]},
{kernel,
[{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672},
{logger,
[{handler,default,logger_std_h,
#{config => #{type => standard_io},
formatter =>
{logger_formatter,
#{legacy_header => true,single_line =>
false}}}}]},
{logger_level,notice},
{logger_sasl_compatible,false},
{shell_docs_ansi,auto},
{shutdown_func,{rabbit_prelaunch,shutdown_func}}]},
{lager,
[{async_threshold,20},
{async_threshold_window,5},
{colored,false},
{colors,
[{debug,"\e[0;38m"},
{info,"\e[1;37m"},
{notice,"\e[1;36m"},
{warning,"\e[1;33m"},
{error,"\e[1;31m"},
{critical,"\e[1;35m"},
{alert,"\e[1;44m"},
{emergency,"\e[1;41m"}]},
{crash_log,"log/crash.log"},
{crash_log_count,5},
{crash_log_date,"$D0"},
{crash_log_msg_size,65536},
{crash_log_rotator,lager_rotator_default},
{crash_log_size,10485760},
{error_logger_format_raw,true},
{error_logger_hwm,5000},
{error_logger_hwm_original,50},
{error_logger_redirect,true},
{extra_sinks,
[{error_logger_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_channel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_connection_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_feature_flags_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_federation_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ldap_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_mirroring_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_prelaunch_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_queue_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ra_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_shovel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_upgrade_lager_event,
[{handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]}]}]},
{handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/rab...@broker1.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]},
{log_root,"/var/log/rabbitmq"},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/rab...@broker1.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]}]},
{mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@broker1"}]},
{observer_cli,[{plugins,[]},{scheduler_usage,disable}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{prometheus,[]},
{public_key,[]},
{ra,[{data_dir,"/var/lib/rabbitmq/mnesia/rabbit@broker1/quorum"},
{logger_module,rabbit_log_ra_shim},
{wal_max_batch_size,4096},
{wal_max_size_bytes,536870912}]},
{rabbit,
[{auth_backends,
[rabbit_auth_backend_oauth2,rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{autocluster,
[{peer_discovery_backend,rabbit_peer_discovery_classic_config}]},
{autoheal_state_transition_timeout,60000},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,2047},
{channel_operation_timeout,15000},
{channel_tick_interval,60000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,ignore},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{config_entry_decoder,[{passphrase,undefined}]},
{connection_max,infinity},
{consumer_timeout,1800000},
{credit_flow_default_credit,{400,200}},
{default_consumer_prefetch,{false,0}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,"200MB"},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{feature_flags_file,
"/var/lib/rabbitmq/mnesia/rabbit@broker1-feature_flags"},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{lager_default_file,"/var/log/rabbitmq/rab...@broker1.log"},
{lager_log_root,"/var/log/rabbitmq"},
{lager_upgrade_file,"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log,
[{file,[{file,"/var/log/rabbitmq/rab...@broker1.log"}]},
{categories,
[{upgrade,
[{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"}]}]}]},
{loopback_users,[<<"guest">>]},
{max_message_size,134217728},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{msg_store_shutdown_timeout,600000},
{num_ssl_acceptors,10},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,
"/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.22/plugins"},
{plugins_expand_dir,
"/var/lib/rabbitmq/mnesia/rabbit@broker1-plugins-expand"},
{proxy_protocol,false},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{quorum_cluster_size,3},
{quorum_commands_soft_limit,32},
{reverse_dns_lookups,false},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[5671]},
{ssl_options,
[{keyfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/privkey.pem"},
{certfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/cert.pem"},
{cacertfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/chain.pem"},
{fail_if_no_peer_cert,false},
{verify,verify_peer}]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{track_auth_attempt_source,false},
{tracking_execution_timeout,15000},
{vhost_restart_strategy,continue},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.7},
{vm_memory_high_watermark_paging_ratio,0.5},
{writer_gc_threshold,1000000000}]},
{rabbit_common,[]},
{rabbitmq_auth_backend_oauth2,
[{extra_scopes_source,<<"rmq">>},
{key_config,
[{default_key,<<"XXXXXXXXX">>},
{signing_keys,
#{<<"XXXXXXXXX">> =>
{pem_file,<<"/etc/rabbitmq//keys/XXXXXXX.pem">>},
<<"XXXXXXXXXXX">> =>
{pem_file,<<"/etc/rabbitmq/keys/XXXXXXXXXX.pem">>}}}]},
{resource_server_id,<<"broker1">>}]},
{rabbitmq_management,
[{content_security_policy,
"script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src
'self'"},
{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_prelaunch,[]},
{rabbitmq_prometheus,[{return_per_object_metrics,false}]},
{rabbitmq_shovel,
[{defaults,
[{prefetch_count,1000},
{ack_mode,on_confirm},
{publish_fields,[]},
{publish_properties,[]},
{reconnect_delay,5}]}]},
{rabbitmq_shovel_management,[]},
{rabbitmq_web_dispatch,[]},
{ranch,[]},
{recon,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,false}]},
{ssl,
[{dtls_protocol_version,['dtlsv1.2']},
{protocol_version,['tlsv1.3','tlsv1.2']}]},
{stdlib,[]},
{stdout_formatter,[]},
{syntax_tools,[]},
{sysmon_handler,
[{busy_dist_port,true},
{busy_port,false},
{gc_ms_limit,0},
{heap_word_limit,0},
{port_limit,100},
{process_limit,100},
{schedule_ms_limit,0}]},
{tools,[{file_util_search_methods,[{[],[]},{"ebin","esrc"},{"ebin","src"}]}]},
{xmerl,[]}]
> <https://groups.google.com/d/msgid/rabbitmq-users/fef63b5f-c4c1-47e6-9cfc-5e56ca836dc3n%40googlegroups.com?utm_medium=email&utm_source=footer>.
thanks for looking into our issue :)
Here is rabbitmq.config:
# only allow TLS-encrypted conncections
listeners.tcp = none
listeners.ssl.default = 5671
ssl_options.certfile = /etc/letsencrypt/live/XXXX/cert.pem
ssl_options.cacertfile = /etc/letsencrypt/live/XXXX/chain.pem
ssl_options.keyfile = /etc/letsencrypt/live/XXXX/privkey.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
# the default user name is 'guest' and can only connect via localhost
default_pass = XXXXXXXXXX
# default is 50MB, that seems a little bit too aggressive
disk_free_limit.absolute = 200MB
# increase memory watermark to make better use of existing RAM
vm_memory_high_watermark.relative = 0.7
And "rabbitmqctl environment" as well:
[{amqp10_client,[]},
{amqp10_common,[]},
{amqp_client,
[{gen_server_call_timeout,130000},
{prefer_ipv6,false},
{ssl_options,[]},
{writer_gc_threshold,1000000000}]},
{asn1,[]},
{aten,
[{detection_threshold,0.99},
{heartbeat_interval,100},
{poll_interval,5000},
{scaling_factor,1.5}]},
{base64url,[]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{credentials_obfuscation,[{enabled,true}]},
{crypto,[{fips_mode,false},{rand_cache_size,896}]},
{cuttlefish,[]},
{gen_batch_server,[]},
{goldrush,[]},
{inets,[]},
{jose,[{crypto_fallback,true}]},
{jsx,[]},
{kernel,
[{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672},
{logger,
[{handler,default,logger_std_h,
#{config => #{type => standard_io},
formatter =>
{logger_formatter,
#{legacy_header => true,single_line =>
false}}}}]},
{logger_level,notice},
{logger_sasl_compatible,false},
{shell_docs_ansi,auto},
{shutdown_func,{rabbit_prelaunch,shutdown_func}}]},
{lager,
[{async_threshold,20},
{async_threshold_window,5},
{colored,false},
{colors,
[{debug,"\e[0;38m"},
{info,"\e[1;37m"},
{notice,"\e[1;36m"},
{warning,"\e[1;33m"},
{error,"\e[1;31m"},
{critical,"\e[1;35m"},
{alert,"\e[1;44m"},
{emergency,"\e[1;41m"}]},
{crash_log,"log/crash.log"},
{crash_log_count,5},
{crash_log_date,"$D0"},
{crash_log_msg_size,65536},
{crash_log_rotator,lager_rotator_default},
{crash_log_size,10485760},
{error_logger_format_raw,true},
{error_logger_hwm,5000},
{error_logger_hwm_original,50},
{error_logger_redirect,true},
{extra_sinks,
[{error_logger_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_channel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_connection_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_feature_flags_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_federation_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ldap_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_mirroring_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_prelaunch_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_queue_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ra_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_shovel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_upgrade_lager_event,
[{handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]}]}]},
{handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/rab...@broker1.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]},
{log_root,"/var/log/rabbitmq"},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/rab...@broker1.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]}]},
{mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@broker1"}]},
{observer_cli,[{plugins,[]},{scheduler_usage,disable}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{prometheus,[]},
{public_key,[]},
{ra,[{data_dir,"/var/lib/rabbitmq/mnesia/rabbit@broker1/quorum"},
{logger_module,rabbit_log_ra_shim},
{wal_max_batch_size,4096},
{wal_max_size_bytes,536870912}]},
{rabbit,
[{auth_backends,
[rabbit_auth_backend_oauth2,rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{autocluster,
[{peer_discovery_backend,rabbit_peer_discovery_classic_config}]},
{autoheal_state_transition_timeout,60000},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,2047},
{channel_operation_timeout,15000},
{channel_tick_interval,60000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,ignore},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{config_entry_decoder,[{passphrase,undefined}]},
{connection_max,infinity},
{consumer_timeout,1800000},
{credit_flow_default_credit,{400,200}},
{default_consumer_prefetch,{false,0}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,"200MB"},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{feature_flags_file,
"/var/lib/rabbitmq/mnesia/rabbit@broker1-feature_flags"},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{lager_default_file,"/var/log/rabbitmq/rab...@broker1.log"},
{lager_log_root,"/var/log/rabbitmq"},
{lager_upgrade_file,"/var/log/rabbitmq/rabbit@broker1_upgrade.log"},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log,
[{file,[{file,"/var/log/rabbitmq/rab...@broker1.log"}]},
{categories,
[{upgrade,
[{file,
"/var/log/rabbitmq/rabbit@broker1_upgrade.log"}]}]}]},
{loopback_users,[<<"guest">>]},
{max_message_size,134217728},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{msg_store_shutdown_timeout,600000},
{num_ssl_acceptors,10},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,
"/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.22/plugins"},
{plugins_expand_dir,
"/var/lib/rabbitmq/mnesia/rabbit@broker1-plugins-expand"},
{proxy_protocol,false},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{quorum_cluster_size,3},
{quorum_commands_soft_limit,32},
{reverse_dns_lookups,false},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[5671]},
{ssl_options,
[{keyfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/privkey.pem"},
{certfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/cert.pem"},
{cacertfile,"/etc/letsencrypt/live/XXXXXXXXXXXXX/chain.pem"},
{fail_if_no_peer_cert,false},
{verify,verify_peer}]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{track_auth_attempt_source,false},
{tracking_execution_timeout,15000},
{vhost_restart_strategy,continue},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.7},
{vm_memory_high_watermark_paging_ratio,0.5},
{writer_gc_threshold,1000000000}]},
{rabbit_common,[]},
{rabbitmq_auth_backend_oauth2,
[{extra_scopes_source,<<"rmq">>},
{key_config,
[{default_key,<<"XXXXXXXXX">>},
{signing_keys,
#{<<"XXXXXXXXX">> =>
{pem_file,<<"/etc/rabbitmq//keys/XXXXXXX.pem">>},
<<"XXXXXXXXXXX">> =>
{pem_file,<<"/etc/rabbitmq/keys/XXXXXXXXXX.pem">>}}}]},
{resource_server_id,<<"broker1">>}]},
{rabbitmq_management,
[{content_security_policy,
"script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src
'self'"},
{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_prelaunch,[]},
{rabbitmq_prometheus,[{return_per_object_metrics,false}]},
{rabbitmq_shovel,
[{defaults,
[{prefetch_count,1000},
{ack_mode,on_confirm},
{publish_fields,[]},
{publish_properties,[]},
{reconnect_delay,5}]}]},
{rabbitmq_shovel_management,[]},
{rabbitmq_web_dispatch,[]},
{ranch,[]},
{recon,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,false}]},
{ssl,
[{dtls_protocol_version,['dtlsv1.2']},
{protocol_version,['tlsv1.3','tlsv1.2']}]},
{stdlib,[]},
{stdout_formatter,[]},
{syntax_tools,[]},
{sysmon_handler,
[{busy_dist_port,true},
{busy_port,false},
{gc_ms_limit,0},
{heap_word_limit,0},
{port_limit,100},
{process_limit,100},
{schedule_ms_limit,0}]},
{tools,[{file_util_search_methods,[{[],[]},{"ebin","esrc"},{"ebin","src"}]}]},
{xmerl,[]}]
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/rabbitmq-users/fef63b5f-c4c1-47e6-9cfc-5e56ca836dc3n%40googlegroups.com
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> <https://groups.google.com/d/msgid/rabbitmq-users/fef63b5f-c4c1-47e6-9cfc-5e56ca836dc3n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Loïc Hoguin
Sep 22, 2021, 8:54:32 AM9/22/21
to rabbitm...@googlegroups.com
If the queue was created on 3.8.19 and you then upgraded to 3.8.22 the files will still be wrong and it will crash at some point.
Please drop the data and recreate the queue on a working version.
--
Loïc Hoguin
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fdaniel-k%2Fe78d3ba808b9422df326b91824ce5bf4&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dPXPyEcY%2FM13gXQ4azLlEF0gPkSe17aHuUYdTVXymZo%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cVYbRlXdK8yILi9GL8dbYz%2BtPa0P1ADdpNfuTYYy4d8%3D&reserved=0>.
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%2540vmware.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qo5pwaPcxtmQ4Niq%2BzIuDl9y%2FFw13QWg1rEcomydTeA%3D&reserved=0>.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F29203b49-dd9b-4d13-7752-8f206501639e%2540enlyze.com&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=j9vEUOw32%2F5G%2FppC%2BcLGPD4zOIG5lbsNFrGI1X6MfQI%3D&reserved=0.
Please drop the data and recreate the queue on a working version.
--
Loïc Hoguin
>
> We basically run 2 vhost: "dev" and "live". Luckily, this issue happened
> both times in the dev vhost, but that might have been pure luck. In
> order to repair it, I couldn't help but remove the whole vhost and
> recreate it based on a backup of the definitions.
>
> While both vhost are more or less identical w.r.t. to queues, exchanges
> and the messages passing through them, the "dev" vhost has some
> additional queues with a TTL set. I've read about some bug related with
> TTL here on the list recently, so I thought it might be worth mentioning.
>
> I have tried to reproduce the issue locally with TTL queues on a Docker
> setup (same Ubuntu/RabbitMQ/Erlang versions) and publishing at high
> rates while restarting the broker constantly, but to no avail so far.
>
> Any help would be very much appreciated!
>
> Best regards,
>
> Daniel Krebs
>
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%2540googlegroups.com&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NQJ5URTvN8b36Zi%2FncuESuSCG1uqjGGIlTIiH22rK7E%3D&reserved=0
> We basically run 2 vhost: "dev" and "live". Luckily, this issue happened
> both times in the dev vhost, but that might have been pure luck. In
> order to repair it, I couldn't help but remove the whole vhost and
> recreate it based on a backup of the definitions.
>
> While both vhost are more or less identical w.r.t. to queues, exchanges
> and the messages passing through them, the "dev" vhost has some
> additional queues with a TTL set. I've read about some bug related with
> TTL here on the list recently, so I thought it might be worth mentioning.
>
> I have tried to reproduce the issue locally with TTL queues on a Docker
> setup (same Ubuntu/RabbitMQ/Erlang versions) and publishing at high
> rates while restarting the broker constantly, but to no avail so far.
>
> Any help would be very much appreciated!
>
> Best regards,
>
> Daniel Krebs
>
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F78727ed2-a08c-4ea6-ad94-fe96c7cfde4an%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cVYbRlXdK8yILi9GL8dbYz%2BtPa0P1ADdpNfuTYYy4d8%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%2540vmware.com&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MzIqh%2FLPUe2OsQt7pyxOqeX27WNEQSSZrABDqoqHiss%3D&reserved=0
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To view this discussion on the web, visit
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F1A03CF6C-2FBC-464E-B8F5-5B8E610AA56F%2540vmware.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qo5pwaPcxtmQ4Niq%2BzIuDl9y%2FFw13QWg1rEcomydTeA%3D&reserved=0>.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To view this discussion on the web, visit https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F29203b49-dd9b-4d13-7752-8f206501639e%2540enlyze.com&data=04%7C01%7Clhoguin%40vmware.com%7Cf9fc10ce444e4324c81f08d97dc4bb24%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637679106186996711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=j9vEUOw32%2F5G%2FppC%2BcLGPD4zOIG5lbsNFrGI1X6MfQI%3D&reserved=0.
Reply all
Reply to author
Forward
0 new messages