Virtual host / experienced an error on node rabbit@rabbitnodeone and may be inaccessible

[Atif Waqar]

Hello RabbitMQ Team!

We have a cluster of 3 nodes (windows VMs), It was working splendidly, till we had some network outage for a short period of time and then we started seeing a yellow banner saying "Virtual host / experienced an error on node rabbit@rabbitnodeone and may be inaccessible". When we checked the logs, we see a warning saying "Unable to start queue mirror on node 'rabbit@rabbitnodeone'. Target virtual host is not running:"

Anyidea what could be the reason behind this?

We are using : 

RabbitMQ Version : 3.7.11
Erlang Version : 21.3

And here is the Error Log : 

2019-10-07 08:38:57.268 [info] <0.17524.49> accepting AMQP connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:38:57.299 [info] <0.17524.49> Connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceOne

2019-10-07 08:38:57.315 [info] <0.17524.49> connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672 - ServiceOne): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:39:01.002 [info] <0.17731.49> accepting AMQP connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:39:01.033 [info] <0.17731.49> Connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceTwo

2019-10-07 08:39:01.033 [info] <0.17731.49> connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672 - ServiceTwo): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:39:02.049 [info] <0.17582.49> accepting AMQP connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:39:02.096 [info] <0.17582.49> Connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceThree

2019-10-07 08:39:02.111 [info] <0.17582.49> connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672 - ServiceThree): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:40:37.302 [warning] <0.1418.0> closing AMQP connection <0.1418.0> (xxx.xxx.xxx.101:24915 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1587.0> closing AMQP connection <0.1587.0> (xxx.xxx.xxx.101:13096 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1731.0> closing AMQP connection <0.1731.0> (xxx.xxx.xxx.101:7417 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1761.0> closing AMQP connection <0.1761.0> (xxx.xxx.xxx.101:64841 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:49.365 [info] <0.18489.49> accepting AMQP connection <0.18489.49> (xxx.xxx.xxx.101:59302 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:40:49.365 [info] <0.18489.49> connection <0.18489.49> (xxx.xxx.xxx.101:59302 -> xxx.xxx.xxx.22:5672): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:40:49.412 [info] <0.18481.49> Mirrored queue 'Service-RPC-rabbitnodetwo' in vhost '/': Adding mirror on node rabbit@rabbitnodethree: <7470.4561.14>

2019-10-07 08:40:49.428 [warning] <0.18481.49> Mirrored queue 'Service-RPC-rabbitnodetwo' in vhost '/': Unable to start queue mirror on node 'rabbit@rabbitnodeone'. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}

[Wesley Peng]

Hi

on 2019/11/11 23:10, Atif Waqar wrote:
> 2019-10-07 08:40:49.428 [warning] <0.18481.49> Mirrored queue
> 'Service-RPC-rabbitnodetwo' in vhost '/': Unable to start queue mirror
> on node 'rabbit@rabbitnodeone'. Target virtual host is not running:
> {vhost_supervisor_not_running,<<"/">>}

Try restart the broker on this node and see if it can join the cluster
correctly. Also please check there is no physical error for OS like
disk, memory etc.

regards.

[Atif Waqar]

> Try restart the broker on this node and see if it can join the cluster correctly. 

Yes! we tried this, along with restarting the entire node (windows restart). But it didnt fix the issue.

>Also please check there is no physical error for OS like disk, memory etc. 

Is there a way to see such error in RabbitMQ Logs?

[Wesley Peng]

on 2019/11/12 16:44, Atif Waqar wrote:
> >Also please check there is no physical error for OS like disk, memory
> etc.
> Is there a way to see such error in RabbitMQ Logs?

I meant you should check the system logs for disk, memory and other
errors, if there are those errors, they will influence the work of rabbitmq.

regards.

[Atif Waqar]

Ah, I see. Makes sense. 

Well, We are checking the logs and there are no such errors in system logs.

[Atif Waqar]

Correction about Erlang Version : its 21.2

[Atif Waqar]

Got error log of next one minute which I think logs some further errors : 

This block is already posted above
***************************************************

2019-10-07 08:38:57.268 [info] <0.17524.49> accepting AMQP connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:38:57.299 [info] <0.17524.49> Connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceOne

2019-10-07 08:38:57.315 [info] <0.17524.49> connection <0.17524.49> (xxx.xxx.xxx.101:20968 -> xxx.xxx.xxx.22:5672 - ServiceOne): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:39:01.002 [info] <0.17731.49> accepting AMQP connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:39:01.033 [info] <0.17731.49> Connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceTwo

2019-10-07 08:39:01.033 [info] <0.17731.49> connection <0.17731.49> (xxx.xxx.xxx.101:45132 -> xxx.xxx.xxx.22:5672 - ServiceTwo): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:39:02.049 [info] <0.17582.49> accepting AMQP connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:39:02.096 [info] <0.17582.49> Connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceThree

2019-10-07 08:39:02.111 [info] <0.17582.49> connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672 - ServiceThree): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:40:37.302 [warning] <0.1418.0> closing AMQP connection <0.1418.0> (xxx.xxx.xxx.101:24915 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1587.0> closing AMQP connection <0.1587.0> (xxx.xxx.xxx.101:13096 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1731.0> closing AMQP connection <0.1731.0> (xxx.xxx.xxx.101:7417 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:37.318 [warning] <0.1761.0> closing AMQP connection <0.1761.0> (xxx.xxx.xxx.101:64841 -> xxx.xxx.xxx.22:5672, vhost: '/', user: 'Atif_Admin'):

client unexpectedly closed TCP connection

2019-10-07 08:40:49.365 [info] <0.18489.49> accepting AMQP connection <0.18489.49> (xxx.xxx.xxx.101:59302 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:40:49.365 [info] <0.18489.49> connection <0.18489.49> (xxx.xxx.xxx.101:59302 -> xxx.xxx.xxx.22:5672): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:40:49.412 [info] <0.18481.49> Mirrored queue 'Service-RPC-rabbitnodetwo' in vhost '/': Adding mirror on node rabbit@rabbitnodethree: <7470.4561.14>

2019-10-07 08:40:49.428 [warning] <0.18481.49> Mirrored queue 'Service-RPC-rabbitnodetwo' in vhost '/': Unable to start queue mirror on node 'rabbit@rabbitnodeone'. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}

***************************************************

This block is added now

***************************************************

2019-10-07 08:41:22.553 [info] <0.394.0> rabbit on node rabbit@rabbitnodeone down

2019-10-07 08:41:22.724 [info] <0.394.0> Keeping rabbit@rabbitnodeone listeners: the node is already back

2019-10-07 08:41:23.646 [info] <0.394.0> node rabbit@rabbitnodeone down: connection_closed

2019-10-07 08:41:27.990 [info] <0.18666.49> accepting AMQP connection <0.18666.49> (xxx.xxx.xxx.101:13619 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:41:28.021 [info] <0.18870.49> Mirrored queue 'ServiceFive-rabbitnodeone' in vhost '/': Adding mirror on node rabbit@rabbitnodethree: <7470.4683.14>

2019-10-07 08:41:28.037 [info] <0.18870.49> Mirrored queue 'ServiceFive-rabbitnodeone' in vhost '/': Synchronising: 0 messages to synchronise

2019-10-07 08:41:28.037 [info] <0.18870.49> Mirrored queue 'ServiceFive-rabbitnodeone' in vhost '/': Synchronising: batch size: 4096

2019-10-07 08:41:28.037 [info] <0.18887.49> Mirrored queue 'ServiceFive-rabbitnodeone' in vhost '/': Synchronising: all slaves already synced

2019-10-07 08:41:36.667 [info] <0.99.0> RabbitMQ is asked to stop...

2019-10-07 08:41:36.791 [info] <0.17362.49> closing AMQP connection <0.17362.49> (xxx.xxx.xxx.101:12605 -> xxx.xxx.xxx.22:5672 - Log Collector, vhost: '/', user: 'Atif_Admin')

2019-10-07 08:41:36.822 [info] <0.17478.49> closing AMQP connection <0.17478.49> (xxx.xxx.xxx.101:11325 -> xxx.xxx.xxx.22:5672 - HandlerService (Module Synchronization Service), vhost: '/', user: 'Atif_Admin')

2019-10-07 08:41:36.838 [info] <0.2408.0> closing AMQP connection <0.2408.0> (xxx.xxx.xxx.101:26664 -> xxx.xxx.xxx.22:5672 - HandlerService (ModuleUserSyncSubscriptionService), vhost: '/', user: 'Atif_Admin')

2019-10-07 08:41:36.885 [info] <0.17582.49> closing AMQP connection <0.17582.49> (xxx.xxx.xxx.101:52225 -> xxx.xxx.xxx.22:5672 - HandlerService (Module User Sync Subscription Service), vhost: '/', user: 'Atif_Admin')

2019-10-07 08:41:36.885 [info] <0.17449.49> closing AMQP connection <0.17449.49> (xxx.xxx.xxx.101:24354 -> xxx.xxx.xxx.22:5672 - CommandServer, vhost: '/', user: 'Atif_Admin')

2019-10-07 08:41:36.978 [info] <0.99.0> Stopping RabbitMQ applications and their dependencies in the following order:

    rabbitmq_management

    amqp_client

    rabbitmq_web_dispatch

    cowboy

    cowlib

    rabbitmq_management_agent

    rabbit

    mnesia

    rabbit_common

    sysmon_handler

    os_mon

2019-10-07 08:41:36.978 [info] <0.99.0> Stopping application 'rabbitmq_management'

2019-10-07 08:41:36.978 [warning] <0.819.0> RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls

2019-10-07 08:41:36.978 [info] <0.99.0> Stopping application 'amqp_client'

2019-10-07 08:41:36.978 [info] <0.43.0> Application rabbitmq_management exited with reason: stopped

2019-10-07 08:41:36.994 [info] <0.43.0> Application amqp_client exited with reason: stopped

2019-10-07 08:41:36.994 [info] <0.99.0> Stopping application 'rabbitmq_web_dispatch'

2019-10-07 08:41:37.010 [info] <0.99.0> Stopping application 'cowboy'

2019-10-07 08:41:37.010 [info] <0.43.0> Application rabbitmq_web_dispatch exited with reason: stopped

2019-10-07 08:41:37.010 [info] <0.99.0> Stopping application 'cowlib'

2019-10-07 08:41:37.010 [info] <0.43.0> Application cowboy exited with reason: stopped

2019-10-07 08:41:37.010 [info] <0.99.0> Stopping application 'rabbitmq_management_agent'

2019-10-07 08:41:37.010 [info] <0.43.0> Application cowlib exited with reason: stopped

2019-10-07 08:41:37.025 [info] <0.99.0> Stopping application 'rabbit'

2019-10-07 08:41:37.025 [info] <0.43.0> Application rabbitmq_management_agent exited with reason: stopped

2019-10-07 08:41:37.025 [info] <0.290.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.

2019-10-07 08:41:37.025 [info] <0.793.0> stopped TCP listener on 0.0.0.0:5672

2019-10-07 08:41:37.025 [error] <0.17440.49> Error on AMQP connection <0.17440.49> (xxx.xxx.xxx.101:8615 -> xxx.xxx.xxx.22:5672 - HandlerService (Module Messaging Service), vhost: '/', user: 'Atif_Admin', state: running), channel 0:

 operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"

2019-10-07 08:42:13.283 [info] <0.8.0> Log file opened with Lager

2019-10-07 08:42:21.174 [info] <0.43.0> Application mnesia exited with reason: stopped

2019-10-07 08:42:28.893 [error] <0.195.0> Mnesia(rabbit@rabbitnodetwo): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, rabbit@rabbitnodeone}

2019-10-07 08:42:30.268 [info] <0.311.0> 

 Starting RabbitMQ 3.7.11 on Erlang 21.2

 Copyright (C) 2007-2019 Pivotal Software, Inc.

 Licensed under the MPL.  See http://www.rabbitmq.com/

2019-10-07 08:42:30.268 [info] <0.311.0> 

 node           : rabbit@rabbitnodetwo

 home dir       : C:\Windows\system32\config\systemprofile

 config file(s) : c:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/advanced.config

 cookie hash    : xxxxxxxxxxxxxxxxxxxxxxx

 log(s)         : C:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/log/RABBIT~1.LOG

                : C:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/log/rabbit@rabbitnodetwo_upgrade.log

 database dir   : c:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/db/RABBIT~1

2019-10-07 08:42:35.440 [info] <0.393.0> Memory high watermark set to 13107 MiB (13743706931 bytes) of 32767 MiB (34359267328 bytes) total

2019-10-07 08:42:36.127 [info] <0.469.0> Enabling free disk space monitoring

2019-10-07 08:42:36.127 [info] <0.469.0> Disk free limit set to 50MB

2019-10-07 08:42:36.143 [info] <0.472.0> Limiting to approx 8092 file handles (7280 sockets)

2019-10-07 08:42:36.143 [info] <0.473.0> FHC read buffering:  OFF

2019-10-07 08:42:36.143 [info] <0.473.0> FHC write buffering: ON

2019-10-07 08:42:36.174 [info] <0.311.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2019-10-07 08:42:36.174 [info] <0.311.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2019-10-07 08:42:36.205 [info] <0.311.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2019-10-07 08:42:36.205 [info] <0.311.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.

2019-10-07 08:42:36.205 [info] <0.311.0> Priority queues enabled, real BQ is rabbit_variable_queue

2019-10-07 08:42:36.237 [info] <0.502.0> Starting rabbit_node_monitor

2019-10-07 08:42:37.096 [info] <0.311.0> Management plugin: using rates mode 'basic'

2019-10-07 08:42:37.158 [info] <0.550.0> Making sure data directory 'c:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists

2019-10-07 08:42:37.690 [info] <0.550.0> Starting message stores for vhost '/'

2019-10-07 08:42:37.690 [info] <0.554.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index

2019-10-07 08:42:37.705 [info] <0.550.0> Started message store of type transient for vhost '/'

2019-10-07 08:42:37.705 [info] <0.557.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index

2019-10-07 08:42:37.752 [warning] <0.557.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent" : recovery terms differ from present

rebuilding indices from scratch

2019-10-07 08:42:37.752 [info] <0.550.0> Started message store of type persistent for vhost '/'

2019-10-07 08:42:37.768 [info] <0.550.0> Mirrored queue 'ManageTimersQueue' in vhost '/': Adding mirror on node rabbit@rabbitnodetwo: <0.576.0>

2019-10-07 08:42:37.799 [warning] <0.654.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.

2019-10-07 08:42:37.815 [info] <0.668.0> started TCP listener on [::]:5672

2019-10-07 08:42:37.815 [warning] <0.677.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.

2019-10-07 08:42:37.815 [info] <0.691.0> started TCP listener on 0.0.0.0:5672

2019-10-07 08:42:37.815 [info] <0.311.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@rabbitnodetwo

2019-10-07 08:42:37.815 [info] <0.311.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@rabbitnodetwo

2019-10-07 08:42:37.815 [info] <0.502.0> rabbit on node rabbit@rabbitnodethree up

2019-10-07 08:42:38.080 [warning] <0.753.0> Could not find handle.exe, please install from sysinternals

2019-10-07 08:42:38.409 [info] <0.502.0> rabbit on node rabbit@rabbitnodeone up

2019-10-07 08:42:38.471 [info] <0.818.0> Management plugin: HTTP (non-TLS) listener started on port 15672

2019-10-07 08:42:38.471 [info] <0.924.0> Statistics database started.

2019-10-07 08:42:38.471 [notice] <0.106.0> Changed loghwm of C:/Users/Atif_Admin/AppData/Roaming/RabbitMQ/log/RABBIT~1.LOG to 50

2019-10-07 08:42:38.627 [info] <0.8.0> Server startup complete; 3 plugins started.

 * rabbitmq_management

 * rabbitmq_management_agent

 * rabbitmq_web_dispatch

2019-10-07 08:42:42.971 [info] <0.940.0> accepting AMQP connection <0.940.0> (xxx.xxx.xxx.101:26808 -> xxx.xxx.xxx.22:5672)

2019-10-07 08:42:42.987 [info] <0.940.0> Connection <0.940.0> (xxx.xxx.xxx.101:26808 -> xxx.xxx.xxx.22:5672) has a client-provided name: ServiceTen Service@rabbitnodetwo

2019-10-07 08:42:43.002 [info] <0.940.0> connection <0.940.0> (xxx.xxx.xxx.101:26808 -> xxx.xxx.xxx.22:5672 - ServiceTen Service@rabbitnodetwo): user 'Atif_Admin' authenticated and granted access to vhost '/'

2019-10-07 08:42:43.065 [info] <0.952.0> Mirrored queue 'ServiceEleven' in vhost '/': Adding mirror on node rabbit@rabbitnodethree: <14416.895.0>

2019-10-07 08:42:43.081 [warning] <0.952.0> Mirrored queue 'ServiceEleven' in vhost '/': Unable to start queue mirror on node 'rabbit@rabbitnodeone'. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}

On Monday, November 11, 2019 at 4:10:26 PM UTC+1, Atif Waqar wrote:

***************************************************

[Wesley Peng]

Hi

on 2019/11/12 20:29, Atif Waqar wrote:
> 2019-10-07 08:42:28.893 [error] <0.195.0> Mnesia(rabbit@rabbitnodetwo):
> ** ERROR ** mnesia_event got {inconsistent_database,
> starting_partitioned_network, rabbit@rabbitnodeone}

I saw there is an error shows that mnesia database got inconsistent.

You could google for how to fixup it.

regards.

[Atif Waqar]

Found one response from RabbitMQ team, But its from 2012 http://rabbitmq.1065348.n5.nabble.com/What-can-cause-mnesia-partitioning-td22535.html

It says : 
> Mnesia('rabbit@node-1'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit@node-2'}
This indicates that a netsplit has occurred; A situation that mnesia (and therefore clustered rabbit) does not handle well.

And at the end of the response they say : 
Mnesia will not magically recover from netsplits without intervention, so this problem is actually non-trivial to fix - nevertheless, we are actively looking for solutions to this situation!

My only concern is that this response is too old, it was posted in 2012.
@Wesley Peng Do you think the above statement is still valid?

[wes...@pengdrive.com]

Yes I think so. I know distribution system well, it is actually hard to maintain data consistent well in a complicated environment. Sometime we just hate those so called distributed products. 

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/44030ca4-0683-4e4f-b9ef-061b89475cc4%40googlegroups.com.

[Atif Waqar]

One more thing, Does having pause_minority has any effect over this issue?  Or pause_minority has nothing to do with this?

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.