How to bringup a second node as service on windows 2019

[richard]

Hi all,

how can i have a 2nd RMQ node as windows service running...

I have tried it now for 2 days to bring my 2nd service up and running ... no success so far.

What i have done on the WIN2019 machine:

• Installed latest Erlang
• Installed latest RMQ Windows-Installer
• enabled management-plugin
  
• started the service

Since i want 2 services with different log and db folders i have added another admin/user account and logged in with it. 1st RMQ node service is still running.

So the folder for the 2nd node is now C:\Users\blsadmin2\AppData\Roaming\RabbitM

Copied .erlang.cookie from 1st node %HOMEPATH% to actual users %HOMEPATH%

Also copied the RabbitMQ folder from 1st node user folder and deleted the files in the db and log folders.

Then copied then RMQ folder C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.4  to C:\Program Files\RabbitMQ Server\rabbitmq_server-logging-3.8.4

Set env-variables to following:

set RABBITMQ_USE_LONGNAME=true
set RABBITMQ_SERVICENAME=RabbitMQ-LOG
set RABBITMQ_NODENAME=rabbit2@BLSENTWWEB1
set RABBITMQ_NODE_PORT=5673
set RABBITMQ_SERVER_START_ARGS=-rabbitmq_management listener [{port,15673}]

Enabled management-plugin.

Running rabbitmq-server.bat   ... 2nd node is up and running.

Canceling the node-app  (CTRL-C)

Changed the rabbitmq-service.bat file ... added following (yellow lines):

REM Get default settings with user overrides for (RABBITMQ_)<var_name>
REM Non-empty defaults should be set in rabbitmq-env
call "%TDP0%\rabbitmq-env.bat" %~n0


set RABBITMQ_USE_LONGNAME=true
set RABBITMQ_SERVICENAME=RabbitMQ-LOG
set RABBITMQ_NODENAME=rabbit2@BLSENTWWEB1
set RABBITMQ_NODE_PORT=5673
set RABBITMQ_SERVER_START_ARGS=-rabbitmq_management listener [{port,15673}]


REM Check for the short names here too

Running rabbitmq-service.bat install  ... Service is added.

Running rabbitmq-service.bat start   ... Service is started ... but stops after a few seconds.

There is no new info in the log folders ... last entries are from startup as app with the rabbitmq-server.bat file..

erl_crash.dump file after service is stopped with following message inside (first few lines):

=erl_crash_dump:0.5
Thu May 28 18:03:24 2020
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{erlang_dist_running_with_unexpected_nodename,rabbit2@BLSENTWW
System version: Erlang/OTP 23 [erts-11.0.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64]

Following the registry-entry of the second node:

Schlüsselname:          HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson\Erlang\ErlSrv\1.1\RabbitMQ-LOG
Klassenname:            <KEINE KLASSE>
Letzter Schreibzugriff: 28.05.2020 - 17:48
Wert 0
  Name:            StopAction
  Typ:             REG_SZ
  Daten:           rabbit:stop_and_halt().


Wert 1
  Name:            OnFail
  Typ:             REG_DWORD
  Daten:           0x1


Wert 2
  Name:            Machine
  Typ:             REG_EXPAND_SZ
  Daten:           C:\Program Files\erl-23.0.1\erts-11.0.1\bin\erl.exe


Wert 3
  Name:            Env
  Typ:             REG_MULTI_SZ
  Daten:           APPDATA=C:\Users\blsadmin2\AppData\Roaming
                   ERL_LIBS=C:\Program Files\RabbitMQ Server\rabbitmq_server-logging-3.8.4\plugins
                   ERL_MAX_ETS_TABLES=50000
                   ERL_MAX_PORTS=65536


Wert 4
  Name:            WorkDir
  Typ:             REG_EXPAND_SZ
  Daten:           C:\Users\blsadmin2\AppData\Roaming\RabbitMQ


Wert 5
  Name:            Priority
  Typ:             REG_DWORD
  Daten:           0x20


Wert 6
  Name:            SName
  Typ:             REG_SZ
  Daten:          


Wert 7
  Name:            Name
  Typ:             REG_SZ
  Daten:           rabbit2@BLSENTWWEB1


Wert 8
  Name:            Args
  Typ:             REG_EXPAND_SZ
  Daten:            -s "rabbit" boot -boot "start_sasl" +W w +A "64" +MBas ageffcbf +MHas ageffcbf +MBlmbcs 512 +MHlmbcs 512 +MMmcs 30 +P 1048576 +t 5000000 +stbt db +zdbbl 128000  -rabbitmq_management listener [{port,15673}] -lager crash_log false -lager handlers "[]"


Wert 9
  Name:            DebugType
  Typ:             REG_DWORD
  Daten:           0


Wert 10
  Name:            InternalServiceName
  Typ:             REG_SZ
  Daten:           RabbitMQ-LOG


Wert 11
  Name:            Comment
  Typ:             REG_SZ
  Daten:           Multi-protocol open source messaging broker

I have no idea what i can look for ... 

Why is the message in the crash-dump saying unexpected_nodename ?  

Thank's for any help on this !

greetings richard.

[Luke Bakken]

Hi Richard,

It's very adventurous of you to get two RabbitMQ nodes running on the same Windows server. Just note that they will compete for resources.

Rather than modify the batch files that ship with RabbitMQ, you should create the following file when logged in as blsadmin2 -

%AppData%\RabbitMQ\rabbitmq-env-conf.bat

...with this content. Note that I removed USE_LONGNAME as you are not using long DNS names. This is the cause of the "unexpected nodename" error, I believe.

set SERVICENAME=RabbitMQ-LOG
set NODENAME=rabbit2@BLSENTWWEB1
set NODE_PORT=5673
set SERVER_ADDITIONAL_ERL_ARGS=-rabbitmq_management listener [{port,15673}]

There is no need to copy C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.4 to another directory if you use the above file.

Then, remove, re-install and re-start the RabbitMQ-LOG service, and I think you should be good to go.

Thanks,

Luke

[richard]

Hi Luke,

thanks for the fast reply...

Have tried your suggestions, but still crash-dump with same error...

Thanks for further help...

greetigs richard.

[Luke Bakken]

Hi Richard,

Please post the registry keys for the RabbitMQ and RabbitMQ-LOG services. Export to files and attach to your response.

Thanks,

Luke

[richard]

I have now restarted the server to see if there is some other loginfo...

In the eventlogs from windows there are following entries after restart and before logon:

•  ErlSrv, RabbitMQ-LOG,   Erlang service started successfully.
•  ErlSrv, RabbitMQ,   Erlang service started successfully. 
•  ErlSrv, RabbitMQ-LOG,   Restarted erlang machine. 
•  ErlSrv,  RabbitMQ-LOG,   Erlang machine seems to die continously, not restarted. 
•  Service Control Manager, RabbitMQ-LOG, param2 Beendet 
•  Service Control Manager, RabbitMQ-LOG,   param2 %%1067 

Afterwards i have tried to start the service again.... following eventlog entry is added:

• ErlSrv, RabbitMQ-LOG,   Erlang machine stopped instantly (distribution name conflict?). The service is not restarted, ignoring OnFail option. 

 

The crash-dump file still shows the same error

=erl_crash_dump:0.5
Thu May 28 20:59:24 2020

Slogan: Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{erlang_dist_running_with_unexpected_nodename,rabbit2@BLSENTWW
System version: Erlang/OTP 23 [erts-11.0.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64]

Compiled: Wed May 20 14:04:00 2020

Any new idea whats going on ?

Thanks,

greetings richard.

[richard]

Hi Luke,

please find the files attached...

greetings, richard.

[Luke Bakken]

Hi Richard,

From the RabbitMQ service registry keys, you can see the full name specified:

Wert 6
  Name:            SName
  Typ:             REG_SZ
  Daten:           rabbit@BLSENTWWEB

Here's the same value from RabbitMQ-LOG:

Wert 6
  Name:            SName
  Typ:             REG_SZ
  Daten:           rabbit2

Note that the hostname is not part of the value. In your previous message, you try to set the node name to rabbit2@BLSENTWWEB1 (with the extra 1 at the end). Do you intend to use different hostnames on the same server? They both must resolve in DNS.

Could you please attach the rabbitmq-env-conf.bat file that you created for the RabbitMQ-LOG service? Actually, just attach all of your configuration files and log files. making it clear which service they belong to.

Thanks,

Luke

[richard]

Hi Luke,

no, the value of the host is just not in the registry, but in the file... please see the attached ... it's not allowed to attach a .bat so i have renamed it to .txt...

Additionaly i have found this in the log of the first running node ... saying the config-file is not supported...?

2020-05-28 20:52:09.635 [notice] <0.278.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 20:52:09.673 [info] <0.278.0>
 Starting RabbitMQ 3.8.4 on Erlang 23.0.1
 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com
2020-05-28 20:52:09.674 [notice] <0.278.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 20:52:09.689 [notice] <0.278.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 20:52:09.710 [info] <0.278.0>
 node           : rabbit@BLSENTWWEB1
 home dir       : C:\Windows\system32\config\systemprofile
 config file(s) : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/advanced.config
 cookie hash    : LU62YF63fdafCdHLu+Z1Gw==
 log(s)         : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/log/rabbit@BLSENTWWEB1.log
                : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/log/rabbit@BLSENTWWEB1_upgrade.log
 database dir   : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/db/rabbit@BLSENTWWEB1-mnesia
2020-05-28 20:52:09.735 [info] <0.278.0> Running boot step pre_boot defined by app rabbit
2020-05-28 20:52:09.736 [info] <0.278.0> Running boot step rabbit_core_metrics defined by app rabbit

[richard]

Hi Luke,

if it is true that the rabbitmq-env-conf.bat is not loaded by the service on startup, how are the settings applied ?

greetings Richard.

[Luke Bakken]

Hi Richard,

There was a huge rewrite of how settings were loaded in RabbitMQ 3.8.4, and support for rabbitmq-env-conf.bat was lost, apparently.

Here is what you will have to do:

• Log in as user blsadmin
• Delete the rabbitmq-env-conf.bat file, which should only have been used for user blsadmin2 (as I instructed here https://groups.google.com/d/msg/rabbitmq-users/VgdYACHQ9vM/r0yUFZFNAAAJ) - C:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat
• Stop, remove, re-install, and re-start the RabbitMQ windows service. You should not see the "...is not implemented for Windows" message in the log file.
• Log in as user blsadmin2, and open the "RabbitMQ Command Prompt (sbin dir)" start menu item, or an admin command prompt
• Stop and remove the RabbitMQ-LOG windows service.
• Set these environment variables in your current command prompt session. Note that the RABBITMQ_ prefix is significant:

set RABBITMQ_SERVICENAME=RabbitMQ-LOG
set RABBITMQ_NODENAME=rabbit2@BLSENTWWEB1

set RABBITMQ_NODE_PORT=5673

set RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-rabbitmq_management listener [{port,15673}]

• You can now run the set command without arguments to double-check that the above variables are set.
• Run these commands to re-install the service:

.\rabbitmq-service.bat install

• Export the RabbitMQ-LOG registry keys. The SName value should be rabbit2@BLSENTWWEB1. If that's the case, proceed. If not, reply with the exported registry data.
• Start the RabbitMQ-LOG service:

.\rabbitmq-service.bat start

[richard]

Hi Luke,

there is no rabbitmq-env-conf.bat file under the blsadmin user folders.... only in the blsadmin2 folder.

I have removed both services, installed the first one under user blsadmin ... service is running OK.

Logged in as blsadmin2, running cmd shell with admin rights , seting the env variables as shown in your post.

Installed service, exported the registry key of the 2nd service ... values look ok ... see attached file.

Started the service ... again crash-dump with same error and servicee stopped.

The logfile of the 1st running service does still show the message loading of env-file not implemented on windows, even there is no such file there.

2020-05-28 22:28:15.559 [info] <0.277.0> Log file opened with Lager
2020-05-28 22:28:15.596 [debug] <0.293.0> Lager installed handler lager_backend_throttle into lager_event
2020-05-28 22:28:15.606 [info] <0.277.0> HiPE disabled: no modules were natively recompiled.
2020-05-28 22:28:32.748 [notice] <0.277.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 22:28:32.778 [info] <0.277.0>
 Starting RabbitMQ 3.8.4 on Erlang 23.0.1

 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com

2020-05-28 22:28:32.779 [notice] <0.277.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 22:28:32.794 [notice] <0.277.0> Loading of $RABBITMQ_CONF_ENV_FILE (c:/Users/blsadmin/AppData/Roaming/RabbitMQ/rabbitmq-env-conf.bat) is not implemented for Windows
2020-05-28 22:28:32.808 [info] <0.277.0>
 node           : rabbit@BLSENTWWEB1
 home dir       : C:\Windows\system32\config\systemprofile
 config file(s) : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/advanced.config
 cookie hash    : LU62YF63fdafCdHLu+Z1Gw==

 log(s)         : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/log/rabbit@BLSENTWWEB1.log
                : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/log/rabbit@BLSENTWWEB1_upgrade.log
 database dir   : c:/Users/blsadmin/AppData/Roaming/RabbitMQ/db/rabbit@BLSENTWWEB1-mnesia

2020-05-28 22:28:32.843 [info] <0.277.0> Running boot step pre_boot defined by app rabbit

greetings, Richard.

[richard]

Hi Luke,

one more info i have found....

If a change the account under which the service is running from Systemaccount to blsadmin2, the service starts and continues to runn, but cant start the listeners because of using the same ports of the 1st allready running node.

The info in the log is:

2020-05-28 23:29:34.706 [error] <0.537.0> Failed to start Ranch listener {acceptor,{0,0,0,0,0,0,0,0},5672} in ranch_tcp:listen([{cacerts,'...'},{key,'...'},{cert,'...'},{ip,{0,0,0,0,0,0,0,0}},{port,5672},inet6,{backlog,128},{nodelay,true},{linger,{true,0}},{exit_on_close,false}]) for reason eaddrinuse (address already in use)
2020-05-28 23:29:34.707 [error] <0.535.0> Supervisor {<0.535.0>,ranch_listener_sup} had child ranch_acceptors_sup started with ranch_acceptors_sup:start_link({acceptor,{0,0,0,0,0,0,0,0},5672}, ranch_tcp) at undefined exit with reason {listen_error,{acceptor,{0,0,0,0,0,0,0,0},5672},eaddrinuse} in context start_error
2020-05-28 23:29:34.707 [error] <0.278.0> Failed to start TCP listener [::]:5672, error: eaddrinuse
2020-05-28 23:29:34.707 [error] <0.537.0> CRASH REPORT Process <0.537.0> with 0 neighbours exited with reason: {listen_error,{acceptor,{0,0,0,0,0,0,0,0},5672},eaddrinuse} in ranch_acceptors_sup:listen_error/5 line 66
2020-05-28 23:29:34.708 [error] <0.534.0> Supervisor {<0.534.0>,tcp_listener_sup} had child {ranch_listener_sup,{acceptor,{0,0,0,0,0,0,0,0},5672}} started with ranch_listener_sup:start_link({acceptor,{0,0,0,0,0,0,0,0},5672}, ranch_tcp, #{connection_type => supervisor,handshake_timeout => 5000,max_connections => infinity,num_acceptors => ...,...}, rabbit_connection_sup, []) at undefined exit with reason {shutdown,{failed_to_start_child,ranch_acceptors_sup,{listen_error,{acceptor,{0,0,0,0,0,0,0,0},5672},eaddrinuse}}} in context start_error
2020-05-28 23:29:34.737 [info] <0.44.0> Application mnesia exited with reason: stopped
2020-05-28 23:29:34.737 [error] <0.278.0>
2020-05-28 23:29:34.738 [error] <0.278.0> BOOT FAILED
2020-05-28 23:29:34.739 [error] <0.278.0> ===========
2020-05-28 23:29:34.739 [error] <0.278.0> Error during startup: {error,{could_not_start_listener,"::",5672,eaddrinuse}}
2020-05-28 23:29:34.740 [error] <0.278.0>
2020-05-28 23:29:35.735 [error] <0.277.0> CRASH REPORT Process <0.277.0> with 0 neighbours exited with reason: {{could_not_start_listener,"::",5672,eaddrinuse},{rabbit,start,[normal,[]]}} in application_master:init/4 line 138
2020-05-28 23:29:35.736 [info] <0.44.0> Application rabbit exited with reason: {{could_not_start_listener,"::",5672,eaddrinuse},{rabbit,start,[normal,[]]}}
2020-05-28 23:29:35.737 [info] <0.489.0> Closing all connections in vhost '/' on node 'rabbit2@BLSENTWWEB1' because the vhost is stopping
2020-05-28 23:29:35.738 [info] <0.505.0> Stopping message store for directory 'c:/Users/blsadmin2/AppData/Roaming/RabbitMQ/db/rabbit2@BLSENTWWEB1-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'

and the erl-crash.dump reads:

=erl_crash_dump:0.5
Thu May 28 23:32:16 2020
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{could_not_start_listener,"::",5672,eaddrinuse},{rabbit,start,[normal,[]]}}})

System version: Erlang/OTP 23 [erts-11.0.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64]
Compiled: Wed May 20 14:04:00 2020

So from my point of view it looks like since the config-files where the ports are defined are not read by the service-start, it is not possible to runn more than one service node on same windows.machine.

Hopefully you find something how to set the ports or to load the config-file for the 2nd node...

Is there any disadvantage starting a node not as service (beside to login on the machine) ?

For today i will close now...

Thanks,

greetings Richard.

[Luke Bakken]

Hi Richard,

OK, we're getting closer. The RABBITMQ_NODE_PORT is not set in the registry when the service is installed (as I thought) but when RabbitMQ starts up. If the rabbitmq-env-conf.bat file were working, that file would be read when the service starts, and you'd be set.

But, what you can do since you have modified the RabbitMQ-LOG service to run in the blsadmin2 account, is to set a user environment variable via the "system properties" dialog -

RABBITMQ_NODE_PORT=5673

You don't have to remove and re-install the service after that change. Just open a new admin command prompt to ensure the variable is set for that user, and re-start the service. It will be picked up.

We're working on fixing the rabbitmq-env-conf.bat support right now.

Thanks,

Luke

[richard]

Hi Luke,

so this means this user have to be logged in to let the service running... otherwise the user-env is not set... and maybe the service must be delayed at startup, because login is later than service startup..

Is there any drawback , beside someone must be logged in, to start the node as app .. not as service ?

Because with service nodes now (without env-config gfile) there are only 2 nodes possible ... if thy are started as app ... there ist theoreticaly not this limit.

Thanks , richard.

[richard]

Hi Luke,

i have now tested this behavior ....

It is working if the variables(i have set all 4 variables) are set by the user, and the service is running under this account... this way i think also more than 2 service nodes are able to run, and the service does not have to delayed at startup,  also no user must login !

Thanks, richard.

[Luke Bakken]

I'm glad it works. Thank you for following up.