epmd reports: node 'rabbit' not running at all

[htang]

Hi, This is not new topic, but I did search all the past topics related to this error, but it is all different from my case here.

Some background, I am using centOS 7, installed RabbitMQ (RPM) 3.6.10 (have to use 3.6.x due to other related projects). After install, I am able to start and stop server ok, and able to run java clients with publisher and consumer using AMQP protocol, no users authentication for these test clients.  However I need to use MQTT plugin, and have to setup users to access the RabbitMQ broker from different host. And the problem starts when I try to add rabbitmq.config file under /etc/rabbitmq folder. The config files does not exist in that folder during the installation. If I remove the rabbitmq.config file, I can execute "sudo systemctl start rabbitmq-server"  to start the server successfully.

errors:

sudo systemctl start rabbitmq-server
[sudo] password for htang:
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journal ctl -xe" for details.

[htang@sdt2 ~]$ sudo systemctl status rabbitmq-server.service -l
[sudo] password for htang:
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2018-03-05 21:21:54 EST; 46min ago
  Process: 8173 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
  Process: 7814 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 7814 (code=exited, status=1/FAILURE)
   Status: "Exited."

Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: rabbit@sdt2:
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * connected to epmd (port 4369) on sdt2
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * epmd reports: node 'rabbit' not running at all
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: no other nodes on sdt2
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * suggestion: start the node
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: current node details:
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: - node name: 'rabbitmq-cli-51@sdt2'
Mar 05 21:21:54 sdt2.corp.cox.com systemd[1]: Failed to start RabbitMQ broker.
Mar 05 21:21:54 sdt2.corp.cox.com systemd[1]: Unit rabbitmq-server.service entered failed state.
Mar 05 21:21:54 sdt2.corp.cox.com systemd[1]: rabbitmq-server.service failed.
[htang@sdt2 ~]$

 sudo journalctl -xe -l

--------------------------------------

Mar 05 22:10:01 sdt2.corp.cox.com systemd[1]: Starting Session 187330 of user root.
-- Subject: Unit session-187330.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-187330.scope has begun starting up.
Mar 05 22:10:01 sdt2.corp.cox.com CROND[8685]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 05 22:10:53 sdt2.corp.cox.com sshd[8693]: Connection closed by 10.11.5.38 [preauth]
Mar 05 22:12:24 sdt2.corp.cox.com sudo[8696]:    htang : TTY=pts/1 ; PWD=/home/htang ; USER=root ; COMMAND=/bin/journalctl -xe -l
lines 3081-3129/3129 (END)

and the following is my rabbitmq.config file

[{rabbit,        [{tcp_listeners,    [5672]}]},
 {rabbitmq_mqtt, [{default_user,     <<"mqtt-test">>},
                  {default_pass,     <<"mqtt-test">>},
                  {allow_anonymous,  true},
                  {vhost,            <<"/">>},
                  {exchange,         <<"amq.topic">>},
                  {subscription_ttl, 1800000},
                  {prefetch,         10},
                  {ssl_listeners,    []},
                  %% Default MQTT with TLS port is 8883
                  %% {ssl_listeners,    [8883]}
                  {tcp_listeners,    [1883]},
                  {tcp_listen_options, [{backlog,   128},
                                        {nodelay,   true}]}]}
]

please let me know what I did is wrong, really appreciate any help. Thanks.

[htang]

I found the problem, I missed "dot" at the end of the rabbitmq.config file, now the server can start w/o problem.  Now I face the different issue below, will check log file to see why.

got exception:com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

java.lang.NullPointerException

[htang]

The following is the error from the log file. It is weird, it still uses default guest.  I have already provide the rabbitmq.config file with different user mqtt-test, and I also deleted guest in the admin console, and add and enable the user "mqtt-test" via admin console.  According to the rabbitmq document, the config file should overwrite the run time default user.

=INFO REPORT==== 6-Mar-2018::08:00:05 ===

started MQTT TCP Listener on [::]:1883

=INFO REPORT==== 6-Mar-2018::08:00:05 ===

Server startup complete; 7 plugins started.

 * rabbitmq_mqtt

 * rabbitmq_management

 * rabbitmq_web_dispatch

 * rabbitmq_management_agent

 * cowboy

 * cowlib

 * amqp_client

=INFO REPORT==== 6-Mar-2018::08:00:45 ===

accepting AMQP connection <0.484.0> (10.11.4.73:56756 -> 10.11.5.42:5672)

=ERROR REPORT==== 6-Mar-2018::08:00:45 ===

Error on AMQP connection <0.484.0> (10.11.4.73:56756 -> 10.11.5.42:5672, state: starting):

PLAIN login refused: user 'guest' - invalid credentials

=INFO REPORT==== 6-Mar-2018::08:00:45 ===

closing AMQP connection <0.484.0> (10.11.4.73:56756 -> 10.11.5.42:5672)

On Monday, March 5, 2018 at 10:20:50 PM UTC-5, htang wrote:

[Michael Klishin]

Please start with effective configuration verification:
http://www.rabbitmq.com/configure.html#configuration-files.

Also note that the default_user and default_pass will only be used

if the client provides no credentials.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[htang]

Hi, Michael,  Thank you very much for the help to direct the place to debug.  I did list the effective environment, and tried with various configurations, even use the force-reset,  what I found is that even I set the default_user at rabbit and rabbitmq_mqtt for different user, say "mqtt-test", it still takes the guest as user no matter what (see that in log file).  Even my client program sets the user as "mqtt-test".

If I removed the guest user from admin console, it will never work. when I add "guest" user back and specify loopback user null,  it works and the client programs can pass user authentication and establish the connection. So in my case it always uses the "guest" user, and still not sure what I did wrong that i could not "truly" use other user rather than "guest". Any other suggestions for debugging?

My rabbitmq.config file

[{rabbit,        [

                 {cluster_nodes, {['rabbit@sdt2', 'rabbit@sdt2'], disc}},

                 {default_user,<<"mqtt-test">>},

                 {default_pass,<<"mqtt-test">>},

                 {loopback_users, []},

                 {tcp_listeners,    [5672]}]},

 {rabbitmq_mqtt, [{default_user,     <<"mqtt-test">>},

                  {default_pass,     <<"mqtt-test">>},

                  {allow_anonymous,  true},

                  {vhost,            <<"/">>},

                  {exchange,         <<"amq.topic">>},

                  {subscription_ttl, 1800000},

                  {prefetch,         10},

                  {ssl_listeners,    []},

                  %% Default MQTT with TLS port is 8883

                  %% {ssl_listeners,    [8883]},

                  {tcp_listeners,    [1883]},

                  {tcp_listen_options, [{backlog,   128},

                                        {nodelay,   true}]}]}

].

lines from environment

{rabbit,

     [{auth_backends,[rabbit_auth_backend_internal]},

      {auth_mechanisms,['PLAIN','AMQPLAIN']},

      {background_gc_enabled,false},

      {background_gc_target_interval,60000},

      {backing_queue_module,rabbit_priority_queue},

      {channel_max,0},

      {channel_operation_timeout,15000},

      {cluster_keepalive_interval,10000},

      {cluster_nodes,{[rabbit@sdt2,rabbit@sdt2],disc}},

      {cluster_partition_handling,ignore},

      {collect_statistics,fine},

      {collect_statistics_interval,5000},

      {config_entry_decoder,

          [{cipher,aes_cbc256},

           {hash,sha512},

           {iterations,1000},

           {passphrase,undefined}]},

      {credit_flow_default_credit,{400,200}},

      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

      {default_user,<<"mqtt-test">>},

      {default_user_tags,[administrator]},

      {default_vhost,<<"/">>},

      {delegate_count,16},

      {disk_free_limit,50000000},

      {disk_monitor_failure_retries,10},

log files:

=INFO REPORT==== 7-Mar-2018::09:13:51 ===

accepting AMQP connection <0.607.0> (10.11.4.73:58934 -> 10.11.5.42:5672)

=INFO REPORT==== 7-Mar-2018::09:13:51 ===

connection <0.607.0> (10.11.4.73:58934 -> 10.11.5.42:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 7-Mar-2018::09:13:52 ===

MQTT vhost picked using plugin configuration or default

=INFO REPORT==== 7-Mar-2018::09:13:52 ===

accepting MQTT connection <0.623.0> (10.11.4.73:58935 -> 10.11.5.42:1883)

my Code:

     conOpts.setUserName("mqtt-test");

     conOpts.setPassword("mqtt-test".toCharArray());

     conOpts.setCleanSession(true);

     conOpts.setKeepAliveInterval(60);

Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: rab...@sdt2:

Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * connected to epmd (port 4369) on sdt2
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * epmd reports: node 'rabbit' not running at all
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: no other nodes on sdt2
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: * suggestion: start the node
Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: current node details:

Mar 05 21:21:54 sdt2.corp.cox.com rabbitmqctl[8173]: - node name: 'rabbitm...@sdt2'

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hello,

I tried to reproduce what you report using RabbitMQ 3.6.10 and Erlang 19.3. Here are the steps I took:

* Start RabbitMQ with the following configuration:

[

    {rabbitmq_mqtt, [

        {default_user,    <<"mqtt-test">>},

        {default_pass,    <<"mqtt-test">>},

        {allow_anonymous, true},

        {vhost,           <<"/">>},

        {exchange,        <<"amq.topic">>}

    ]}

].

* Create the mqtt-test user and assign permissions:

rabbitmqctl add_user mqtt-test mqtt-test

rabbitmqctl set_permissions -p / mqtt-test '.*' '.*' '.*'

* Delete the guest user:

rabbitmqctl delete_user guest

* Create an admin user and set up a queue and binding -

rabbitmqctl add_user admin test1234

rabbitmqctl set_user_tags admin administrator

rabbitmqctl set_permissions -p / admin '.*' '.*' '.*'

rabbitmqadmin --username=admin --password=test1234 declare queue name=mqtt-test

rabbitmqadmin --username=admin --password=test1234 declare binding source=amq.topic destination=mqtt-test routing_key=hello

* Publish a message:

mosquitto_pub -d -h 127.0.0.1 -p 1883 -q 0 -t hello -m "Hello World"

When I check the mqtt-test queue, the message is there. Note that by not specifying the user/password when running mosquitto_pub it is using anonymous access, which must be using the mqtt-test user

Thanks,

Luke

[htang]

Hi, Luke,

 Thanks for making efforts to reproduce and try this.  Just want to clarify a few points here, what you are doing here is to set up queue name "mqtt-test", and it happens to be the default_user name you have in the config.  is rabitmq mqtt using the same user name as its "topic/queue" name?  It would be good if you can validate the rabbit log file in your case to see who is the user when publishing the message, whether it is "guest" or "mqtt-test", in my case, even I removed the guest user, the log file clearly indicate it is "guest" user.  Also in my test, I used different "topic" name (mqtt-topic) from the user name (mqtt-test), I explicitly set the user names in my java program as following.

     conOpts.setUserName("mqtt-test");

     conOpts.setPassword("mqtt-test".toCharArray());

It works fine with MQTT publisher and subscriber, i.e. publish and receiving is good, however the rabbit log file just "stubbornly" showing it is "guest" user in the connection if I did not remove the guest user. see below log files.

=INFO REPORT==== 7-Mar-2018::15:35:55 ===

accepting AMQP connection <0.9296.0> (10.11.4.73:58566 -> 10.11.5.42:5672)

=INFO REPORT==== 7-Mar-2018::15:35:55 ===

connection <0.9296.0> (10.11.4.73:58566 -> 10.11.5.42:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 7-Mar-2018::15:35:55 ===

MQTT vhost picked using plugin configuration or default

If I remove the guest user, then it is not working no matter what.

I also checked my effective environment parameters, and the rabbit default_user is "mqtt-test" not "guest".  

I may try 3.7.x to see if this is same issue or not.

Thanks again for the help.

Harry

[htang]

Hi, Luke, Michael;

   I just examined the code and tried different way to set the username and password, and now I figured it out and guess what could be the reason behind it.  There are two places one can set username and password for MQTT client or publisher  1. At the MqttConnectOptions level as connection options parameters, 2. at the RabbitMQ client connectionFactory level.

 Before I always set username and password at MqttConnectOptions level after RabbitMQ connection instance is created. That is the problem. connectionFactory can spin many connections, RabbitMQ is looking for username and password when connection is created, if it does not see any, it tries to use default_user ("guest") - this part I still don't understand why RabbitMQ uses "guest" not "mqtt-test" from the configuration (even environment confirms the default is "mqtt-test").  Anyway when I set the username and password at connectionFactory with "mqtt-test", the log file shows the right user properly. see below log file. 

=INFO REPORT==== 7-Mar-2018::20:32:15 ===

accepting AMQP connection <0.11989.0> (10.24.150.184:62402 -> 10.11.5.42:5672)

=INFO REPORT==== 7-Mar-2018::20:32:15 ===

connection <0.11989.0> (10.24.150.184:62402 -> 10.11.5.42:5672): user 'mqtt-test' authenticated and granted access to vhost '/'

=INFO REPORT==== 7-Mar-2018::20:32:16 ===

MQTT vhost picked using plugin configuration or default

And the java code to set user/password at factory level is:

connectionFactory = new ConnectionFactory();

     connectionFactory.setHost(host);

     connectionFactory.setUsername("mqtt-test");

     connectionFactory.setPassword("mqtt-test");

     conn = connectionFactory.newConnection();

MQTT does not have connectionFactory for its client, it is needed by RabbitMQ to bind the queue to topic as following steps and flow. So if one wants to use specific user, it needs to be set at connectionFactory level even before connection is created (or maybe during the connection is created), otherwise it will use the default_user.

conntionFactory->Connection->Channel->bind Queue to Topic via AMQ.topic exchange, after all these setup, then MQTT client can publish to the topic.

On Wednesday, March 7, 2018 at 12:36:38 PM UTC-5, Luke Bakken wrote:

[Luke Bakken]

Hi Harry -

When you set username and password at the MqttConnectOptions level the library you use must have a bug where it doesn't send authentication information, or perhaps it actually sends guest. If you use a network tracing tool like Wireshark you can confirm what is actually sent to RabbitMQ. If you provide a complete set of code that I can compile and run that reproduces the issue I will check it out.

You asked - "it would be good if you can validate the rabbit log file in your case to see who is the user when publishing the message" - yes, the mqtt-test user is the one that shows up in the "authenticated and granted access" log message. There is no correlation between user name and queue name used by RabbitMQ.

Thanks,

Luke

[htang]

Thanks, Luke, I will try more debugging on why it uses guest for authentication,not the specified "mqtt-test" user.