Cannot start mqtt plugin after enabling and disabling rabbitmq_event_exchange

[CJ]

Hi I am new to rabbitmq and am trying to set up rabbitmq using mqtt. Here are the plugins I have enabled and running fine.

[e*] amqp_client                       3.6.2
[e*] cowboy                            1.0.3
[e*] cowlib                            1.0.1
[e*] mochiweb                          2.13.1
[  ] rabbitmq_amqp1_0                  3.6.2
[E*] rabbitmq_auth_backend_cache      
[E*] rabbitmq_auth_backend_http        
[  ] rabbitmq_auth_backend_ldap        3.6.2
[  ] rabbitmq_auth_mechanism_ssl       3.6.2
[  ] rabbitmq_consistent_hash_exchange 3.6.2
[  ] rabbitmq_event_exchange           3.6.2
[  ] rabbitmq_federation               3.6.2
[  ] rabbitmq_federation_management    3.6.2
[E*] rabbitmq_management               3.6.2
[e*] rabbitmq_management_agent         3.6.2
[  ] rabbitmq_management_visualiser    3.6.2
[  ] rabbitmq_mqtt                     3.6.2
[  ] rabbitmq_recent_history_exchange  1.2.1
[  ] rabbitmq_sharding                 0.1.0
[  ] rabbitmq_shovel                   3.6.2
[  ] rabbitmq_shovel_management        3.6.2
[e*] rabbitmq_stomp                    3.6.2
[  ] rabbitmq_tracing                  3.6.2
[e*] rabbitmq_web_dispatch             3.6.2
[  ] rabbitmq_web_mqtt                
[E*] rabbitmq_web_stomp                3.6.2
[  ] rabbitmq_web_stomp_examples       3.6.2
[e*] sockjs                            0.3.4
[e*] webmachine                        1.10.3

The moment I am trying to enable: rabbitmq-plugins enable rabbitmq_mqtt I get the following:

The following plugins have been enabled:
  rabbitmq_mqtt

Applying plugin configuration to rabbit@CentOS-72-64-minimal... failed.
Error: {could_not_start,rabbitmq_mqtt,
           {{shutdown,
                {failed_to_start_child,rabbit_mqtt_retainer_sup,
                    {badarg,
                        [{erlang,binary_to_atom,["/",utf8],[]},
                         {rabbit_mqtt_retainer_sup,
                             '-child_specs/2-lc$^0/1-0-',2,
                             [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                              {line,60}]},
                         {rabbit_mqtt_retainer_sup,init,1,
                             [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                              {line,57}]},
                         {supervisor2,init,1,
                             [{file,"src/supervisor2.erl"},{line,350}]},
                         {gen_server,init_it,6,
                             [{file,"gen_server.erl"},{line,328}]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,240}]}]}}},
            {rabbit_mqtt,start,[normal,[]]}}}

This has worked perfectly before, but does not after I enabled rabbitmq_event_exchange, I subsequently disabled rabbitmq_event_exchange again.

I cannot find anything online to guide me to understand what I am doing wrong.
Any help would be appreciated.

[Michael Klishin]

MQTT plugin retainer (which is in memory and node-local by default) fails to start. I cannot think of
how rabbitmq_event_exchange could affect this. What's in the log files?

Restarting the node should help since no retainer state is stored on disk (by default).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[CJ]

Thanks for your response,

I just came to realise something else is causing this and not related to the event exchange.

I have reinstalled the server and cleared out: /var/lib/rabbitmq/mnesia

yum install esl-erlang_18.3-1~centos~7_amd64.rpm

yum install esl-erlang-compat-R1801-1.noarch.rpm

yum install rabbitmq-server-3.6.2-1.noarch.rpm 

cp rabbitmq_auth_backend_cache.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.2/plugins/

cp rabbitmq_auth_backend_http-3.6.x-3dfe5950.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.2/plugins/

Config:

[
 {ssl, [{versions, ['tlsv1.2', 'tlsv1.1', tlsv1]}]},
 {rabbit,
  [
   {ssl_listeners, [5671]},
   {log_levels, [{connection, info}]},
   {loopback_users, ["test"]},
   {loopback_users, []},
   {ssl_options, [{cacertfile,           "/opt/mycerts/CAcert.pem"},
                  {certfile,             "/opt/locker/certs/rabbitserver.pem"},
                  {keyfile,              "/opt/locker/certs/rabbitserver.key"},
                  {versions, ['tlsv1.2', 'tlsv1.1', tlsv1]},
                  {verify,               verify_none},
                  {fail_if_no_peer_cert, false}]},
   {auth_backends, [rabbit_auth_backend_cache]},
   {default_vhost,       "/"},
   {default_user,        "guest"},
   {default_pass,        "guest"},
   {default_permissions, [".*", ".*", ".*"]},
   {default_user_tags, [administrator]}
  ]},
 {rabbitmq_mqtt,
  [%% Set the default user name and password. Will be used as the default login
  ]},
  {rabbitmq_auth_backend_http,
   [{http_method,  post},
    {user_path,     "http://auth.myauthserver.com/rmq_auth_user.php"},
    {vhost_path,    "http://auth.myauthserver.com/rmq_auth_vhost.php"},
    {resource_path, "http://auth.myauthserver.com/rmq_auth_resource.php"}]},
  {rabbitmq_auth_backend_cache,
   [{cached_backend, rabbit_auth_backend_http},
    {cache_ttl, 60000}]}
].

Only enabled the plugins:

rabbitmq-plugins enable rabbitmq_auth_backend_http

rabbitmq-plugins enable rabbitmq_auth_backend_cache

service rabbitmq-server restart

Ports open in the firewall: 5671/tcp 5672/tcp 15675/tcp 4369/tcp 15672/tcp

Everything is fine so far:

[root@CentOS-72-64-minimal plugins]# rabbitmq-plugins list
 Configured: E = explicitly enabled; e = implicitly enabled
 | Status:   * = running on rabbit@CentOS-72-64-minimal
 |/
[e*] amqp_client                       3.6.2
[  ] cowboy                            1.0.3
[  ] cowlib                            1.0.1

[e*] mochiweb                          2.13.1
[  ] rabbitmq_amqp1_0                  3.6.2
[E*] rabbitmq_auth_backend_cache      
[E*] rabbitmq_auth_backend_http        
[  ] rabbitmq_auth_backend_ldap        3.6.2
[  ] rabbitmq_auth_mechanism_ssl       3.6.2
[  ] rabbitmq_consistent_hash_exchange 3.6.2
[  ] rabbitmq_event_exchange           3.6.2
[  ] rabbitmq_federation               3.6.2
[  ] rabbitmq_federation_management    3.6.2

[  ] rabbitmq_management               3.6.2
[  ] rabbitmq_management_agent         3.6.2

[  ] rabbitmq_management_visualiser    3.6.2
[  ] rabbitmq_mqtt                     3.6.2
[  ] rabbitmq_recent_history_exchange  1.2.1
[  ] rabbitmq_sharding                 0.1.0
[  ] rabbitmq_shovel                   3.6.2
[  ] rabbitmq_shovel_management        3.6.2

[  ] rabbitmq_stomp                    3.6.2
[  ] rabbitmq_tracing                  3.6.2
[  ] rabbitmq_web_dispatch             3.6.2
[  ] rabbitmq_web_mqtt                
[  ] rabbitmq_web_stomp                3.6.2
[  ] rabbitmq_web_stomp_examples       3.6.2
[  ] sockjs                            0.3.4
[  ] webmachine                        1.10.3

But again: rabbitmq-plugins enable rabbitmq_mqtt gives me the same: This what I can see in the logfile located here: /var/log/rabbitmq/rabbit\@CentOS-72-64-minimal-sasl.log

=CRASH REPORT==== 27-Jun-2016::15:47:21 ===
  crasher:
    initial call: supervisor2:init/1
    pid: <0.340.0>
    registered_name: []
    exception exit: {badarg,

                        [{erlang,binary_to_atom,["/",utf8],[]},
                         {rabbit_mqtt_retainer_sup,
                             '-child_specs/2-lc$^0/1-0-',2,
                             [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                              {line,60}]},
                         {rabbit_mqtt_retainer_sup,init,1,
                             [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                              {line,57}]},
                         {supervisor2,init,1,
                             [{file,"src/supervisor2.erl"},{line,350}]},
                         {gen_server,init_it,6,
                             [{file,"gen_server.erl"},{line,328}]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,240}]}]}

      in function  gen_server:init_it/6 (gen_server.erl, line 352)
    ancestors: [rabbit_mqtt_sup,<0.337.0>]
    messages: []
    links: [<0.338.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 379
  neighbours:

=SUPERVISOR REPORT==== 27-Jun-2016::15:47:21 ===
     Supervisor: {local,rabbit_mqtt_sup}
     Context:    start_error
     Reason:     {badarg,

                     [{erlang,binary_to_atom,["/",utf8],[]},
                      {rabbit_mqtt_retainer_sup,'-child_specs/2-lc$^0/1-0-',
                          2,
                          [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                           {line,60}]},
                      {rabbit_mqtt_retainer_sup,init,1,
                          [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                           {line,57}]},
                      {supervisor2,init,1,
                          [{file,"src/supervisor2.erl"},{line,350}]},
                      {gen_server,init_it,6,
                          [{file,"gen_server.erl"},{line,328}]},
                      {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,240}]}]}

     Offender:   [{pid,undefined},
                  {name,rabbit_mqtt_retainer_sup},
                  {mfargs,
                      {rabbit_mqtt_retainer_sup,start_link,
                          [{local,rabbit_mqtt_retainer_sup}]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,supervisor}]


=CRASH REPORT==== 27-Jun-2016::15:47:21 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.336.0>
    registered_name: []
    exception exit: {{shutdown,

                      {failed_to_start_child,rabbit_mqtt_retainer_sup,
                       {badarg,
                        [{erlang,binary_to_atom,["/",utf8],[]},
                         {rabbit_mqtt_retainer_sup,
                          '-child_specs/2-lc$^0/1-0-',2,
                          [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                           {line,60}]},
                         {rabbit_mqtt_retainer_sup,init,1,
                          [{file,"src/rabbit_mqtt_retainer_sup.erl"},
                           {line,57}]},
                         {supervisor2,init,1,
                          [{file,"src/supervisor2.erl"},{line,350}]},
                         {gen_server,init_it,6,
                          [{file,"gen_server.erl"},{line,328}]},
                         {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,240}]}]}}},
                     {rabbit_mqtt,start,[normal,[]]}}

      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.335.0>]
    messages: [{'EXIT',<0.337.0>,normal}]
    links: [<0.335.0>,<0.7.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 183
  neighbours:

Hope that helps, all I can see is references to rabbit_mqtt_retainer_sup

[CJ]

I did some further testing, I managed to get my server going again as follows (cleared the db):

1. Enabled all the plugins including the rabbitmq_event_exchange with the default config and it runs.

2. Enabled all the plugins excluding the rabbitmq_event_exchange with the my config and it runs. But the moment I enabled the rabbitmq_event_exchange plugin it corrupts the db and fails to restart, with the same error. So this looks like my configuration enables the crash with this plugin.

[Michael Klishin]

default_vhost, default_user, default_pass either must be excluded (you use
all default values) or provided as binaries:

<<"/">>, <<"guest">>, <<"guest">>

We now handle it in a bunch of places but not MQTT retainer, apparently. Filed a usability issue:

https://github.com/rabbitmq/rabbitmq-mqtt/issues/86

But what I'd recommend is simply excluding those keys since you use their default values anyway.

[CJ]

Hi Michael,

Thanks for looking into this, that worked like a charm, (you were right, I didn't need those values in any case).

Much appreciated.