Crash: MQTT + Auth backend HTTP

[Rafael Bachiller]

Hi all,

I've just updated RabbitMQ from 3.6.15 to 3.7.3 and after a few hours of testing, I've realised that the MQTT publishing makes RabbitMQ crashing, but subscribing works correctly. I'm using auth backend http for authorisation (if I use the internal one, it works correctly). Has anyone any idea what I'm doing wrong? Thanks in advance!!

This is the content of rabbitmq.conf:

loopback_users.guest = false

auth_backends.1 = rabbit_auth_backend_http

auth_http.http_method = post

auth_http.user_path = http://localhost:8080/auth/user

auth_http.vhost_path = http://localhost:8080/auth/vhost

auth_http.resource_path = http://localhost:8080/auth/resource

auth_http.topic_path = http://localhost:8080/auth/topic

I'm using the example http server (Spring) included in the plugin documentation.

This is the error that appears in the log:

2018-02-18 00:01:47.972 [info] <0.631.0> MQTT vhost picked using plugin configuration or default

2018-02-18 00:01:47.979 [info] <0.631.0> accepting MQTT connection <0.631.0> (127.0.0.1:53996 -> 127.0.0.1:1883)

2018-02-18 00:01:47.990 [info] <0.631.0> closing MQTT connection <0.631.0> (127.0.0.1:53996 -> 127.0.0.1:1883)

2018-02-18 00:01:47.991 [error] <0.644.0> ** Generic server <0.644.0> terminating

** Last message in was {'$gen_cast',{method,{'basic.publish',0,<<"amq.topic">>,<<"a">>,false,false},{content,60,{'P_basic',undefined,undefined,[{<<"x-mqtt-publish-qos">>,byte,0},{<<"x-mqtt-dup">>,bool,false}],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"1010/9002,0/153,0">>]},flow}}

** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.641.0>,<0.641.0>,<0.634.0>,<<"127.0.0.1:53996 -> 127.0.0.1:1883">>,{lstate,<0.643.0>,false},none,1,{[],[]},{user,<<"guest">>,[administrator,management],[{rabbit_auth_backend_http,none}]},<<"/">>,<<>>,#{},{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},erlang},#{},#{},{set,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<0.638.0>,{state,none,5000,undefined},false,1,{{0,nil},{0,nil}},[],[],{{0,nil},{0,nil}},[{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,bool,true},{<<"consumer_cancel_notify">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true}],none,10,none,flow,[]}

** Reason for termination == 

** {{noproc,{gen_server,call,[<0.634.0>,{info,[amqp_params]},15000]}},[{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},{rabbit_channel,check_topic_authorisation,5,[{file,"src/rabbit_channel.erl"},{line,855}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1081}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,523}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

2018-02-18 00:01:47.992 [error] <0.644.0> CRASH REPORT Process <0.644.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(<0.634.0>, {info,[amqp_params]}, 15000) in gen_server:call/3 line 214 in gen_server2:terminate/3 line 1166

2018-02-18 00:01:47.993 [error] <0.642.0> Supervisor {<0.642.0>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.641.0>, <0.641.0>, <0.634.0>, <<"127.0.0.1:53996 -> 127.0.0.1:1883">>, rabbit_framing_amqp_0_9_1, {user,<<"guest">>,[administrator,management],[{rabbit_auth_backend_http,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.638.0>, <0.643.0>) at <0.644.0> exit with reason no such process or port in call to gen_server:call(<0.634.0>, {info,[amqp_params]}, 15000) in gen_server:call/3 line 214 in context child_terminated

2018-02-18 00:01:47.993 [error] <0.642.0> Supervisor {<0.642.0>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.641.0>, <0.641.0>, <0.634.0>, <<"127.0.0.1:53996 -> 127.0.0.1:1883">>, rabbit_framing_amqp_0_9_1, {user,<<"guest">>,[administrator,management],[{rabbit_auth_backend_http,none}]}, <<"/">>, [{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,...},...], <0.638.0>, <0.643.0>) at <0.644.0> exit with reason reached_max_restart_intensity in context shutdown

This is the result of rabbitmqctl status:

Status of node rabbit@ip-172-31-21-187 ...

[{pid,32527},

 {running_applications,

     [{rabbitmq_mqtt,"RabbitMQ MQTT Adapter","3.7.4-rc.1"},

      {rabbitmq_auth_backend_http,"RabbitMQ HTTP Authentication Backend",

          "3.7.4-rc.1"},

      {amqp_client,"RabbitMQ AMQP Client","3.7.4-rc.1"},

      {rabbit,"RabbitMQ","3.7.4-rc.1"},

      {rabbit_common,

          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",

          "3.7.4-rc.1"},

      {ranch_proxy_protocol,"Ranch Proxy Protocol Transport","1.4.4"},

      {ranch,"Socket acceptor pool for TCP protocols.","1.4.0"},

      {ssl,"Erlang/OTP SSL application","8.2.1"},

      {public_key,"Public key infrastructure","1.5"},

      {asn1,"The Erlang ASN1 compiler version 5.0.3","5.0.3"},

      {os_mon,"CPO  CXC 138 46","2.4.3"},

      {crypto,"CRYPTO","4.1"},

      {xmerl,"XML parser","1.3.15"},

      {jsx,"a streaming, evented json parsing toolkit","2.8.2"},

      {mnesia,"MNESIA  CXC 138 12","4.15.1"},

      {inets,"INETS  CXC 138 49","6.4.2"},

      {recon,"Diagnostic tools for production use","2.3.2"},

      {lager,"Erlang logging framework","3.5.1"},

      {goldrush,"Erlang event stream processor","0.1.9"},

      {compiler,"ERTS  CXC 138 10","7.1.2"},

      {syntax_tools,"Syntax tools","2.1.3"},

      {sasl,"SASL  CXC 138 11","3.1"},

      {stdlib,"ERTS  CXC 138 10","3.4.2"},

      {kernel,"ERTS  CXC 138 10","5.4"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [kernel-poll:true]\n"},

 {memory,

     [{connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,13656},

      {queue_procs,0},

      {queue_slave_procs,0},

      {plugins,93632},

      {other_proc,19700992},

      {metrics,184384},

      {mgmt_db,0},

      {mnesia,74928},

      {other_ets,1998240},

      {binary,529312},

      {msg_index,29168},

      {code,25561454},

      {atom,1074369},

      {other_system,8951577},

      {allocated_unused,12386944},

      {reserved_unallocated,0},

      {strategy,rss},

      {total,[{erlang,58211712},{rss,66506752},{allocated,70598656}]}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{mqtt,1883,"::"}]},

 {vm_memory_calculation_strategy,rss},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,839976550},

 {disk_free_limit,50000000},

 {disk_free,27441467392},

 {file_descriptors,

     [{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},

 {processes,[{limit,1048576},{used,228}]},

 {run_queue,0},

 {uptime,523},

 {kernel,{net_ticktime,60}}]

[Michael Klishin]

The exception in question is an in-flight publish method that was handled

after the connection was already closed (for what reason, I cannot tell from this snippet).

I don't see how the authN backend used has any impact here.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Rafael Bachiller]

Hi Michael,

Thanks for the reply! 

I'm using mosquitto_pub for sending data to RabbitMQ. I thought it was related with authN backend because it works correctly when I use authN internal. Subscribing with mosquito_sub works for both configurations.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rafael Bachiller]

Just in case someone finds the same problem, it is explained here: https://github.com/rabbitmq/rabbitmq-server/issues/1538

[Michael Klishin]

Should be addressed as of https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.5-beta.2,

please give it a try.

On Mon, Mar 26, 2018 at 7:25 PM, Rafael Bachiller <rafadi...@gmail.com> wrote:

Just in case someone finds the same problem, it is explained here: https://github.com/rabbitmq/rabbitmq-server/issues/1538

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.