Exception on `amqp_connection:start/2` when server is malfunctioning behind tls termination proxy

[Eugene A]

Hi All.

I'm getting an exception exit when trying to connect to malfunctioning rabbitmq behind nginx TLS termination proxy.

I would expect to receive an error tuple in this case, because other socket errors are returned as error tuples.

Is this a bug, or is exception an expected behavior?

Adding all technical information below.

Env:

$ erl -v
Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]
$ cat deps/amqp_client/git-revisions.txt
RabbitMQ AMQP Client 3.7.17

Exception:

> amqp_connection:start(#amqp_params_network{port=5673,ssl_options=[]}).
16:15:54.200 [warning] Connection (<0.163.0>): Server name indication is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
16:15:54.200 [warning] Connection (<0.163.0>): Certificate chain verification is not enabled for this TLS connection. Please see https://rabbitmq.com/ssl.html for more information.
** exception exit: {{function_clause,[{amqp_gen_connection,terminate,
                                                           [{{badmatch,{error,closed}},
                                                             [{amqp_network_connection,handshake,3,
                                                                                       [{file,"/home/user/project/deps/amqp_client/src/amqp_network_connection.erl"},
                                                                                        {line,191}]},
                                                              {amqp_network_connection,try_handshake,3,
                                                                                       [{file,"/home/user/project/deps/amqp_client/src/amqp_network_connection.erl"},
                                                                                        {line,184}]},
                                                              {amqp_gen_connection,handle_call,3,
                                                                                   [{file,"/home/user/project/deps/amqp_client/src/amqp_gen_connection.erl"},
                                                                                    {line,171}]},
                                                              {gen_server,try_handle_call,4,
                                                                          [{file,"gen_server.erl"},{line,661}]},
                                                              {gen_server,handle_msg,6,
                                                                          [{file,"gen_server.erl"},{line,690}]},
                                                              {proc_lib,init_p_do_apply,3,
                                                                        [{file,"proc_lib.erl"},{line,249}]}]},
                                                            {<0.196.0>,
                                                             #amqp_params_network{username = <<"guest">>,
                                                                                  password = <<"guest">>,virtual_host = <<"/">>,
                                                                                  host = "localhost",port = 5673,channel_max = 2047,
                                                                                  frame_max = 0,heartbeat = 10,connection_timeout = 60000,
                                                                                  ssl_options = [],
                                                                                  auth_mechanisms = [fun amqp_auth_mechanisms:plain/3,
                                                                                                     fun amqp_auth_mechanisms:amqplain/3],
                                                                                  client_properties = [],socket_options = []}}],
                                                           [{file,"/home/user/project/deps/amqp_client/src/amqp_gen_connection.erl"},
                                                            {line,239}]},
                                      {gen_server,try_terminate,3,
                                                  [{file,"gen_server.erl"},{line,673}]},
                                      {gen_server,terminate,10,
                                                  [{file,"gen_server.erl"},{line,858}]},
                                      {proc_lib,init_p_do_apply,3,
                                                [{file,"proc_lib.erl"},{line,249}]}]},
                    {gen_server,call,[<0.197.0>,connect,60000]}}
     in function  gen_server:call/3 (gen_server.erl, line 223)
16:15:54.302 [error] gen_server <0.197.0> terminated with reason: no function clause matching amqp_gen_connection:terminate({{badmatch,{error,closed}},[{amqp_network_connection,handshake,3,[{file,"/home/user/project/..."},...]},...]}, {<0.196.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5673,2047,0,10,60000,...}}) line 239
16:15:54.302 [error] CRASH REPORT Process <0.197.0> with 0 neighbours crashed with reason: no function clause matching amqp_gen_connection:terminate({{badmatch,{error,closed}},[{amqp_network_connection,handshake,3,[{file,"/home/user/project/..."},...]},...]}, {<0.196.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5673,2047,0,10,60000,...}}) line 239
16:15:54.302 [error] Supervisor {<0.195.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.196.0>, {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5673,2047,0,10,60000,[],[fun amqp_..>,...],...}) at <0.197.0> exit with reason no function clause matching amqp_gen_connection:terminate({{badmatch,{error,closed}},[{amqp_network_connection,handshake,3,[{file,"/home/user/project/..."},...]},...]}, {<0.196.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5673,2047,0,10,60000,...}}) line 239 in context child_terminated
16:15:54.303 [error] Supervisor {<0.195.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.196.0>, {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5673,2047,0,10,60000,[],[fun amqp_..>,...],...}) at <0.197.0> exit with reason reached_max_restart_intensity in context shutdown

Steps to reproduce using nginx inside a docker container:

1) Run this bash script to setup nginx container:

WORKSPACE=/tmp/amqpopenbug
mkdir -p $WORKSPACE && cd $WORKSPACE # create a workspace
openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem -days 365 -subj '/CN=localhost' # generate self-signed cert for TLS
cat > nginx.conf <<EOF
user  nginx;
worker_processes 1;
events {
    worker_connections  1024;
}
stream {
  upstream stream_backend {
    server localhost:12345; # non-existent port to simulate an error
  }

  server {
    listen                 5673 ssl;
    ssl_certificate        /etc/nginx/cert.pem;
    ssl_certificate_key    /etc/nginx/key.pem;

    proxy_pass             stream_backend;
  }
}
EOF
docker run --rm --name nginx-tcp-tls-terminator -v $WORKSPACE/nginx.conf:/etc/nginx/nginx.conf:ro -v $WORKSPACE/key.pem:/etc/nginx/key.pem:ro -v $WORKSPACE/cert.pem:/etc/nginx/cert.pem:ro -p 5673:5673 nginx

2) Run `amqp_connection:start(#amqp_params_network{port=5673,ssl_options=[]}).` in erl console


Thanks, Eugene A.

PS: was originally reported for elixir AMQP library here.

[Michael Klishin]

> {error,closed}




sounds like a reasonable error value to me. What would you expect to see besides the "connection closed" marker/atom?


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/720c8117-e7db-449d-b025-62042c3784ba%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Eugene A]

Yeah, it would be perfectly ok if {error, closed} was returned from amqp_connection:start.

The issue here is that {error, closed} is not returned - erlang function clause exception is raised instead.

I would expect to use amqp_connection:start like this:




case amqp_connection:start(#amqp_params_network{port=5673,ssl_options=[]}) of
  {ok, Conn} -> process_conn(Conn)
  {error, Reason} -> process_error(Reason)
end





But because of this issue I have to use it with something like this:





try amqp_connection:start(#amqp_params_network{port=5673,ssl_options=[]}) of
  {ok, Conn} -> process_conn(Conn)
  {error, Reason} -> process_error(Reason)
catch
  exit:Exit -> process_error(Exit)
end


Thanks, Eugene A.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/720c8117-e7db-449d-b025-62042c3784ba%40googlegroups.com.

[Michael Klishin]

You are welcome to contribute some error handling improvements. They sound reasonable

(but not all socket errors would be easy to propagate).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7f30943f-2edc-4702-9ba1-332bc20b2da9%40googlegroups.com.