RMQ 3.9.x Stream Connection

[Evan No]

Hi there,

I've been playing around with the new stream feature in 3.9.x

Locally, I can connect to the new stream port perfectly fine with the standard guest user.  But in a cluster I upgraded to 3.9.4, I'm unable to connect.

I've tried this with both the go client, and java client and am getting the same results.  And since it works on a local docker/rabbit cluster, I'm guessing there's something about my server configs that I'm overlooking.

The client logs are..

```

2021/08/18 13:29:35 [warn] - error removing heartbeat: Response #{id} not found 2021/08/18 13:29:45 [error] - timeout 10000000000 ms - waiting Code, operation: commandSaslAuthenticate

2021/08/18 13:29:45 timeout 10000000000 ms - waiting Code, operation: commandSaslAuthenticate
```

While I believe the relevant log lines from the server when attempting connection are..

```

2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> crasher: 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> initial call: rabbit_stream_reader:init/1 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> pid: <0.31492.2550> 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> registered_name: [] 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> exception error: bad argument 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> in function list_to_binary/1 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> called as list_to_binary({error,einval}) 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> *** argument 1: not an iolist term 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> in call from rabbit_stream_reader:handle_frame_pre_auth/4 (src/rabbit_stream_reader.erl, line 1204) 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> in call from lists:foldl/3 (lists.erl, line 1267) 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> in call from rabbit_stream_reader:handle_info/3 (src/rabbit_stream_reader.erl, line 426) 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 1194) 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> ancestors: [<0.23231.2550>,<0.752.0>,<0.751.0>,<0.750.0>,<0.748.0>, 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> <0.747.0>,rabbit_stream_sup,<0.743.0>] 2021-08-18 09:25:29.836551-06:00 [error] <0.31492.2550> message_queue_len: 0    
```

The `commandSaslAuthenticate` made me think it was a user/pass issue in the provided URI, but I'm able to use the same credentials on the standard AMQP port perfectly fine.

One difference, is that the non local cluster, uses the LDAP plugin.  I've tried disabling that temporarily to no avail.

This is via a non-TLS connection at the moment.  I can also confirm the port is being listened to through netstat (I just have it set to standard 5552).

If anyone has any ideas I'd be very appreciate of any help!

[Gabriele Santomaggio]

Thank you for reporting. 
We will check it.

-

Gabriele

[Gabriele Santomaggio]

In the meantime:
- Can the client reach all the hostnames on the cluster?

- Do you have a load balancer? 

Please read this blog post about stream connection:

https://blog.rabbitmq.com/posts/2021/07/connecting-to-streams/ 

Also, this one can be useful:

https://rabbitmq.com/stream.html#advertised-host-port

-

Gabriele

[Evan No]

Hey, thanks for the literature.  I'll check those out. Answers to your questions below.

Can the client reach all the hostnames on the cluster? - Yes, however, I should note for very basic PoC I've been running the go client binary directly on one of the nodes in the cluster.  So the URI I'm using looks something like `rabbitmq-stream://<user>:<guest>@127.0.0.1:5552`

Do you have a load balancer? We use haproxy for typical incoming connections, but as I mentioned in the answer to the last question, I'm currently just attempting to run the binary directly from one of the nodes.

As far as the advertised host link goes we have the following set (obfuscating node name).

stream.listeners.tcp = 5552

stream.advertised_host = <node_name>

stream.advertised_port = 5552

It certainly isn't my plan to run this process directly from a node in the cluster long term, but I figured it would suffice to achieve the proof of concept I'm going towards.