Problem shovelling messages between Azure Service Bus and RabbitMQ

[CT]

Hi. I'm trying to use the Shovel plugin to shovel messages between RabbitMQ and the Azure Service Bus (as suggested here: https://groups.google.com/forum/#!topic/rabbitmq-users/MzhRt_L2BFo), but I've not managed to get this working. I'm fairly new to both RabbitMQ and the Azure Service Bus, so I suspect I'm missing something obvious but no idea what.

I'm connecting to Azure as AMQP 1.0, and RabbitMQ as 0.9.1. The Shovel is installed on the same RabbitMQ instance, and was set up using the shovel_management plugin. 

 

amqp10

amqp://test1:[redacted]@zzzzzzzzzzzz.servicebus.windows.net:5671

testqueue2

5

amqp091

amqp://

azureexchange

○

1s

on-confirm

never

As soon as the shovel is started, it very quickly goes to a terminated state with the message: 

{normal,{gen_fsm,sync_send_all_state_event,[<0.26205.0>,begin_session]}}

Other shovels between RabbitMQ instances over 0.9.1 work ok, so I'm guessing not the shovel plugin itself at fault.

I can also connect to the Service Bus using a different AMQP client (https://github.com/Azure/amqpnetlite) using the same credentials, so not a firewall problem etc.

Does anyone have any ideas as to what's happening?

[Luke Bakken]

Hello,

Please let us know what version of Erlang and RabbitMQ you are using. It would be helpful to get a complete set of RabbitMQ logs (and crash log, if there is one) rather than a single message.

Thanks,

Luke

[CT]

Thanks very much for getting back to me. 

It's a completely fresh version of both Erlang and RabbitMQ, so 20.3 and 3.7.5 respectively, and I'm running it within an Azure VM (Windows Server 2016 Datacentre) but seeing the same error on premise as well (Windows Server 2012).

I've attached the relevant logs.

[Karl Nilsson]

Hi,

It looks like the remote server is closing your connection. The reason for this is very likely to be incorrect configuration or authentication.

The first thing I'd suggest you try is to append &sasl=true to your uri. I don't think it will use sasl implicitly without this.

Depending on which type of Azure SB queue you are interacting with you may also need to set the 'x-opt-partition-key' message header (if using partitions) and the `group_id` property if using sessions. [1]

I'm not sure you can set message_annotations and properties through the shovel management UI - you may need to use the HTTP api directly [2]

1. https://groups.google.com/forum/#!msg/rabbitmq-users/vUKM8MIsmEM/XL3VYZZoBAAJ

2. https://www.rabbitmq.com/shovel-dynamic.html

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[CT]

Hi,

Thanks for the reply - it gave me a bunch of things to investigate, but no luck yet. 

I'm now working on a queue without partitions or sessions to try and rule that out. And realised I should have been using amqps:// rather than amqp://

After some TLS warnings within the log files, I've now created a rabbitmq.conf file containing

ssl_options.verify = verify_none

ssl_options.fail_if_no_peer_cert = false

to try and bypass any certificate issues.

Adding the &sasl=true gives an error of

{{badmatch,{error,{no_default_port,amqps,"amqps://test1:y...@zzz.servicebus.windows.net/&sasl=true:5672"}}} 

(shovel.erl line 105) 

In the log files, error suggests:- "no match of right hand value" - have I added the sasl part correctly? '&' or '?' gives the same response.

As far as I can tell from other generic amqp documentation I need something like the following connection string? 

amqps://test1:[redact]@zzz.servicebus.windows.net:5671

but that gives the error

{shutdown,
    {gen_fsm,sync_send_event,
        [<0.608.0>,
         {attach,
             #{name => <<"test15_eb7ce1240d49f69d_-receiver">>,
               rcv_settle_mode => first,
               role =>
                   {receiver,
                       #{address => <<"testqueue3">>,
                         durable => unsettled_state},
                       <0.464.0>},
               snd_settle_mode => unsettled}}]}}

with nothing further in rabbit.log, and crash.log is empty.

I think I've run through every combination of protocol, port and parameters, but I can't find the magic sequence. If anyone has an existing config which they'd be happy to share which makes this work then I'd be very grateful! Or any other ideas of what I'm missing? 

Thanks!

[Philip Patrick]

I am too struggling with shoveling from on-prem to Azure Service Bus and also unsuccessful (although I am trying to achieve that with dynamic shovels), but in this case, the URL looks phishy - the port should be after the host name, also the question mark is missed. Any chance you have mistyped it?

Current: amqps://test1:yyy@zzz.servicebus.windows.net/&sasl=true:5672

Should be: amqps://test1:yyy@zzz.servicebus.windows.net:5672?&sasl=true

Philip

On Tuesday, May 22, 2018 at 5:58:53 PM UTC+3, CT wrote:

Hi,

Thanks for the reply - it gave me a bunch of things to investigate, but no luck yet. 

I'm now working on a queue without partitions or sessions to try and rule that out. And realised I should have been using amqps:// rather than amqp://

After some TLS warnings within the log files, I've now created a rabbitmq.conf file containing

ssl_options.verify = verify_none

ssl_options.fail_if_no_peer_cert = false

to try and bypass any certificate issues.

Adding the &sasl=true gives an error of

{{badmatch,{error,{no_default_port,amqps,"amqps://test1:yyy@zzz.servicebus.windows.net/&sasl=true:5672"}}} 

[CT]

Oops - yes, that would have helped, thanks very much for pointing that one out! 

Have changed that now, and running that on port 5671 gives the same "unsettled_state" error as without that parameter. (I think 5671 is right, but not totally sure). 

On 5672, I still get a similar error though.

2018-05-22 15:34:47.802 [error] <0.474.0> CRASH REPORT Process <0.474.0> with 0 neighbours exited with reason: no match of right hand value {error,{shutdown,{failed_to_start_child,reader,{{badmatch,{error,closed}},[{amqp10_client_frame_reader,init,1,[{file,"src/amqp10_client_frame_reader.erl"},{line,120}]},{gen_fsm,init_it,6,[{file,"gen_fsm.erl"},{line,348}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}}} in rabbit_amqp10_shovel:connect/7 line 106 in gen_server2:terminate/3 line 1166

Are you seeing similar errors with a dynamic shovel, or getting anything different?

[Michael Klishin]

5671 is AMQP with TLS. The error basically says that a connection could not initialize

because a socket was unexpectedly closed. Which would happen if TLS is not actually enabled in the client.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[CT]

Thanks very much, that confirms what I thought. 

So when you say TLS not enabled in the client, what do you mean by the client? Is that the RabbitMQ shovel itself? Or the service bus? If the shovel, how do I enable it?

I can use the same address as I'm trying to use in the shovel: amqps://test1:[redacted]@[,,,].servicebus.windows.net:5671  - with a .net AMQP client (using the amqpnetlite library here: https://github.com/Azure/amqpnetlite), and am able to connect to the service bus successfully. So suggests TLS is operating on the service bus? 

[Michael Klishin]

I mean a RabbitMQ client library/app, including Shovels (that use the Erlang client under the hood).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Shovels must be configured to use TLS just like any other clients.

This is done using a standard set of TLS options that should be no different

for AMQP 1.0 Shovels (and the AMQP 1.0 Erlang client they use):

http://www.rabbitmq.com/shovel.html#tls-connections

Curiously I cannot find any traces of TLS support with a quick search in the AMQP 1.0 Shovel module.

I may be that TLS was overlooked.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

The underlying client does support TLS options, though:

https://github.com/rabbitmq/rabbitmq-amqp1.0-client/blob/83082f3f6691058b2c4da6f068d2d9fc890e8d27/README.md#L32

So please check whether your URIs provide client certificate and key file paths. If they don't then the Shovel cannot know that you intend

to use TLS.

On Wed, May 23, 2018 at 11:34 AM, Michael Klishin <mkli...@pivotal.io> wrote:

Shovels must be configured to use TLS just like any other clients.

This is done using a standard set of TLS options that should be no different

for AMQP 1.0 Shovels (and the AMQP 1.0 Erlang client they use):

http://www.rabbitmq.com/shovel.html#tls-connections

Curiously I cannot find any traces of TLS support with a quick search in the AMQP 1.0 Shovel module.

I may be that TLS was overlooked.

[Michael Klishin]

The current Shovel implementation does support TLS for AMQP 0-9-1 and as far as I can tell, AMQP 1.0.

There shouldn't be anything specific to AMQP 1.0 in the URI part. The docs are at http://www.rabbitmq.com/uri-query-parameters.html.

On Wed, May 23, 2018 at 12:44 PM, 'CT' via rabbitmq-users <rabbitm...@googlegroups.com> wrote:

Thanks. The URIs I've used previously definitely don't include client certs or key file paths. I've never seen anything around using amqp to connect to an Azure Service Bus that includes these details, so I don't know what I'd put in here.

But more generally, if the current Shovel implementation doesn't support TLS, does that also mean that I can't use one to connect to an Azure Service Bus - even if I can find out the client cert details?

On Wednesday, 23 May 2018 11:36:36 UTC+1, Michael Klishin wrote:

The underlying client does support TLS options, though:

https://github.com/rabbitmq/rabbitmq-amqp1.0-client/blob/83082f3f6691058b2c4da6f068d2d9fc890e8d27/README.md#L32

So please check whether your URIs provide client certificate and key file paths. If they don't then the Shovel cannot know that you intend

to use TLS.

On Wed, May 23, 2018 at 11:34 AM, Michael Klishin <mkli...@pivotal.io> wrote:

Shovels must be configured to use TLS just like any other clients.

This is done using a standard set of TLS options that should be no different

for AMQP 1.0 Shovels (and the AMQP 1.0 Erlang client they use):

http://www.rabbitmq.com/shovel.html#tls-connections

Curiously I cannot find any traces of TLS support with a quick search in the AMQP 1.0 Shovel module.

I may be that TLS was overlooked.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roy Salisbury]

Has anyone got this working yet?  I have searched everywhere, and I can't find any reference to a success. 

[cocowalla]

I'm someone else who has spent a lot of time trying... and failed! My aim was to get a bidirection shovel working, from an on-prem RabbitMQ instance to both Azure Service Bus and Event Hub, and vice versa. I must have tried every possible permutation of both source and destination URIs, but never got it to connect.

If anyone else ever does get this working, I'd love to know how!.

[Michael Klishin]

We plan on adding Azure-specific tests to CI but currently there are more pressing issues. Some known non-standard assumptions of Azure

from AMQP 1.0 clients should be documented. The client part was tested against at least 3 implementations, including Azure.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roy J. Salisbury]

I had planned to transfer (shovel) from my on premise RabbitMQ to the azure service bus so I could use an Azure function to process the data. My next plan was to possibly use an Azure function directly against the cloud version of RabbitMQ (its hosted in Azure) .. again, no luck.  

On Fri, Jun 8, 2018 at 1:00 PM, Michael Klishin <mkli...@pivotal.io> wrote:

We plan on adding Azure-specific tests to CI but currently there are more pressing issues. Some known non-standard assumptions of Azure

from AMQP 1.0 clients should be documented. The client part was tested against at least 3 implementations, including Azure.

On Fri, Jun 8, 2018 at 10:20 PM, cocowalla <colin.an...@gmail.com> wrote:

I'm someone else who has spent a lot of time trying... and failed! My aim was to get a bidirection shovel working, from an on-prem RabbitMQ instance to both Azure Service Bus and Event Hub, and vice versa. I must have tried every possible permutation of both source and destination URIs, but never got it to connect.

If anyone else ever does get this working, I'd love to know how!.

On Friday, 8 June 2018 18:09:24 UTC+1, Roy Salisbury wrote:

Has anyone got this working yet?  I have searched everywhere, and I can't find any reference to a success. 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/GRMKkXEq80o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

[Michael Klishin]

I'm afraid we cannot suggest much if our input information is "no luck".

[cocowalla]

I get this is not going to be a priority for Pivotal - I'm more hoping for another user to figure it out and be gracious enough to share some info :)

On Friday, 8 June 2018 21:00:23 UTC+1, Michael Klishin wrote:

We plan on adding Azure-specific tests to CI but currently there are more pressing issues. Some known non-standard assumptions of Azure

from AMQP 1.0 clients should be documented. The client part was tested against at least 3 implementations, including Azure.

On Fri, Jun 8, 2018 at 10:20 PM, cocowalla <colin.an...@gmail.com> wrote:

I'm someone else who has spent a lot of time trying... and failed! My aim was to get a bidirection shovel working, from an on-prem RabbitMQ instance to both Azure Service Bus and Event Hub, and vice versa. I must have tried every possible permutation of both source and destination URIs, but never got it to connect.

If anyone else ever does get this working, I'd love to know how!.

On Friday, 8 June 2018 18:09:24 UTC+1, Roy Salisbury wrote:

Has anyone got this working yet?  I have searched everywhere, and I can't find any reference to a success. 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[CT]

I wasn't able to get this working either, but a big thanks to Michael anyway for his input and the various suggestions. I was unable to find the necessary logs in Azure (not even sure if they exist) to be able to investigate the connection at that end, so ran out of things to test.

I've worked around it temporarily by hacking together my own .net-based shovel using the official Azure Service Bus and RabbitMQ components respectively. But I think a shovel built by Pivotal and integrated into RabbitMQ is going to be far more robust so I'm also hopeful that someone else with more knowledge and experience of both ends can get this going!

[Roy J. Salisbury]

This is exactly what I have done. Its running locally on the raspberry pi that RabbitMQ is on, and works pretty good. But I agree, one built in would be far more robust.

[Michael Klishin]

We definitely want cross-protocol Shovel to work well, Azure or not. We may get to adding more tests in the next few weeks.

On Sat, Jun 9, 2018 at 12:24 AM, Roy J. Salisbury <ro...@express-is.net> wrote:

This is exactly what I have done. Its running locally on the raspberry pi that RabbitMQ is on, and works pretty good. But I agree, one built in would be far more robust.

On Fri, Jun 8, 2018 at 2:07 PM, 'CT' via rabbitmq-users <rabbitmq-users@googlegroups.com> wrote:

I wasn't able to get this working either, but a big thanks to Michael anyway for his input and the various suggestions. I was unable to find the necessary logs in Azure (not even sure if they exist) to be able to investigate the connection at that end, so ran out of things to test.

I've worked around it temporarily by hacking together my own .net-based shovel using the official Azure Service Bus and RabbitMQ components respectively. But I think a shovel built by Pivotal and integrated into RabbitMQ is going to be far more robust so I'm also hopeful that someone else with more knowledge and experience of both ends can get this going!

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[V Z]

Michael, did anything transpire from your tests? Could you please share against which AMQP 1.0 implementations Rabbit's shovel plug-in was certified. Is IBM MQ one of  them?

[Karl Nilsson]

We tested the erlang AMQP 1.0 client against MQ Lite, not sure if they share the AMQP 1.0 core with IBM MQ or not (I suspect both are based on Apache QPid Proton).

The main problem is typically knowing which headers, properties, message annotation etc need to be set for any given message system. Traffic captures using other clients is how I've been working things out so far. The erlang AMQP 1.0 client is fairly low level which is also the API the shovel provides.

What I am missing from these discussions is exactly what is not working. Are the shovels configured never able to connect or does it connect and fail later on (at message publish).

On Thu, 9 Aug 2018 at 05:01 V Z <uvzu...@gmail.com> wrote:

Michael, did anything transpire from your tests? Could you please share against which AMQP 1.0 implementations Rabbit's shovel plug-in was certified. Is IBM MQ one of  them?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Karl Nilsson]

Ok having looked at this a bit more there are a couple of things that are likely to need to be included in the amqps uri when connecting to Azure ServiceBus.

You will need a URI that looks a bit like this amqps://SBUSER:SBPASS@SBENDPOINT:4561?sasl=plain&hostname=SBHOSTNAME - you may also need to specify the TLS version(s) to use for your connection as I think Azure only supports tls1.1 and above. Unfortunately this is not currently possible through the URI so would have to be done for the entire RabbitMQ erlang ssl application environment through the protocol_version configuration key. See: https://www.rabbitmq.com/ssl.html#disabling-tls-versions

Once you are connected to Azure SB you also need to configure the group_id property with the appropriate group if you use sessions and the x-opt-partition-key message annotation if you use partitions.

For a static shovel AMQP 1.0 destination it would look something like:

{properties, [{group_id, <<"the-group-id">>}]}

{message_annotations, [{"x-opt-partition-key", 12345}]}

Cheers

Karl

[Michael Klishin]

FTR, this PR [1] makes some of what Karl's described less painful.

1. https://github.com/rabbitmq/rabbitmq-amqp1.0-client/pull/20

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.