RabbitMQ shovel to Azure Service Bus Queue

[Rob Bowman]

Does anyone know of sample RabbitMQ config for a shovel to send messages to an Azure Service Bus queue? I only need one-way push. I currently publish to the Azure Service queue using a Shared Access Key from a C# application that first reads from the Rabbit queue. It would be nice if I could remove this C# application and rely on the RabbitMQ shovel service to do the same.

[Luke Bakken]

Hi Rob,

I suggest searching the archive of this mailing list using the terms "azure shovel".

Please also see these docs: https://www.rabbitmq.com/shovel.html https://www.rabbitmq.com/shovel-static.html#example-config-amqp091-amqp10

Thanks,

Luke

[Jim Weier]

these example truly dont work!!! If take the example put it in the Application.config, Rabbit will fail to Start ! I can sort of get it to start if you use Dynamic rabbitmqctl set_parameter shovel ... But if you try to pass any application Variables it Pukes with badarg...

[Jim Weier]

Did you ever get this to work?

[Luke Bakken]

Hi Jim,

Please start a new discussion instead of replying to one that is old.

Rather than give brief, non-descriptive statements about things not working in your environment, you should help us help you by doing the following:

• Provide the version of RabbitMQ and Erlang you are using, as well as on which operating system
• Attach your complete RabbitMQ configuration files
• Attach your complete RabbitMQ log files
• Provide the output of rabbitmqctl report as a file attachment.
• Provide exact steps to reproduce what you are seeing.

Thanks,

Luke

[Jim Weier]

• Provide the version of RabbitMQ and Erlang you are using, as well as on which operating system

• RabbitMQ 3.83 Erlang 22.3 on Windows Server 2016
• Advanced.config attached with all "sensitive data removed"
• no log file generated on Start when fails to start
• Report attached with confidential infor removed
• Steps to reproduce... Install Erlang, Install Rabbit, Install Shovel and Shovel Admin, Put 091 Shovel source/AMQP 10 Destination into Config, Save Config, Restart rabbit
• Tried Dynamic Config by using rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://[redacgted]:[redacted]@rMYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net', 'dest-address': '[]redacted' } - This will work !!!"
• by when you add groupID abbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://[redacgted]:[redacted]@rMYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net', 'dest-address': '[]redacted', 'application_properties':[ {<<"group_id">>, <<"mygroupId_for_session">>}]} - WIL FAILL with BADARG... t
• rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://loansaveuser:aMe0WMysHubuCpbs2V4TUm3AV1NaJ1qYXsiMiO1rtTg=@rabbitmqbridge.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=rabbitmqbridge.servicebus.windows.net', 'dest-address': 'loansaverecieve', 'dest-application-properties'::[ {<<"group_id">>, <<"my_exchange">>}]} }" Fails with BADARG
  
• rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://loansaveuser:aMe0WMysHubuCpbs2V4TUm3AV1NaJ1qYXsiMiO1rtTg=@rabbitmqbridge.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=rabbitmqbridge.servicebus.windows.net', 'dest-address': 'loansaverecieve', 'dest-application-properties':'[{<<"group_id">>, <<"my_exchange">>}]}'}" FAILS with Error:{:function_clause, [{:rabbit_data_coercion, :to_proplist, ["[{<<group_id>>, <<my_exchange>>}]}"], [file: 'src/rabbit_data_coercion.erl', line: 51]}, {:rabbit_shovel_parameters, :validate_amqp10_map, 2, [file: 'src/rabbit_shovel_parameters.erl', line: 213]}, {:rabbit_parameter_validation, :"-proplist/3-fun-0-", 3, [file: 'src/rabbit_parameter_validation.erl', line: 65]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1263]}, {:rabbit_parameter_validation, :proplist, 3, [file: 'src/rabbit_parameter_validation.erl', line: 61]}, {:rabbit_shovel_parameters, :validate, 5, [file: 'src/rabbit_shovel_parameters.erl', line: 49]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 152]}, {:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}]}
• rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://loansaveuser:aMe0WMysHubuCpbs2V4TUm3AV1NaJ1qYXsiMiO1rtTg=@rabbitmqbridge.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=rabbitmqbridge.servicebus.windows.net', 'dest-address': 'loansaverecieve', 'dest-application-properties':'<<"group_id">>, <<"my_exchange">>'}" fails with Error:{:function_clause, [{:rabbit_data_coercion, :to_proplist, ["<<group_id>>, <<my_exchange>>"], [file: 'src/rabbit_data_coercion.erl', line: 51]}, {:rabbit_shovel_parameters, :validate_amqp10_map, 2, [file: 'src/rabbit_shovel_parameters.erl', line: 213]}, {:rabbit_parameter_validation, :"-proplist/3-fun-0-", 3, [file: 'src/rabbit_parameter_validation.erl', line: 65]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1263]}, {:rabbit_parameter_validation, :proplist, 3, [file: 'src/rabbit_parameter_validation.erl', line: 61]}, {:rabbit_shovel_parameters, :validate, 5, [file: 'src/rabbit_shovel_parameters.erl', line: 49]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 152]}, {:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}]}
• rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://loansaveuser:aMe0WMysHubuCpbs2V4TUm3AV1NaJ1qYXsiMiO1rtTg=@rabbitmqbridge.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=rabbitmqbridge.servicebus.windows.net', 'dest-address': 'loansaverecieve', 'dest-application-properties':'"group_id">>, <<"my_exchange">>'}" Fails with Error:{:function_clause, [{:rabbit_data_coercion, :to_proplist, ["group_id>>, <<my_exchange>>"], [file: 'src/rabbit_data_coercion.erl', line: 51]}, {:rabbit_shovel_parameters, :validate_amqp10_map, 2, [file: 'src/rabbit_shovel_parameters.erl', line: 213]}, {:rabbit_parameter_validation, :"-proplist/3-fun-0-", 3, [file: 'src/rabbit_parameter_validation.erl', line: 65]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1263]}, {:rabbit_parameter_validation, :proplist, 3, [file: 'src/rabbit_parameter_validation.erl', line: 61]}, {:rabbit_shovel_parameters, :validate, 5, [file: 'src/rabbit_shovel_parameters.erl', line: 49]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 152]}, {:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}]}

[Luke Bakken]

Hi Jim,

Thanks for all of that information.

I'm assuming that you are using cmd.exe to run these rabbitmqctl.bat commands. Unfortunately, the combination of quoted strings, JSON data, and cmd.exe is guaranteed to cause headaches. Using the echoargs.exe program shows why:

C:\Users\lbakken>echoargs rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://[redacgted]:[redacted]@rMYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net', 'dest-address': '[]redacted', 'application_properties':[ {<<"group_id">>, <<"mygroupId_for_session">>}]}"

Arg 0 is <rabbitmqctl>

Arg 1 is <set_parameter>

Arg 2 is <shovel>

Arg 3 is <RabbitMQAzureBridge>

Arg 4 is <{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://[redacgted]:[redacted]@rMYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net', 'dest-address': '[]redacted', 'application_properties':[ {<<group_id>>, <<mygroupId_for_session>>}]}>

Command line:

"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" rabbitmqctl set_parameter shovel RabbitMQAzureBridge "{'src-protocol': 'amqp091', 'src-uri': 'amqp://', 'src-queue': 'stubQueue', 'dest-protocol': 'amqp10', 'dest-uri': 'amqps://[redacgted]:[redacted]@rMYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net', 'dest-address': '[]redacted', 'application_properties':[ {<<"group_id">>, <<"mygroupId_for_session">>}]}"

Notice that the double-quotes within the double-quoted string are stripped away from the group_id application properties. The following command line will pass the double quotes correctly by doubling the double quotes:

rabbitmqctl.bat set_parameter shovel RabbitMQAzureBridge "{'src-protocol':'amqp091','src-uri':'amqp://','src-queue':'stubQueue','dest-protocol':'amqp10','dest-uri':'amqps://[REDACTED]:[REDACTED]@MYNAMESPACE.servicebus.windows.net:5671/?auth_mechanism=plain&auth_mechanism=amqplain&verify=verify_none&server_name_indication=MYNAMESPACE.servicebus.windows.net','dest-address':'REDACTED','application_properties':[{<<""group_id"">>,<<""mygroupId_for_session"">>}]}"

I don't have access to Azure or an AMQP 1.0 destination to test this, so let me know how it works in your environment.

Thanks

Luke

[Jim Weier]

No this does not work, I get a JSON decoding error. Reason: badarg . I am not positive how the Parser works... But I can tell you the group id will contain /  something like /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/NameSpace

If possible, Id would rather do this in the Advanced.config, But cant get that to work either

[Luke Bakken]

Hi Jim,

So that I'm not guessing as to exactly what commands you're running and their output, I need you to provide transcripts of what you're doing. Please do the following:

* Install echoargs.exe (I use chocolatey), and use it in the same manner I did to verify that the command you're running is parsed correctly. Capture all output (as I did below) into a file and attach the file to your response.

* Run the command without using echoargs.exe. Same as before - capture the command and the output into a file.

* Please attach the advanced.config file you're trying to use.

I am uncertain if the JSON error is coming from how the rabbitmqctl command is being run, or from when the shovel is being set up. I'm guessing it is from how the command is being run which is why I need step-by-step verification.

Thanks,

Luke