RabbitMQ and Azure

[Espen Kleivane]

Hi, i need a bit help here.

I have done this tutorial 3-4 times and no success

How to integrate Service Bus with RabbitMQ - Azure Service Bus | Microsoft Learn

Steps:

Version:
RabbitMQ 3.10.7, Erlang 25.0

Adding a new Azure Service Bus Namespace
* rabbitmq01
* basic
* public endpoint

Creating our Azure Service Bus Queue
* from-rabbitmq01

Enabling the RabbitMQ Shovel Plugin
* rabbitmq-plugins enable rabbitmq_shovel_management

The following plugins have been enabled:
* rabbitmq_shovel
* rabbitmq_shovel_management

```log
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0> Server startup complete; 6 plugins started.
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_shovel_management
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_shovel
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_amqp1_0
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_management
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_web_dispatch
2023-07-08 18:49:58.048000+02:00 [info] <0.576.0>  * rabbitmq_management_agent

```

Created queue in RabbitMQ:
* telemetry01

Connecting RabbitMQ to Azure Service Bus
* Add SAS Policy, Manage now, tested with just send also.
* rabbitmq-shovel01
* Primary Connection String

Connection String to AMQP
* https://red-mushroom-0f7446a0f.azurestaticapps.net/

TNC
* Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True
* Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 443 = True

Nothing fancy here, straight forward and tnc is success.

rabbitmq.conf only has

amqp1_0.default_user  = guest
amqp1_0.default_vhost = /
amqp1_0.protocol_strict_mode = false

Since I enabled rabbitmq_amqp1_0.

Shovel was done exactly like the tutorial

Log: (TEXT-HERE-TEXT-HERE since removed crypt string)

```log

2023-07-08 19:08:56.406000+02:00 [info] <0.578.0> Server startup complete; 6 plugins started.
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_shovel_management
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_shovel
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_amqp1_0
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_management
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_web_dispatch
2023-07-08 19:08:56.406000+02:00 [info] <0.578.0>  * rabbitmq_management_agent
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>     supervisor: {<0.742.0>,amqp10_client_connection_sup}
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>     errorContext: start_error
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>     reason: {badmatch,{error,closed}}
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>     offender: [{pid,undefined},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {id,reader},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {mfargs,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                    {amqp10_client_frame_reader,start_link,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                        [<0.742.0>,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                         #{address => "rabbitmq01.servicebus.windows.net",
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                           hostname => <<"rabbitmq01.servicebus.windows.net">>,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                           notify => <0.688.0>,notify_when_closed => <0.688.0>,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                           notify_when_opened => <0.688.0>,port => 5671,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                           sasl =>
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                               {plain,<<"rabbitmq-shovel01">>,
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                                   <<"TEXT-HERE-TEXT-HERE">>},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                           tls_opts => {secure_port,[]}}]}},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {restart_type,transient},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {significant,false},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {shutdown,5000},
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>                {child_type,worker}]
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0>
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** State machine <0.744.0> terminating
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** When server state  = {undefined,undefined}
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Reason for termination = error:{badmatch,{error,closed}}
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Callback modules = [amqp10_client_frame_reader]
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Callback mode = undefined
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Stacktrace =
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> **  [{amqp10_client_frame_reader,init,1,
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0>                                  [{file,"amqp10_client_frame_reader.erl"},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0>                                   {line,109}]},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0>      {gen_statem,init_it,6,[{file,"gen_statem.erl"},{line,1001}]},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0>
2023-07-08 19:09:56.386000+02:00 [error] <0.688.0> Shovel 'rmq-2-sb' could not connect to destination
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>     supervisor: {<0.687.0>,rabbit_shovel_dyn_worker_sup}
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>     errorContext: child_terminated
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>     reason: shutdown
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>     offender: [{pid,<0.688.0>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                {id,{<<"/">>,<<"rmq-2-sb">>}},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                {mfargs,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                    {rabbit_shovel_worker,start_link,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                        [dynamic,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                         {<<"/">>,<<"rmq-2-sb">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                         [{<<"dest-uri">>,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                           [{encrypted,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                                <<"TEXT-HERE-TEXT-HERE">>}]},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"src-uri">>,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                           [{encrypted,
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                                <<"TEXT-HERE-TEXT-HERE">>}]},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"ack-mode">>,<<"on-confirm">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"dest-add-forward-headers">>,false},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"dest-address">>,<<"from-rabbitmq01">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"dest-protocol">>,<<"amqp10">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"reconnect-delay">>,30},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"src-delete-after">>,<<"never">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"src-protocol">>,<<"amqp091">>},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                          {<<"src-queue">>,<<"telemetry01">>}]]}},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                {restart_type,{permanent,30}},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                {shutdown,4294967295},
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0>                {child_type,worker}]
2023-07-08 19:09:56.452000+02:00 [warning] <0.818.0> Description: "Authenticity is not established by certificate path validation"
2023-07-08 19:09:56.452000+02:00 [warning] <0.818.0>      Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-07-08 19:09:56.452000+02:00 [warning] <0.818.0>



```

Thanks in advanced

[Espen Kleivane]

Importing Data from RabbitMQ into Azure Data Explorer via Event Hubs

https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688

I just wanted to make a RabbitMQ shovel to Azure Event Hub (or Azure Service bus there is a tutorial for that also on MS)

Both tutorials use the same foundation, SAS policy, URI convert etc.

Findings:

Posted on SO, and made a bug incident in MS docs also (Did some test for MS and waiting for the answer from them)

https://stackoverflow.com/questions/76652187/rabbitmq-shovel-to-azure-event-hub-or-service-bus/76690922#76690922

My last 72 hours.....

Almost endless troubleshooting, but:

Downloaded cert DigiCert Global Root CA from

https://learn.microsoft.com/en-us/azure/security/fundamentals/azure-ca-details?tabs=root-and-subordinate-cas-list

Installed RabbitMQ 3.12.1 Erlang 26.0 and moved from policy send to using new policy: Add SAS Policy (Manage) name: policymanage

Remapped the example connection string 

Endpoint=sb://<your-namespace>.servicebus.windows.net/;SharedAccessKeyName=eh-rmq-bridge;SharedAccessKey=<SharedAccessKey>;EntityPath=event-hubs-kusto-eh

after viewing https://www.erlang.org/blog/otp-26-highlights/

[...]

Erlang/OTP 26 Highlights May 16, 2023 [...] Erlang/OTP 26 Highlights In OTP 26, the default value for the verify option is now verify_peer instead of verify_none. 

Host verification requires trusted CA certificates to be supplied using one of the options cacerts or cacertsfile.

to (added TLS https://www.rabbitmq.com/uri-query-parameters.html )

amqps://policymanage:SAS-...@XXXXXX.servicebus.windows.net?cacertfile=c:/RabbitMQBaseFolder/cert/DigiCertGlobalRootCA.pem&verify=verify_none 

Result:2023-07-15 14:06:44.420000+02:00 [error] <0.4342.0> Shovel 'test_04' in virtual host '/' is stopping, reason: {outbound_link_detached, [..]                                                         

<<"amqp:unauthorized-access">>}

So this must be on Event Hub side and the unauthorized-access must be fixed for this to work.

Med vennlig hilsen / Kind regards

 Espen k

 mob: +47 41376635

                     

                   

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/eEgCbk_UHWE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8f473370-8ef6-4f25-8c2c-301c021b7361n%40googlegroups.com.

[Espen Kleivane]

It works now, yes!!!

test/it_works.jpg at main · spawnmarvel/test · GitHub