Pushing messages to the RabbitMQ instance over HTTPS
838 views
Skip to first unread message
Wojtek Gawroński
Mar 2, 2015, 9:17:11 AM3/2/15
to rabbitm...@googlegroups.com
Hello,
For our current project, we are evaluating several options for setting up queuing architecture - one of ideas incorporates RabbitMQ. But we have got one significant, let's call it a "requirement" ;) - we have to push messages with use of HTTPS protocol. I know that it sounds hilariously at first, but it is required by integration used inside project - we are using Salesforce.com, which is a main component and produces events consumed by our back-end infrastructure. We'd like to avoid introducing another small service (like this one https://github.com/smallfish/rabbitmq-http) which will receive HTTPS call from Salesforce and transport it to the broker with use of AMQP (because it is another component on path which can fail).
For these people that don't know - unfortunately, you can't invoke any outbound call from Salesforce.com platform via protocol different that HTTP or HTTPS.
We have evaluated several plug-ins for RabbitMQ (and obviously using management API is not an option - as it is stated in docs):
- https://github.com/pbarford/rabbitmq-rest-plugin
- https://github.com/ddossot/rabbitmq-http-safe
- https://github.com/jbrisbin/rabbitmq-webhooks
We don't want to host broker by ourselves and in order to do that, we need to use SaaS provider - private instance of CloudAMQP in our case, which is using the newest versions of both - RabbitMQ and Erlang (and that's good, no one likes the maintenance of older version from the operational point of view). Unluckily - none of these works with newest versions (e.g. build process is broken for the Erlang 17.x or some internals of RabbitMQ changed and plug-in can't be compiled).
Have someone used one of these plug-ins successfully before (even for the earlier versions)?
Or maybe we missed some valuable plug-in here - please let me know.
Best regards,
Wojtek Gawroński
For our current project, we are evaluating several options for setting up queuing architecture - one of ideas incorporates RabbitMQ. But we have got one significant, let's call it a "requirement" ;) - we have to push messages with use of HTTPS protocol. I know that it sounds hilariously at first, but it is required by integration used inside project - we are using Salesforce.com, which is a main component and produces events consumed by our back-end infrastructure. We'd like to avoid introducing another small service (like this one https://github.com/smallfish/rabbitmq-http) which will receive HTTPS call from Salesforce and transport it to the broker with use of AMQP (because it is another component on path which can fail).
For these people that don't know - unfortunately, you can't invoke any outbound call from Salesforce.com platform via protocol different that HTTP or HTTPS.
We have evaluated several plug-ins for RabbitMQ (and obviously using management API is not an option - as it is stated in docs):
- https://github.com/pbarford/rabbitmq-rest-plugin
- https://github.com/ddossot/rabbitmq-http-safe
- https://github.com/jbrisbin/rabbitmq-webhooks
We don't want to host broker by ourselves and in order to do that, we need to use SaaS provider - private instance of CloudAMQP in our case, which is using the newest versions of both - RabbitMQ and Erlang (and that's good, no one likes the maintenance of older version from the operational point of view). Unluckily - none of these works with newest versions (e.g. build process is broken for the Erlang 17.x or some internals of RabbitMQ changed and plug-in can't be compiled).
Have someone used one of these plug-ins successfully before (even for the earlier versions)?
Or maybe we missed some valuable plug-in here - please let me know.
Have someone worked with such scenario and integrated RabbitMQ with Salesforce or just over HTTPS?
I would love to hear how did you do that.
Another option could be to create a small plug-in for RabbitMQ (pushing messages via HTTPS endpoint with simplest possible token-based security). I saw this article - http://videlalvaro.github.io/2013/06/writing-rabbitmq-plugins-with-elixir.html - and I have to admit that all my knowledge about RabbitMQ architecture comes only from Alvaro's blog and it is very likely that I missed something non-obvious at first sight.
How hard it will be to develop such plugin, assuming that someone knows Erlang/Elixir?
I would love to hear how did you do that.
Another option could be to create a small plug-in for RabbitMQ (pushing messages via HTTPS endpoint with simplest possible token-based security). I saw this article - http://videlalvaro.github.io/2013/06/writing-rabbitmq-plugins-with-elixir.html - and I have to admit that all my knowledge about RabbitMQ architecture comes only from Alvaro's blog and it is very likely that I missed something non-obvious at first sight.
How hard it will be to develop such plugin, assuming that someone knows Erlang/Elixir?
Best regards,
Wojtek Gawroński
Michael Klishin
Mar 2, 2015, 9:24:17 AM3/2/15
to rabbitm...@googlegroups.com, Wojtek Gawroński
On 2 March 2015 at 17:17:12, Wojtek Gawroński (afro...@gmail.com) wrote:
> Unluckily - none of these works with newest versions (e.g. build
> process is broken for the Erlang 17.x or some internals of RabbitMQ
> changed and plug-in can't be compiled).
>
> Have someone used one of these plug-ins successfully before
> (even for the earlier versions)?
> Or maybe we missed some valuable plug-in here - please let me know.
>
> Have someone worked with such scenario and integrated RabbitMQ
> with Salesforce or just over HTTPS?
You should be able to publish over HTTP using the HTTP API:
> Unluckily - none of these works with newest versions (e.g. build
> process is broken for the Erlang 17.x or some internals of RabbitMQ
> changed and plug-in can't be compiled).
>
> Have someone used one of these plug-ins successfully before
> (even for the earlier versions)?
> Or maybe we missed some valuable plug-in here - please let me know.
>
> Have someone worked with such scenario and integrated RabbitMQ
> with Salesforce or just over HTTPS?
http://www.rabbitmq.com/management.html
https://raw.githack.com/rabbitmq/rabbitmq-management/rabbitmq_v3_4_4/priv/www/api/index.html
It won't be as efficient as using any other protocol but you can do it. I'm not sure if
RabbitMQ HTTP API on CloudAMQP offers TLS but if you are eyeing a private instance,
they can help you with that.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Alvaro Videla
Mar 2, 2015, 9:27:18 AM3/2/15
to Wojtek Gawroński, rabbitm...@googlegroups.com
Hi,
There's a plugin development guide here: https://www.rabbitmq.com/plugin-development.html otherwise take a look at the last chapter of RabbitMQ in Action (it might be slightly outdated, but general principles stayed the same).
Out of curiosity, you say you want to host RabbitMQ with CloudAMQP, do they support custom plugins written by users?
I know you don't want to run just another service, but perhaps it would end up being easier to just have your own little service listening on HTTPS from Salesforce, and then transforming and submitting those messages via AMQP?
There's a plugin development guide here: https://www.rabbitmq.com/plugin-development.html otherwise take a look at the last chapter of RabbitMQ in Action (it might be slightly outdated, but general principles stayed the same).
Out of curiosity, you say you want to host RabbitMQ with CloudAMQP, do they support custom plugins written by users?
I know you don't want to run just another service, but perhaps it would end up being easier to just have your own little service listening on HTTPS from Salesforce, and then transforming and submitting those messages via AMQP?
Regards,
Alvaro
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Wojtek Gawroński
Mar 2, 2015, 9:43:34 AM3/2/15
to Alvaro Videla, mkli...@pivotal.io, rabbitm...@googlegroups.com
Thanks for the responses.
@Michael: Yup, I've noticed that there is a possibility for pushing a
message via management API but the annotation in docs:
"Please note that the publish / get paths in the HTTP API are intended
for injecting test messages, diagnostics etc - they do not implement
reliable delivery and so should be treated as a sysadmin's tool rather
than a general API for messaging."
does not look encouraging for me. Or it is not as bad as it is stated
in docs? ;)
@Alvaro: Thanks for the additional resources! :)
Regarding plug-ins and providers - I've talked with CloudAMQP support,
and they've assured me that they will support any custom plugin if it
is compliant with specified versions of Erlang and RabbitMQ (17.4 and
3.4.3). Another requirement for custom plug-in is a paid plan with a
dedicated instance (but it sounds like a reasonable requirement for me
- they'd like to avoid collateral damage on shared instances caused by
strange plug-ins ;)).
This attitude related with "no additional service" it is not a my
requirement (I didn't state that explicitly) and I have to stick with
that, but from what I'm seeing the actual difference will be really
insignificant (in other words - plug-in will be such additional
application, the only difference is that it will be hosted side by
side with a RabbitMQ instance).
Best regards,
--
Wojciech Gawroński (afronski)
@Michael: Yup, I've noticed that there is a possibility for pushing a
message via management API but the annotation in docs:
"Please note that the publish / get paths in the HTTP API are intended
for injecting test messages, diagnostics etc - they do not implement
reliable delivery and so should be treated as a sysadmin's tool rather
than a general API for messaging."
does not look encouraging for me. Or it is not as bad as it is stated
in docs? ;)
@Alvaro: Thanks for the additional resources! :)
Regarding plug-ins and providers - I've talked with CloudAMQP support,
and they've assured me that they will support any custom plugin if it
is compliant with specified versions of Erlang and RabbitMQ (17.4 and
3.4.3). Another requirement for custom plug-in is a paid plan with a
dedicated instance (but it sounds like a reasonable requirement for me
- they'd like to avoid collateral damage on shared instances caused by
strange plug-ins ;)).
This attitude related with "no additional service" it is not a my
requirement (I didn't state that explicitly) and I have to stick with
that, but from what I'm seeing the actual difference will be really
insignificant (in other words - plug-in will be such additional
application, the only difference is that it will be hosted side by
side with a RabbitMQ instance).
Best regards,
--
Wojciech Gawroński (afronski)
Michael Klishin
Mar 2, 2015, 9:48:07 AM3/2/15
to Wojtek Gawroński, Alvaro Videla, rabbitm...@googlegroups.com
On 2 March 2015 at 17:43:31, Wojtek Gawroński (afro...@gmail.com) wrote:
> @Michael: Yup, I've noticed that there is a possibility for
> pushing a
> message via management API but the annotation in docs:
>
> "Please note that the publish / get paths in the HTTP API are intended
> for injecting test messages, diagnostics etc - they do not implement
> reliable delivery and so should be treated as a sysadmin's tool
> rather
> than a general API for messaging."
>
> does not look encouraging for me. Or it is not as bad as it is stated
> in docs? ;)
That says you won't have Publisher Confirms [1]. Most HTTP-based
> @Michael: Yup, I've noticed that there is a possibility for
> pushing a
> message via management API but the annotation in docs:
>
> "Please note that the publish / get paths in the HTTP API are intended
> for injecting test messages, diagnostics etc - they do not implement
> reliable delivery and so should be treated as a sysadmin's tool
> rather
> than a general API for messaging."
>
> does not look encouraging for me. Or it is not as bad as it is stated
> in docs? ;)
services don't have any alternative either. And yes, I'm only recommending
it because nothing else is going to work in your case.
You can always build your own HTTP gateway, as Álvaro points out.
1. http://www.rabbitmq.com/confirms.html
Wojtek Gawroński
Mar 2, 2015, 9:59:12 AM3/2/15
to Michael Klishin, Alvaro Videla, rabbitm...@googlegroups.com
Okay - now it makes perfect sense, I didn't know that it affects only
the publisher acknowledgement.
Thanks! :)
--
Wojciech Gawroński (afronski)
the publisher acknowledgement.
Thanks! :)
--
Wojciech Gawroński (afronski)
Carl Hörberg
Mar 2, 2015, 12:20:54 PM3/2/15
to rabbitm...@googlegroups.com, mkli...@pivotal.io, videl...@gmail.com
Could there be a point in using publishing confirm when using the publishing via the HTTP API? So it won't return 200 until it has gotten a confirmation? Optionally only when appending a query string .../publish?confirm=true. And then remove the warning text in the HTTP API doc that scares a lot of ppl?
(Yes, the HTTP API is TLS protected on all CloudAMQP RabbitMQ servers/clusters)
(Yes, the HTTP API is TLS protected on all CloudAMQP RabbitMQ servers/clusters)
Simon MacMullen
Mar 2, 2015, 12:23:54 PM3/2/15
to Carl Hörberg, rabbitm...@googlegroups.com, mkli...@pivotal.io, videl...@gmail.com
Yes, there could :-)
https://github.com/rabbitmq/rabbitmq-management/blob/master/src/rabbit_mgmt_wm_exchange_publish.erl#L54
So maybe the docs should be rephrased; we do use confirms on publish in
the HTTP API, and publish is generally trustworthy. The real issue is
delivering messages - there's no sensible way to do message acks using
the HTTP API.
Cheers, Simon
On 02/03/15 17:20, Carl Hörberg wrote:
> Could there be a point in using publishing confirm when using the
> publishing via the HTTP API? So it won't return 200 until it has gotten
> a confirmation? Optionally only when appending a query string
> .../publish?confirm=true. And then remove the warning text in the HTTP
> API doc that scares a lot of ppl?
>
> (Yes, the HTTP API is TLS protected on all CloudAMQP RabbitMQ
> servers/clusters)
>
> On Monday, March 2, 2015 at 10:59:12 PM UTC+8, Wojtek Gawroński wrote:
>
> Okay - now it makes perfect sense, I didn't know that it affects only
> the publisher acknowledgement.
> Thanks! :)
> --
> Wojciech Gawroński (afronski)
>
>
> On 2 March 2015 at 15:47, Michael Klishin <mkli...@pivotal.io
https://github.com/rabbitmq/rabbitmq-management/blob/master/src/rabbit_mgmt_wm_exchange_publish.erl#L54
So maybe the docs should be rephrased; we do use confirms on publish in
the HTTP API, and publish is generally trustworthy. The real issue is
delivering messages - there's no sensible way to do message acks using
the HTTP API.
Cheers, Simon
On 02/03/15 17:20, Carl Hörberg wrote:
> Could there be a point in using publishing confirm when using the
> publishing via the HTTP API? So it won't return 200 until it has gotten
> a confirmation? Optionally only when appending a query string
> .../publish?confirm=true. And then remove the warning text in the HTTP
> API doc that scares a lot of ppl?
>
> (Yes, the HTTP API is TLS protected on all CloudAMQP RabbitMQ
> servers/clusters)
>
> On Monday, March 2, 2015 at 10:59:12 PM UTC+8, Wojtek Gawroński wrote:
>
> Okay - now it makes perfect sense, I didn't know that it affects only
> the publisher acknowledgement.
> Thanks! :)
> --
> Wojciech Gawroński (afronski)
>
>
> On 2 March 2015 at 15:47, Michael Klishin <mkli...@pivotal.io
> <javascript:>) wrote:
> >> @Michael: Yup, I've noticed that there is a possibility for
> >> pushing a
> >> message via management API but the annotation in docs:
> >>
> >> "Please note that the publish / get paths in the HTTP API are
> intended
> >> for injecting test messages, diagnostics etc - they do not
> implement
> >> reliable delivery and so should be treated as a sysadmin's tool
> >> rather
> >> than a general API for messaging."
> >>
> >> does not look encouraging for me. Or it is not as bad as it is
> stated
> >> in docs? ;)
> >
> > That says you won't have Publisher Confirms [1]. Most HTTP-based
> > services don't have any alternative either. And yes, I'm only
> recommending
> > it because nothing else is going to work in your case.
> >
> > You can always build your own HTTP gateway, as Álvaro points out.
> >
> > 1. http://www.rabbitmq.com/confirms.html
> <http://www.rabbitmq.com/confirms.html>
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
>
> >> @Michael: Yup, I've noticed that there is a possibility for
> >> pushing a
> >> message via management API but the annotation in docs:
> >>
> >> "Please note that the publish / get paths in the HTTP API are
> intended
> >> for injecting test messages, diagnostics etc - they do not
> implement
> >> reliable delivery and so should be treated as a sysadmin's tool
> >> rather
> >> than a general API for messaging."
> >>
> >> does not look encouraging for me. Or it is not as bad as it is
> stated
> >> in docs? ;)
> >
> > That says you won't have Publisher Confirms [1]. Most HTTP-based
> > services don't have any alternative either. And yes, I'm only
> recommending
> > it because nothing else is going to work in your case.
> >
> > You can always build your own HTTP gateway, as Álvaro points out.
> >
> > 1. http://www.rabbitmq.com/confirms.html
> <http://www.rabbitmq.com/confirms.html>
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> To post to this group, send email to rabbitm...@googlegroups.com
> <mailto:rabbitm...@googlegroups.com>.
Simon MacMullen
Mar 2, 2015, 12:30:18 PM3/2/15
to Carl Hörberg, rabbitm...@googlegroups.com, mkli...@pivotal.io, videl...@gmail.com
Hopefully this is clearer now:
https://github.com/rabbitmq/rabbitmq-management/commit/5840f0e5d0d897a30967d9c97d2ddd6bc7981fb3
Cheers, Simon
https://github.com/rabbitmq/rabbitmq-management/commit/5840f0e5d0d897a30967d9c97d2ddd6bc7981fb3
Cheers, Simon
Carl Hörberg
Mar 2, 2015, 12:37:12 PM3/2/15
to rabbitm...@googlegroups.com, ca...@cloudamqp.com, mkli...@pivotal.io, videl...@gmail.com
Ok ;)
Yes, that's better! But if HTTP Keep-alive is used a new TCP connections isn't technically required, get your point though..
The HTTP get message method isn't more dangerous than BasicGet or auto-ack, is it?
Yes, that's better! But if HTTP Keep-alive is used a new TCP connections isn't technically required, get your point though..
The HTTP get message method isn't more dangerous than BasicGet or auto-ack, is it?
>> <mailto:rabbitmq-users+unsub...@googlegroups.com>.
Michael Klishin
Mar 2, 2015, 12:44:06 PM3/2/15
to rabbitm...@googlegroups.com, videl...@gmail.com, Carl Hörberg
On 2 March 2015 at 20:37:13, Carl Hörberg (ca...@cloudamqp.com) wrote:
> The HTTP get message method isn't more dangerous than BasicGet
> or auto-ack, is it?
It's pretty much basic.get with auto-ack, as far as guarantees go.
> The HTTP get message method isn't more dangerous than BasicGet
> or auto-ack, is it?
Reply all
Reply to author
Forward
0 new messages