GCE Pub/Sub vs RabbitMQ\AMQP?

[Oren Zomer]

Hi,

What are the pros and cons of choosing Pub/Sub vs RabbitMQ\AMQP?

I noticed that cloudamqp.com provides an aaS solution over GCE: https://www.cloudamqp.com/gce.html .

Thanks

[Adam]

More specifically, CloudAMQP provides their own managed RabbitMQ service, as well as a deployment solution on Google Compute Engine. Google also provides a click-to-deploy solution of RabbitMQ as part of the Cloud Launcher. Finally there is Cloud Pub/Sub as an alternative.

The trade-offs between these are largely ease of management versus cost versus ecosystem. If you have services already local to GCP then Pub/Sub or GCE hosted RabbitMQ might be a good solution. If your code is already tied into Google APIs you might want to stick with Pub/Sub. RabbitMQ has a free hosted tier if you need a message broker and are just developing an app. On the flip side, RabbitMQ also is more mature than PubSub and you might depend on some of its features like Web-Stomp. It really depends on your own unique situation.

[Kir Titievsky]

The advantage of RabbitMQ over Pub/Sub is the ecosystem of connectors. A persistent complaint we hear from our users about RabbitMQ is its reliability. Of course, it is a function of how expert you are at setting up and maintaining this OSS package and how much you invest in learning and maintenance.  RabbitMQ is successfully deployed by many.

Still, getting to consistent, global scaling from 1 message a second to millions -- which Cloud Pub/Sub -- is likely worth the effort of working through some of the growing pains of our client libraries, which I know you've encountered Oren.  As with all managed services this value is in being able to invest your attention, particularly in the long term, where maintenance and scaling come into play, into your business logic layers rather than infrastructure.

Kir

Product Manager

Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/9d180244-8193-4102-9d02-ff99d8d70008%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Sharif Alvis]

Would you expect better latency from a gke hosted rabbitmq or pubsub (if my app is also in the same gke cluster)?

Thanks,

Sharif

[Kir Titievsky]

I suspect you can push Rabbit to give you lower latency since it does not require data replication and gives you some flexibility on persistence configuration.  From what I understand, it does not even write data to disk by default.  Here's more on various issues there.

By contrast, Pub/Sub will ensure that all messages are durably persisted and delivered with consistent, although, >> milisecond latency, with no configuration.

If you are talking about latency to Publish message, which presumably unblocks your main worker thread, that's often immaterial since Pub/Sub publishing is typically done asynchronously anyway.

End-to-end, however, you can't beat an in memory queue as long as it can stand the load.

On Tue, Mar 21, 2017 at 6:07 PM, Sharif Alvis <sharif...@gmail.com> wrote:

Would you expect better latency from a gke hosted rabbitmq or pubsub (if my app is also in the same gke cluster)?

Thanks,

Sharif

On Monday, 6 February 2017 14:26:40 UTC, Kir Titievsky wrote:

The advantage of RabbitMQ over Pub/Sub is the ecosystem of connectors. A persistent complaint we hear from our users about RabbitMQ is its reliability. Of course, it is a function of how expert you are at setting up and maintaining this OSS package and how much you invest in learning and maintenance.  RabbitMQ is successfully deployed by many.

Still, getting to consistent, global scaling from 1 message a second to millions -- which Cloud Pub/Sub -- is likely worth the effort of working through some of the growing pains of our client libraries, which I know you've encountered Oren.  As with all managed services this value is in being able to invest your attention, particularly in the long term, where maintenance and scaling come into play, into your business logic layers rather than infrastructure.

Kir

Product Manager

Google Cloud Pub/Sub

On Sun, Feb 5, 2017 at 5:27 PM, 'Adam' via Google Cloud Pub/Sub Discussions <cloud-pubs...@googlegroups.com> wrote:

More specifically, CloudAMQP provides their own managed RabbitMQ service, as well as a deployment solution on Google Compute Engine. Google also provides a click-to-deploy solution of RabbitMQ as part of the Cloud Launcher. Finally there is Cloud Pub/Sub as an alternative.

The trade-offs between these are largely ease of management versus cost versus ecosystem. If you have services already local to GCP then Pub/Sub or GCE hosted RabbitMQ might be a good solution. If your code is already tied into Google APIs you might want to stick with Pub/Sub. RabbitMQ has a free hosted tier if you need a message broker and are just developing an app. On the flip side, RabbitMQ also is more mature than PubSub and you might depend on some of its features like Web-Stomp. It really depends on your own unique situation.

On Sunday, February 5, 2017 at 7:41:01 AM UTC-5, Oren Zomer wrote:

Hi,

What are the pros and cons of choosing Pub/Sub vs RabbitMQ\AMQP?

I noticed that cloudamqp.com provides an aaS solution over GCE: https://www.cloudamqp.com/gce.html .

Thanks

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/9d180244-8193-4102-9d02-ff99d8d70008%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/3e979be3-3fd7-4c85-890b-87ad588bdc35%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vinicius Carvalho]

My 2c on the subject (disclaimer I'm employed by Pivotal who maintains rabbitMQ), I'm also part of the Spring Cloud Stream (and we maintain binders for rabbit/kafka/jms and pubsub [one day] )

Rabbit is an amazing broker if you need to rely on very complex messaging routing capabilities. The fact that it allows you to create exchanges and map those to receiving queues and use expressions to route your payload as you see fit, is very attractive on traditional EAI scenarios that most companies face. It's also a very mature project and you will find thousands of developers and plugins available for it. It's simple to get started, and has a low memory footprint. Rabbit support message ordering (can be a big thing for some users)

Having said that, rabbit does have it's own issues: It's backed by in memory queues, and I have seen clusters of rabbitmq come to their knees once the number of unack messages reached the millions. Also rabbit has a limitation on how many messages it can process per second, it's related to how erlang mailboxes work and how it handles queues. It's very easy to get to 20~30k messages a second on a queue, but if you want to see that number in the millions, you'd better do some partitioning with queues to support that.

My opinion is that rabbit is a great choice for a lot of traditional EAI cases, even some IoT with low volume thanks to mqtt support. But it can't beat kafka/pubsub when it comes to horizontal scalability

I've also noticed that with a single client, can handle a higher throughput than pubsub

Now, pubsub to in my view, is when you need almost infinite scalability. It looks like it was designed with massive horizontal scalability in mind, so 1 ... 1000 publishers would not even tickle the system, and you can get some peace of mind when scaling out your applications. The other huge thing to me is the replication support cross regions, this is huge, I think it may be one of the only ones out that doing it right (many say they will do it, but mostly fail). Pubsub does make tradeoffs though, there's no message ordering guarantee, which depending on your use case can be a problem. Large volume use cases really seldomly rely on transactional semantics, so if ordering is not the case, just embrace it and be happy.

I really would love to see pubsub offer replay capabilities, since it's already persistent, would be nice if we had the same support kafka has to replay messages given an offset.

Your choice will really depend on your use case, they are all amazing products, very well built and suited for a given use case. 

Regards

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/9d180244-8193-4102-9d02-ff99d8d70008%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/3e979be3-3fd7-4c85-890b-87ad588bdc35%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kir Titievsky]

Vinicius, Thank you for this excellent summary!  

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/9d180244-8193-4102-9d02-ff99d8d70008%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/3e979be3-3fd7-4c85-890b-87ad588bdc35%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/345744cc-a47d-4261-9349-cd050270ef49%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.