High Throughput Distributed Queue with Persistence

[Piyush Katariya]

Hey Guys,

Could you please help me decide the cloud tools which i can use to achieve High Throughput Distributed Queue with Persistence ?
Distributed queue should support at least few thousands of Publishers (PUSH) and subscribers (PULL).
Message ordering is not a priority for my application.
Message PULL latency observed by consumer after publisher is done with PUSH should not be more than 10ms.
Message must not be lost at any cost. (at least once guarantee support or Exactly once guarantee would be awesome)
Minumum required throughput should be 10K messages per second but should be able scale if required.

[Kir Titievsky]

Piyush,

There are two options: TaskQueues on App Engine 1.9 and Cloud Pub/Sub everywhere else.  Briefly, Pub/Sub offers fan-out (not just a queue) and is global (publish from any region, consume from any region). TaskQueues implement a queue (so no fan-out), are regional and are much better integrated with App Engine standard.   It seems you may have more decisions to make here than just the messaging system:

1. How do you publish?  "PUSH" suggests you may have some constraint on the message producer.  Say it's a legacy piece of code that requires a webhook endpoint.  Both services have REST APIs, but are better used with native client libraries. 

2. Where your subscribers live.  Since this is an AppEngine forum, I'll guess you are considering using GAE for processing the messages. Given your throughput, it might be more cost effective and faster to run consumers on GCE/GKE -- somewhere where you can run multi-threaded message consumers. At a steady 10K messages/second you are not going to benefit much from being able to scale down to 0.  This is a guess so take it as such.  There are other considerations

k

[Piyush Katariya]

Thanks for the reply.

No I dont need Pub/sub capabilities, as I mentioned explicitly I need distributed queue

I will be hosting subscribers code in GAE.

[Kir Titievsky]

You can use Cloud Pub/Sub does work as a distributed queue.  So does TaskQueues.

Re-reading your requirements, I think it's the "10ms" message delivery latency.  You can get there at the median, but not in the tail, with these services where you would expect >10ms end-to-end latency.   If you need 10ms latency in the tail (or 10ms for 99.99% of your requests), you may have to implement your own service, likely sacrificing some availability or durability relative to both of the managed offerings.  A popular piece of software there is RabbitMQ.  You can also tune Apache Kafka to go that low, particularly, if you are willing to sacrifice replication and are willing to run it in the same zone as you application.  I hear good things about Redis, as well. 

Might I suggest you start with TaskQueues and see if they get you close enough?  

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/D13aR1woYXA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/856c41a0-6ef0-48c7-9272-e223e623ac04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub