When to use PubSub vs Task Queues

[Joshua Fox]

This Task Queues page  says "Tip: In some cases Google Cloud Pub/Sub is a good alternative to pull queues."

Is there a doc that  says more about that?

- In what cases are Pub/Sub or  Task Queues preferable? 

- Why is PubSub only an alternative for pull task-queues given that both push- and pull-style subscriptions are supported in Pub/Sub?

- Since Pub/Sub appears more general, why do Task Queues exist? Is it just the scheduled push feature? Or the specific support for serialization of Task objects?

[Kim Lewandowski]

Hi Joshua --

It’s a good question, and it has a complicated answer. Broader documentation on when to prefer one over the other is in process, and we hope that the feature set we expect to evolve in Cloud Tasks (Alpha signup here) will make the distinctions clearer.

Much of the answer is historical. Task Queues evolved as part of the App Engine developer libraries (back in 2009) and was (internally) slated for deprecation and replacement with release of Pub/Sub. As you observed: Pub/Sub does seem like a strong generalization of Task Queues. When Pub/Sub launched, however, we found that it did not meet the needs of a significant number of customers. A number of features that we thought weren’t particularly import (de-duplication, scheduled delivery, rate management, concurrency management, tagged tasks, configurable retries, etc) turned out to be critical and not supported in Pub/Sub.

In response to that feedback, we took a step back and tried to define the core differences. The conclusion we arrived at is that Pub/Sub is a networking and big data product. It is agnostic to both the content and the target and focuses on delivering very high throughput capabilities to arbitrary handlers. Task Queues is a managed execution product. It is intended for running and managing large numbers of explicit commands. We’re working to repackage Task Queues as Cloud Tasks and open it up for broader integration with the rest of GCP. For example, we want to make it possible to enqueue Datastore operations or Cloud Functions for asynchronous execution. The idea is to make it as easy as possible to spin up large distributed systems without needing to write handlers and managers.

An example of what we are trying to enable is the App Engine mapreduce library (written on top of Datastore and Task Queues). Instructions (in this case, map or reduce shards) are explicitly encoded and a user-space library can capitalize on Task Queues infrastructure to avoid needing any management tools or orchestration services.

I hope that’s helpful and feel free to send me more details on your use case and we can provide you with more guidance.

-Kim

Product Manager of Task Queues

[Stephen]

On Fri, Feb 3, 2017 at 6:30 PM, 'Kim Lewandowski' via Google App Engine <google-a...@googlegroups.com> wrote:
> We’re working to repackage Task Queues as Cloud Tasks and open it up for
> broader integration with the rest of GCP. For example, we want to make it
> possible to enqueue Datastore operations or Cloud Functions for asynchronous
> execution. The idea is to make it as easy as possible to spin up large
> distributed systems without needing to write handlers and managers.

A feature request for Cloud Tasks:

One nice cloud feature added since task queues appeared is Preemptible VM Instances. I'd like to be able to create tasks and specify that a deadline any time this week is fine, so run it at reduced cost to me whenever convenient for you.

For example, when sending email via Amazon SES there is transaction limit. I want to send a low priority news letter to many addresses no more than 30/s, within 7 days, using a preemptible vm if possible. If a user forgets a password the forgot-password email should jump to the head of the queue, and if no preemptible vm is available a standard one should be started. The 30/s limit should be maintained.

'preemptible' should be an attribute you can give to an app engine machine type.

[Joshua Fox]

Thank you. We indeed use Task Queues for scheduling tasks, so if I understand correctly, PubSub could not serve as a replacement.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/70d72de0-84b8-44bd-974b-df6bca6f0ead%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

JOSHUA FOX

Principal Software Architect | Freightos

☏ (Israel): +972-545691165 | ☏ (US):  +1-3123400953 | Skype: joshuafox.freightos

Smoother shipping with the world's online freight marketplace.

Online Marketplace / In the News / Freightos Blog

[Kim Lewandowski]

Hi Stephen,

Thanks for the feature request. Cost-optimizing and priority queues are definitely something we're considering down the road. Right now we are focused on getting Cloud Tasks to Generally Available before we prioritize new features. I encourage you to sign up for the Cloud Tasks Alpha if you haven't already. Once whitelisted, you'll be added to a private group where we're discussing all things Cloud Tasks queues related. It will be a good place to get more up to date information on our status and to provide us with additional details on your use cases.

[Alex Buchanan]

As of today though, what are the advantages to using Pub/Sub over Pull Queues? For example, are there any time or scale limits to using Pull Queues that Pub/Sub resolves?

Thanks,

Alex

[Evan Jones]

Personal opinion:

* Task queues are probably better to use from App Engine Standard. They've existing for a long time, and I suspect the RPC API that is used to call App Engine services is pretty efficient. Using Pub/Sub is likely to be slightly worse in terms of overhead per API call, since it will have to go through the HTTP API via URL Fetch.

* Pub/Sub is probably better to be used from anywhere else. The API works from anywhere. A single Pub/Sub topic has nearly no limits, while task queues top out at around 500 messages / second. You can also "replicate" a topic to multiple subscribers, which is useful for some applications.

There are some other minor differences, but IMO those are the ones that I would consider. Hope than helps,

Evan