Increasing task queue quotas

[James Cooper]

Hi,

I'm evaluating GAE suitability for one of my clients. It is a B2B app
with very low organic web traffic. However, users upload lists of
data that need to be processed in the background.

In the absence of native MapReduce support, my plan is to use task
queues. But to meet the performance needs of this application, I need
to burst to relatively high levels of concurrency.

My question is about the discrepancy in the GAE quotas for inbound web
traffic vs. task queues.

Inbound web traffic can burst to: 500 qps
Task queues burst to: 20 qps

In my mind, the total concurrency that GAE would provide an app is
equal to:
inbound qps + task queue qps + cron qps.

Given that task queues are implemented as web request handlers, I see
little infrastructural reason to distinguish between task queue and
inbound traffic.

Is it possible to request a task queue quota increase to 500 qps? If
not is there a technical reason for this?

thanks

-- James

[Julian Namaro]

I'm not sure about what you want to do but just a thought: have you
considered Amazon Elastic MapReduce ?
It's sure doable with task queues but you're likely to encounter
various limitation problems as you cite.

[djidjadji]

Is your processing not possible with the deferred module added in 1.2.5

http://code.google.com/appengine/articles/deferred.html

If you launch many tasks to process the uploads you waste a lot of CPU
retrieving the data needed. With the deferred method you run for about
30 sec, then store the state and schedule to get called again. You can
setup multiple deferred tasks to process 1 upload by using parameters
of the function that deferred calls.

2009/11/18 Julian Namaro <namaro...@gmail.com>:

> --
>
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
>
>
>

[James Cooper]

Hi there,

Yes, I've considered Amazon's offering, but would prefer to keep the
entire system on GAE if possible to reduce complexity.

thanks,

-- James

[James Cooper]

Hi,

My primary concern is throughput, not cost. I would be happy to pay
more to gain higher concurrency and higher throughput.

If a customer uploads a list with 100,000 items to process, I would
like to burst to some upper limit of concurrency (higher than 20 qps
at least) and process the list as quickly as possible.

This seems to be something GAE could provide if the concurrency limits
were raised. My question is whether Google would allow a customer to
request a quota increase for task queue concurrency in situations like
this, or whether that's forbidden across the board for some reason.

Anyone from Google who can help answer this?

thanks,

-- James