Using pull queues with tagging to battle contention / batch tasks

[Kaan Soral]

Hi everyone,

I'm wondering whether anyone has any experience with pull queues and tagging, I'm using pull queues actively, they are 100 times more reliable than push queues as far as I'm concerned, yet I haven't tested the experimental tagging capability

Basically, I'm thinking of tagging tasks by key names, and pulling/processing tasks for entities

So basically there will be many tag names and many operations simultaneously

Is this possible?

Here is a related SO question I asked: http://stackoverflow.com/questions/29052160/app-engine-can-pull-queue-tasks-handle-arbitrary-tags

This wouldn't be possible if there is any hidden schema's or limits on parallel tag usages

[Kaan Soral]

Well I went ahead and utilized the pull queues

Replaced my existing taskqueue/db routine that I stress tested here: https://groups.google.com/forum/#!topic/google-appengine/qxQZJGHd8c4

I have to say I love pull queues, out of 1M separate increment tasks, the total result was 1M+25, the error is +~0.00-ish, which I consider acceptable (it might have a separate cause, yet there was a single .delete_tasks exception that might be the cause, unimportant)

Some pro's:

• Didn't notice any chronic put or pull exceptions
• Pull's are chronological, at least in my limited trials they were 100% chronological, they should be (are they chronological as a feature?)
• Arbitrary tags seem to work
• The put and pull performance is very very satisfying, you can repeatedly pull and delete 500 entities for example, from a single tag

If anyone from google reads this, please kiss whoever implemented/designed pull queues for me

I really wish all of the services were this robust, I hope it always continues like this

[Jay]

Glad you have met with success. I have also been using pull queues with task tags as counters and for other aggregating behavior as well. I have also enjoyed success with this approach. 

For simple write heavy counters, I think I *may* prefer the good old sharded counter. There are plenty of use cases where pull queues with tagging are an excellent solution.

[Kaan Soral]

Even in this new pull queue setup, I was haunted by the push queue issues, upon inspection I found out that some tasks that are pushed into the push queues just didn't execute, google ignores these issues for years

(Each pull queue addition triggers a routine with taskqueue's to handle the operations by pulling tasks from the pull queues, if it's not already running)

I ended up configuring the push queues to infinite retries like there's no tomorrow, as other's claimed it to be the solution to the invisible retry depletion bug

I previously had a routine that checks the database for old tasks and re-triggers the push queue routine manually, yet this is not as simple with pull queues, as you can't just query for old tasks

(Although, just leasing ~100 untagged tasks for 1 second and parsing them for leftovers work, but it might disrupt the running routines)