Task queue doesn't run all of the tasks and HTTP requests fail

[David Phillips]

I have an app running on a backend.  It has 11 tasks. The first one is started by /_ah/start and it, in turn, starts the other ten, the worker tasks. The worker tasks have this structure:

while True:
    # Do important stuff
    # time.sleep (30)

The execution behavior on app engine is the same every time. The scheduling task runs and enqueues the 10 worker tasks. The first seven of the ten that are enqueued start running and execute correctly. The last three sit in the queue, never running. The app also stop responding to HTTP requests, returning 503 status codes. The logs do not show that my http handlers getting invoked.

The worker task queue is configured with a maximum rate of 1/s and 2 buckets. It's curious that the admin console shows that the enforced rate is 0.1 sec. Since the tasks run forever, the don't return unsuccessful completion codes. And the cpu load is negligible. The workers mostly do a URL fetch and then wait 30 seconds to do it again.

The logs are not helpful. I don't know where to go to find diagnostics that will help me figure it out. I'm testing in a free account. Could there be a limit of 8 tasks executing at one time? I'd like to run quite a few more tasks in parallel eventually.

Thanks for any advice you can give me.

[Amy Unruh]

David,

The bucket_size and max_concurrent_requests do determine how many tasks are running at once.  It sounds like you probably want to increase your bucket size for your use case.

If you are seeing the last three tasks in the queue even after the others have finished, then as a starting point check two things:

  - the task queue quotas (via the 'Task Queues' link in the admin console sidebar)

  - check that the tasks that remain in the queue aren't throwing errors and being periodically retried

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/S3PAugOQEkwJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.