App Engine can't handle any kind of burst ...

[Kaan Soral]

I'm really fed up with app engine lately

I had a routine to test an internal system, it triggers 1000 tasks that each adds 1000 items to datastore with .put_async / totalling 1M entities, each of these secondary tasks execute 1 second after each other

Another routine handles these entities, and deletes them with .delete_async (deletes are sequential/slow and delayed, so it's not a 1 second datastore contention issue)

When this routine runs, each/any operation on that model fails

.put_async's fail with: suspended generator put(context.py:810) raised DeadlineExceededError(The API call datastore_v3.Put() took too long to respond and was cancelled.)

.delete_async's fail with: suspended generator delete(context.py:844) raised TransactionFailedError(too much contention on these datastore entities. please try again.)

the model locks up

[Jay]

Hmm ... ALL the puts fail? How many instances does the scheduler spin up? Are the entities all in their own entity group? 

You said that each task executes 1 second after the previous task. So you expect the put part of the test to take about 16, 17 minutes. Is that right?

How deep does the task queue(s) that you are using get? Are you saying "countdown=1 but perhaps the tasks are queuing up?

[Kaan Soral]

Many .put_async's failed, I couldn't count the exact number, but I'm guessing 50%, as the result of the operation was ~460k instead of a clean million

I don't utilise entity groups, they are free entities

200 instances spinned up, tasks definitely queued, yet this was a burst operation, so that was expected

on the .put_async front, I don't think .put's should ever timeout

on the .delete_async front, I don't know why delete's are in a transaction, should they really be?

I mean shouldn't you be able to delete an entity 3 times simultaneously?

As a result of the operation, I found a bug that affected the .delete_async part, yet I'm sure the overload would occur without the bug, the .put_async part is separate from the .delete_async part