Is Purging a Task Queue Programmatically Not Working On Map Reduce (mapreduce) Controller Tasks Using Python?

[Camilo Silva]

Dear GAE Team & Enthusiasts:

At work, I run daily automated map reduce jobs using Python as a pipeline from the Datastore to GCS.

The issue is that in Python, the "mrcontroller" is always lingering 

(I already posted this issue - and someone got back to me)

( https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/google-appengine/TUf4OvITSG4/wmftKhQIQBQJ )

Anyways, in order to circumvent for now that challenge, I opted to use the purge queue functionality in order to clear my task queue handling the map reduce stuff after it has been completed (see Python code below):

      q = taskqueue.Queue(configvars.TASK_QUEUE)

      q.purge()

While the call seems to execute without exceptions, unfortunately, the tasks present ("POST /mapreduce/controller_callback/1574180594759CAA69707") in my tasks queue above are never purged.

I always need to go back to the admin panel, pause the queue, and then delete each spawning non-ending "mrcontroller" callback task.

This is a serious issue because these tasks accumulate and keep making call backs which are translated to frontend calls which later on translate to $$$ (hundreds) -- that's right -- we are literally paying for a stupid bug of the mapreduce API.

Anyways, any help or insight on this would be super helpful.

Maybe there's another way to clear the tasks of the taskqueue?

Thanks for your time and help.

Regards,

Camilo

[Nick (Cloud Platform Support)]

Hey Camilo,

This definitely sounds like an issue with the purge function. Your report is pretty detailed, and is good as an explanation of the issue, although it's hard to proceed from here without being able to reproduce the issue for fixing. Is it possible that you have a minimal reproducing example which can show this behaviour? You could file an issue at the github issue tracker for GAE Mapreduce and hopefully get this addressed. There seems to be a related issue already, although you'll want to create a clear issue of your own, linking to it.

Best wishes,

Nick