Python Multiprocessing in App Engine

[Ankit Dwivedi]

Hi Friends,

We are stuck with a issue related to multiprocessing. We tried to use multiple solutions like below.

1. from multiprocessing.dummy import Pool

     pool = Pool(2)

     pool.apply_async(call_api,xxxx)

2. import concurrent.futures

    import requests

    pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)

        pool.submit(lambda:requests.post(xxxxxx))

above given both solutions are working in my local system, when when deploying this functionality is not working, even not getting any error in logs.

can you please help me to understand, why this is not working and what else we can use.

[Elliott (Cloud Platform Support)]

Hello,

Thank you for your question. I understand that you have an issue with multiprocessing in your application, not on the local machine and there are no logs. You used the following code to fix the problem:

1. from multiprocessing.dummy import Pool

     pool = Pool(2)

     pool.apply_async(call_api,xxxx)

2. import concurrent.futures

    import requests

    pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)

        pool.submit(lambda:requests.post(xxxxxx))

You posted to the App Engine Google Groups forum. From the description, I am unable to fully understand what the issue is. Can you please provide additional detail? Also, for the benefit of the community, can you please add comments to the code that indicates what it means to you, which will help us to offer you advice on the issue you are encountering? I understand that non running code can be tricky to troubleshoot.

I was able to find this link to the Stackoverflow article after looking at your code. It describes how the author wants to run threads in a ThreadPoolExecutor until one of them gives a result. The answer indicates to keep checking each future in the sequence until it is found.

Google Groups is meant for general discussion about Google products and services but we will do our best to help.