Multiprocessing app engine issue

[Ankit Dwivedi]

Hi Team,

We are stuck with a issue, we are trying to use multiprocessing. We are using below ways.

1. from multiprocessing.dummy import Pool

    pool = Pool(20)

    pool.apply_async(call_api, xxxxxxxx)

2. import concurrent.futures

    import requests

    pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)

    pool.submit(lambda:requests.post(xxxxx))

We are using python 3.7 and app.yaml file is below.

service: abc

runtime: python37

instance_class: B4_1G

entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker app.main:app

basic_scaling:

  idle_timeout: 10m

  max_instances: 25

Both the methods are working fine in our local machines(computers) but when deploying on app engine these functionality are not working even we are not getting any error in logs.

Our use case is, we have two services A and B in service B we have written a api to upload file into google bucket, and we are calling this api from service A, we are uploading big files so we cant wait to complete, so we want response back from service A after call Service B api.

Please help to understand, what is the issue and how can we achieve this.  

[jlibera]

Hello,

For your use case, would you be able to confirm if the files are indeed being successfully uploaded to your GCS bucket? Also, would you be able to verify if the public documentation for logging with [1] your Python App Engine application was already consulted? Are you able to call other functionalities from your services (downloading, listing, etc.)? Perhaps the issue may lies in your code implementation which I would suggest to reach out to communities such as Stack Overflow [2] or Github for such inquiries as they would be better suited for developmental questions. 

Furthermore, I would advise to look through documentation on how App Engine handles it's requests [4] in order to gain some insight on how your requests are received and responses sent for your application. 

[1] https://cloud.google.com/appengine/docs/standard/python/how-requests-are-handled#logging

[2] https://stackoverflow.com/

[3] https://github.com/GoogleCloudPlatform/python-docs-samples/issues

[4] https://cloud.google.com/appengine/docs/standard/python3/how-requests-are-handled

[Ankit Dwivedi]

Hi, 

yes we are successfully able to upload file to bucket, but as earlier said we have two services A and B and calling service B api from service A to upload the file. All the api's working properly but when calling service B api from service A to upload the file, api request is waiting for completion and we want response back from service A without waiting for the response.

We tried followed app engine cloud task approach but stuck, as we are not able to send image in request body.

Please advise how we can achieve this.

[Ankit Dwivedi]

Hi Team,

Can you please suggest, how we can resolve this?

[George (Cloud Platform Support)]

This discussion group is oriented more towards general opinions, trends, and issues of general nature touching the App Engine. For coding and program architecture, as well as implementing multiprocessing in your code, you may be better served in dedicated forums such as stackoverflow, where experienced programmers are within reach and ready to help.