Cloud Run w/ pub/sub integration never stops running

[Adriano Matos]

I have a Cloud Run service integrated with pub/sub so each time a new file gets added to a bucket, it triggers my Cloud Run service. For some reason, when a push a single file into a bucket, the Cloud Run service is able to perform operations successfully (returning a POST 204), but it runs on the same file endlessly on a loop. I tried purging all my messages in my service's subscription, then retrying but this did not help.  I don't know if this will help, but I'm attaching the logs to this service. The service reads a CSV and runs a Google Earth Engine script. Once the script creates an Earth Engine task, the Cloud Run instance should shut down.

[Andres Fiesco Casasola]

Could you please post the configuration of your pub/sub and cloud run? What type of trigger are you using?

[Adriano Matos]

Sure thing. I'm supplying two .zip files that contain screenshots of all my configurations for pub/sub and cloud run. As for the trigger, I am using Pub/Sub notifications from Cloud Storage. In other words, whenever a new file gets added to a bucket, it triggers my Cloud Run instance by sending a message to the subscription (link to how this works here). I only use the OBJECT_FINALIZE event type, which only triggers the Cloud Run instance when a new file is successfully added/created in the bucket.

[Tracy Hall]

General Advice:

Post Code, not screen shots.  We shouldn't have to re-type it if we can help correct it.

*IF* (big if) it's truly a visual thing  - NOT console logs or code - then paste in an image, not attach a file.  Do you really think we can trust you that much?

Tracy Hall

LeadDreamer

[Adriano Matos]

Sorry about this :(

It's not code. Just the settings for my Cloud Run and Pub/Sub configurations. I just wanted to save time instead of writing everything down. Here it is again in text form.

Cloud Run Configuration:

General:

Container port:  8080

CPU allocation and pricing:

CPU is always allocated

Capacity:

Memory: 512 MB

CPU: 1

Request timeout: 900

Maximum requests per container: 80

Execution Environment 

Default

Autoscaling:

Minimum number of instances: 1

Maximum number of instances: 5

Pub / Sub Configuration (there are no settings to change in the topic so I will just show my subscription settings)

Delivery Type: Push

Message retention duration: 7 days

Expiration period: 31

Acknowledgement deadline: 10

Retry policy: retry immediately

If I skipped any settings in pub/sub, it is because I do not have them enabled. 

[Ernesto Contreras Pinon]

Which type of event is your Pub/Sub topic listening for? Is your Cloud Run service interacting with the same bucket in a way that could trigger subsequent Pub/Sub notifications in an infinite loop? That could be a likely cause for an infinite loop happening in your application, as shown in this example.

[Adriano Matos]

Something I never thought about. Thank you for the input!

The event type is OBJECT_FINALIZE. So once an item reaches a bucket, my Cloud Run instance will download the file and work with the file inside of the instance to generate an output that is uploaded into an entirely separate bucket. The Cloud Run instance should in theory shut down after the upload is complete, but it doesn't. Looking at my code, I don't see any way I am triggering an endless loop. The only time I interact with the bucket that is configured with pub/sub is when I download the file to my Cloud Run instance, which I believe should not endlessly trigger more Cloud Run instances as I have multiple other examples of Cloud Run instances that do this successfully. I will investigate this further to be certain. I can supply the code in a zip file if this would help?

[Adriano Matos]

Update:

I don't see anything that could be causing the loop except for the code itself. Again, this code works fine on my local machine. I was able to stop the loop by commenting out a majority of my code and slowly uncommenting line-by-line and rebuilding the service until the looping occurred. This is the main function I am running and it will starting looping endlessly at the last line fia_res = treemap_rast.projection().nominalScale()

If I were to remove that line, it doesn't loop at all. 

import ee, time, json

ee.Initialize(credentials)

def treemap_extract(xmin, ymin, xmax, ymax, outfile_name, _OUT_BUCKET):
    #################################
    # INPUT EARTH ENGINE PARAMETERS #
    treemap_rast = ee.Image("projects/calfuels/assets/FIA/california_c2014_tree_list")
    dem_raster = ee.Image("USGS/3DEP/10m")
    #boundaries = ee.FeatureCollection("projects/calfuels/assets/FIA/Plumas_TreeMap_Extraction/Plumas_National_forest")
    boundaries = ee.FeatureCollection("projects/calfuels/assets/Test_TreeMap_Extraction/test_area")
    location_code_raster = ee.Image("projects/calfuels/assets/FVSLocCode")
    variant_raster = ee.Image("projects/calfuels/assets/FVSVariant")
    #################################
   
    # Spatial resolution of the FIA raster
    fia_res = treemap_rast.projection().nominalScale()

[Adriano Matos]

Are there any rules pertaining to running Earth Engine code in a Cloud Run instance that I need to know?

[Adriano Matos]

Update:

It looks like this is occurring with any Cloud Run service I create, so it has nothing to do with Earth Engine. Just something I am doing wrong with every build. I'm posting some logs of what I am seeing. This Cloud Run instance prints the message from the Pub/Sub I have configured with the service, but I noticed immediately after printing the message, it looks like another Cloud Run instance gets booted up and it prints the exact message again. I guess this has something to do with the way I'm handling my Pub / Sub? That is where I am going to investigate next.

2022-05-20 09:53:18.672 PDT

[2022-05-20 16:53:18 +0000] [1] [INFO] Starting gunicorn 20.1.0

Default

2022-05-20 09:53:18.674 PDT

[2022-05-20 16:53:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

Default

2022-05-20 09:53:18.674 PDT

[2022-05-20 16:53:18 +0000] [1] [INFO] Using worker: gthread

Default

2022-05-20 09:53:18.692 PDT

[2022-05-20 16:53:18 +0000] [3] [INFO] Booting worker with pid: 3

Default

2022-05-20 09:53:23.039 PDT

{'attributes': {'bucketId': '1-create-tiles-ps-trigger', 'eventTime': '2022-05-20T16:53:16.212759Z', 'eventType': 'OBJECT_FINALIZE', 'notificationConfig': 'projects/_/buckets/1-create-tiles-ps-trigger/notificationConfigs/3', 'objectGeneration': '1653065596204513', 'objectId': 'Test_Area.zip', 'payloadFormat': 'JSON_API_V1'}, 'data': 'ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN0IiwKICAiaWQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9UZXN0X0FyZWEuemlwLzE2NTMwNjU1OTYyMDQ1MTMiLAogICJzZWxmTGluayI6ICJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9zdG9yYWdlL3YxL2IvMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9vL1Rlc3RfQXJlYS56aXAiLAogICJuYW1lIjogIlRlc3RfQXJlYS56aXAiLAogICJidWNrZXQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlciIsCiAgImdlbmVyYXRpb24iOiAiMTY1MzA2NTU5NjIwNDUxMyIsCiAgIm1ldGFnZW5lcmF0aW9uIjogIjEiLAogICJjb250ZW50VHlwZSI6ICJhcHBsaWNhdGlvbi96aXAiLAogICJ0aW1lQ3JlYXRlZCI6ICIyMDIyLTA1LTIwVDE2OjUzOjE2LjIxMloiLAogICJ1cGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInN0b3JhZ2VDbGFzcyI6ICJTVEFOREFSRCIsCiAgInRpbWVTdG9yYWdlQ2xhc3NVcGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInNpemUiOiAiMTAwMyIsCiAgIm1kNUhhc2giOiAiUmNyYWdiZUczWXIreE9XM3BBdStCQT09IiwKICAibWVkaWFMaW5rIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2Rvd25sb2FkL3N0b3JhZ2UvdjEvYi8xLWNyZWF0ZS10aWxlcy1wcy10cmlnZ2VyL28vVGVzdF9BcmVhLnppcD9nZW5lcmF0aW9uPTE2NTMwNjU1OTYyMDQ1MTMmYWx0PW1lZGlhIiwKICAiY3JjMzJjIjogIkpzanc5QT09IiwKICAiZXRhZyI6ICJDT0hybTUzRjd2Y0NFQUU9Igp9Cg==', 'messageId': '4634573438147974', 'message_id': '4634573438147974', 'publishTime': '2022-05-20T16:53:16.425Z', 'publish_time': '2022-05-20T16:53:16.425Z'}

Default

2022-05-20 09:53:28.949 PDT

[2022-05-20 16:53:28 +0000] [1] [INFO] Starting gunicorn 20.1.0

Default

2022-05-20 09:53:28.951 PDT

[2022-05-20 16:53:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

Default

2022-05-20 09:53:28.951 PDT

[2022-05-20 16:53:28 +0000] [1] [INFO] Using worker: gthread

Default

2022-05-20 09:53:28.966 PDT

[2022-05-20 16:53:28 +0000] [3] [INFO] Booting worker with pid: 3

2022-05-20 09:53:33.219 PDT

{'attributes': {'bucketId': '1-create-tiles-ps-trigger', 'eventTime': '2022-05-20T16:53:16.212759Z', 'eventType': 'OBJECT_FINALIZE', 'notificationConfig': 'projects/_/buckets/1-create-tiles-ps-trigger/notificationConfigs/3', 'objectGeneration': '1653065596204513', 'objectId': 'Test_Area.zip', 'payloadFormat': 'JSON_API_V1'}, 'data': 'ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN0IiwKICAiaWQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9UZXN0X0FyZWEuemlwLzE2NTMwNjU1OTYyMDQ1MTMiLAogICJzZWxmTGluayI6ICJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9zdG9yYWdlL3YxL2IvMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9vL1Rlc3RfQXJlYS56aXAiLAogICJuYW1lIjogIlRlc3RfQXJlYS56aXAiLAogICJidWNrZXQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlciIsCiAgImdlbmVyYXRpb24iOiAiMTY1MzA2NTU5NjIwNDUxMyIsCiAgIm1ldGFnZW5lcmF0aW9uIjogIjEiLAogICJjb250ZW50VHlwZSI6ICJhcHBsaWNhdGlvbi96aXAiLAogICJ0aW1lQ3JlYXRlZCI6ICIyMDIyLTA1LTIwVDE2OjUzOjE2LjIxMloiLAogICJ1cGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInN0b3JhZ2VDbGFzcyI6ICJTVEFOREFSRCIsCiAgInRpbWVTdG9yYWdlQ2xhc3NVcGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInNpemUiOiAiMTAwMyIsCiAgIm1kNUhhc2giOiAiUmNyYWdiZUczWXIreE9XM3BBdStCQT09IiwKICAibWVkaWFMaW5rIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2Rvd25sb2FkL3N0b3JhZ2UvdjEvYi8xLWNyZWF0ZS10aWxlcy1wcy10cmlnZ2VyL28vVGVzdF9BcmVhLnppcD9nZW5lcmF0aW9uPTE2NTMwNjU1OTYyMDQ1MTMmYWx0PW1lZGlhIiwKICAiY3JjMzJjIjogIkpzanc5QT09IiwKICAiZXRhZyI6ICJDT0hybTUzRjd2Y0NFQUU9Igp9Cg==', 'messageId': '4634573438147974', 'message_id': '4634573438147974', 'publishTime': '2022-05-20T16:53:16.425Z', 'publish_time': '2022-05-20T16:53:16.425Z'

[Adriano Matos]

Figured out the problem. Increasing the acknowledgement deadline in the Pub/Sub settings and increasing resources in my Cloud Run instances solved it. Never realized that the time it was taking to complete a task was exceeding said deadline.

[Damian Akpan]

I might be too late. But, you can configure a dead letter topic to the subscription. It basically catches any error while the subscriber handled a message. That way, you define max retries then it can failover to that topic.