Cloud Run w/ pub/sub integration never stops running

1,314 views
Skip to first unread message

Adriano Matos

unread,
May 13, 2022, 3:14:16 AM5/13/22
to Google Cloud Developers
I have a Cloud Run service integrated with pub/sub so each time a new file gets added to a bucket, it triggers my Cloud Run service. For some reason, when a push a single file into a bucket, the Cloud Run service is able to perform operations successfully (returning a POST 204), but it runs on the same file endlessly on a loop. I tried purging all my messages in my service's subscription, then retrying but this did not help.  I don't know if this will help, but I'm attaching the logs to this service. The service reads a CSV and runs a Google Earth Engine script. Once the script creates an Earth Engine task, the Cloud Run instance should shut down.
downloaded-logs-20220512-160105.json

Andres Fiesco Casasola

unread,
May 13, 2022, 7:03:20 PM5/13/22
to Google Cloud Developers

Could you please post the configuration of your pub/sub and cloud run? What type of trigger are you using?

Adriano Matos

unread,
May 17, 2022, 11:30:56 AM5/17/22
to Google Cloud Developers

Sure thing. I'm supplying two .zip files that contain screenshots of all my configurations for pub/sub and cloud run. As for the trigger, I am using Pub/Sub notifications from Cloud Storage. In other words, whenever a new file gets added to a bucket, it triggers my Cloud Run instance by sending a message to the subscription (link to how this works here). I only use the OBJECT_FINALIZE event type, which only triggers the Cloud Run instance when a new file is successfully added/created in the bucket.
PubSub.zip
Cloud Run.zip

Tracy Hall

unread,
May 17, 2022, 12:22:28 PM5/17/22
to Google Cloud Developers
General Advice:
Post Code, not screen shots.  We shouldn't have to re-type it if we can help correct it.

*IF* (big if) it's truly a visual thing  - NOT console logs or code - then paste in an image, not attach a file.  Do you really think we can trust you that much?

Tracy Hall
LeadDreamer

Adriano Matos

unread,
May 17, 2022, 12:52:08 PM5/17/22
to Google Cloud Developers
Sorry about this :(

It's not code. Just the settings for my Cloud Run and Pub/Sub configurations. I just wanted to save time instead of writing everything down. Here it is again in text form.

Cloud Run Configuration:

General:
Container port:  8080

CPU allocation and pricing:
CPU is always allocated

Capacity:
Memory: 512 MB
CPU: 1
Request timeout: 900
Maximum requests per container: 80

Execution Environment 
Default

Autoscaling:
Minimum number of instances: 1
Maximum number of instances: 5

Pub / Sub Configuration (there are no settings to change in the topic so I will just show my subscription settings)
Delivery Type: Push
Message retention duration: 7 days
Expiration period: 31
Acknowledgement deadline: 10
Retry policy: retry immediately

If I skipped any settings in pub/sub, it is because I do not have them enabled. 

Ernesto Contreras Pinon

unread,
May 17, 2022, 6:20:59 PM5/17/22
to Google Cloud Developers

Which type of event is your Pub/Sub topic listening for? Is your Cloud Run service interacting with the same bucket in a way that could trigger subsequent Pub/Sub notifications in an infinite loop? That could be a likely cause for an infinite loop happening in your application, as shown in this example.

Adriano Matos

unread,
May 18, 2022, 3:48:59 PM5/18/22
to Google Cloud Developers
Something I never thought about. Thank you for the input!

The event type is OBJECT_FINALIZE. So once an item reaches a bucket, my Cloud Run instance will download the file and work with the file inside of the instance to generate an output that is uploaded into an entirely separate bucket. The Cloud Run instance should in theory shut down after the upload is complete, but it doesn't. Looking at my code, I don't see any way I am triggering an endless loop. The only time I interact with the bucket that is configured with pub/sub is when I download the file to my Cloud Run instance, which I believe should not endlessly trigger more Cloud Run instances as I have multiple other examples of Cloud Run instances that do this successfully. I will investigate this further to be certain. I can supply the code in a zip file if this would help?

Adriano Matos

unread,
May 19, 2022, 5:56:49 PM5/19/22
to Google Cloud Developers
Update:

I don't see anything that could be causing the loop except for the code itself. Again, this code works fine on my local machine. I was able to stop the loop by commenting out a majority of my code and slowly uncommenting line-by-line and rebuilding the service until the looping occurred. This is the main function I am running and it will starting looping endlessly at the last line fia_res = treemap_rast.projection().nominalScale()

If I were to remove that line, it doesn't loop at all. 

import ee, time, json
ee.Initialize(credentials)

def treemap_extract(xmin, ymin, xmax, ymax, outfile_name, _OUT_BUCKET):
    #################################
    # INPUT EARTH ENGINE PARAMETERS #
    treemap_rast = ee.Image("projects/calfuels/assets/FIA/california_c2014_tree_list")
    dem_raster = ee.Image("USGS/3DEP/10m")
    #boundaries = ee.FeatureCollection("projects/calfuels/assets/FIA/Plumas_TreeMap_Extraction/Plumas_National_forest")
    boundaries = ee.FeatureCollection("projects/calfuels/assets/Test_TreeMap_Extraction/test_area")
    location_code_raster = ee.Image("projects/calfuels/assets/FVSLocCode")
    variant_raster = ee.Image("projects/calfuels/assets/FVSVariant")
    #################################
   
    # Spatial resolution of the FIA raster
    fia_res = treemap_rast.projection().nominalScale()


Adriano Matos

unread,
May 19, 2022, 5:58:02 PM5/19/22
to Google Cloud Developers

Are there any rules pertaining to running Earth Engine code in a Cloud Run instance that I need to know?

Adriano Matos

unread,
May 20, 2022, 1:08:14 PM5/20/22
to Google Cloud Developers
Update:

It looks like this is occurring with any Cloud Run service I create, so it has nothing to do with Earth Engine. Just something I am doing wrong with every build. I'm posting some logs of what I am seeing. This Cloud Run instance prints the message from the Pub/Sub I have configured with the service, but I noticed immediately after printing the message, it looks like another Cloud Run instance gets booted up and it prints the exact message again. I guess this has something to do with the way I'm handling my Pub / Sub? That is where I am going to investigate next.

2022-05-20 09:53:18.672 PDT
[2022-05-20 16:53:18 +0000] [1] [INFO] Starting gunicorn 20.1.0
Default
2022-05-20 09:53:18.674 PDT
[2022-05-20 16:53:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
Default
2022-05-20 09:53:18.674 PDT
[2022-05-20 16:53:18 +0000] [1] [INFO] Using worker: gthread
Default
2022-05-20 09:53:18.692 PDT
[2022-05-20 16:53:18 +0000] [3] [INFO] Booting worker with pid: 3
Default
2022-05-20 09:53:23.039 PDT
{'attributes': {'bucketId': '1-create-tiles-ps-trigger', 'eventTime': '2022-05-20T16:53:16.212759Z', 'eventType': 'OBJECT_FINALIZE', 'notificationConfig': 'projects/_/buckets/1-create-tiles-ps-trigger/notificationConfigs/3', 'objectGeneration': '1653065596204513', 'objectId': 'Test_Area.zip', 'payloadFormat': 'JSON_API_V1'}, 'data': 'ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN0IiwKICAiaWQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9UZXN0X0FyZWEuemlwLzE2NTMwNjU1OTYyMDQ1MTMiLAogICJzZWxmTGluayI6ICJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9zdG9yYWdlL3YxL2IvMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9vL1Rlc3RfQXJlYS56aXAiLAogICJuYW1lIjogIlRlc3RfQXJlYS56aXAiLAogICJidWNrZXQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlciIsCiAgImdlbmVyYXRpb24iOiAiMTY1MzA2NTU5NjIwNDUxMyIsCiAgIm1ldGFnZW5lcmF0aW9uIjogIjEiLAogICJjb250ZW50VHlwZSI6ICJhcHBsaWNhdGlvbi96aXAiLAogICJ0aW1lQ3JlYXRlZCI6ICIyMDIyLTA1LTIwVDE2OjUzOjE2LjIxMloiLAogICJ1cGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInN0b3JhZ2VDbGFzcyI6ICJTVEFOREFSRCIsCiAgInRpbWVTdG9yYWdlQ2xhc3NVcGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInNpemUiOiAiMTAwMyIsCiAgIm1kNUhhc2giOiAiUmNyYWdiZUczWXIreE9XM3BBdStCQT09IiwKICAibWVkaWFMaW5rIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2Rvd25sb2FkL3N0b3JhZ2UvdjEvYi8xLWNyZWF0ZS10aWxlcy1wcy10cmlnZ2VyL28vVGVzdF9BcmVhLnppcD9nZW5lcmF0aW9uPTE2NTMwNjU1OTYyMDQ1MTMmYWx0PW1lZGlhIiwKICAiY3JjMzJjIjogIkpzanc5QT09IiwKICAiZXRhZyI6ICJDT0hybTUzRjd2Y0NFQUU9Igp9Cg==', 'messageId': '4634573438147974', 'message_id': '4634573438147974', 'publishTime': '2022-05-20T16:53:16.425Z', 'publish_time': '2022-05-20T16:53:16.425Z'}
Default
2022-05-20 09:53:28.949 PDT
[2022-05-20 16:53:28 +0000] [1] [INFO] Starting gunicorn 20.1.0
Default
2022-05-20 09:53:28.951 PDT
[2022-05-20 16:53:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
Default
2022-05-20 09:53:28.951 PDT
[2022-05-20 16:53:28 +0000] [1] [INFO] Using worker: gthread
Default
2022-05-20 09:53:28.966 PDT
[2022-05-20 16:53:28 +0000] [3] [INFO] Booting worker with pid: 3
2022-05-20 09:53:33.219 PDT
{'attributes': {'bucketId': '1-create-tiles-ps-trigger', 'eventTime': '2022-05-20T16:53:16.212759Z', 'eventType': 'OBJECT_FINALIZE', 'notificationConfig': 'projects/_/buckets/1-create-tiles-ps-trigger/notificationConfigs/3', 'objectGeneration': '1653065596204513', 'objectId': 'Test_Area.zip', 'payloadFormat': 'JSON_API_V1'}, 'data': 'ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN0IiwKICAiaWQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9UZXN0X0FyZWEuemlwLzE2NTMwNjU1OTYyMDQ1MTMiLAogICJzZWxmTGluayI6ICJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9zdG9yYWdlL3YxL2IvMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlci9vL1Rlc3RfQXJlYS56aXAiLAogICJuYW1lIjogIlRlc3RfQXJlYS56aXAiLAogICJidWNrZXQiOiAiMS1jcmVhdGUtdGlsZXMtcHMtdHJpZ2dlciIsCiAgImdlbmVyYXRpb24iOiAiMTY1MzA2NTU5NjIwNDUxMyIsCiAgIm1ldGFnZW5lcmF0aW9uIjogIjEiLAogICJjb250ZW50VHlwZSI6ICJhcHBsaWNhdGlvbi96aXAiLAogICJ0aW1lQ3JlYXRlZCI6ICIyMDIyLTA1LTIwVDE2OjUzOjE2LjIxMloiLAogICJ1cGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInN0b3JhZ2VDbGFzcyI6ICJTVEFOREFSRCIsCiAgInRpbWVTdG9yYWdlQ2xhc3NVcGRhdGVkIjogIjIwMjItMDUtMjBUMTY6NTM6MTYuMjEyWiIsCiAgInNpemUiOiAiMTAwMyIsCiAgIm1kNUhhc2giOiAiUmNyYWdiZUczWXIreE9XM3BBdStCQT09IiwKICAibWVkaWFMaW5rIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2Rvd25sb2FkL3N0b3JhZ2UvdjEvYi8xLWNyZWF0ZS10aWxlcy1wcy10cmlnZ2VyL28vVGVzdF9BcmVhLnppcD9nZW5lcmF0aW9uPTE2NTMwNjU1OTYyMDQ1MTMmYWx0PW1lZGlhIiwKICAiY3JjMzJjIjogIkpzanc5QT09IiwKICAiZXRhZyI6ICJDT0hybTUzRjd2Y0NFQUU9Igp9Cg==', 'messageId': '4634573438147974', 'message_id': '4634573438147974', 'publishTime': '2022-05-20T16:53:16.425Z', 'publish_time': '2022-05-20T16:53:16.425Z'

Adriano Matos

unread,
Jun 9, 2022, 6:00:12 PM6/9/22
to Google Cloud Developers
Figured out the problem. Increasing the acknowledgement deadline in the Pub/Sub settings and increasing resources in my Cloud Run instances solved it. Never realized that the time it was taking to complete a task was exceeding said deadline.

Damian Akpan

unread,
Jul 13, 2022, 7:23:21 PM7/13/22
to Google Cloud Developers
I might be too late. But, you can configure a dead letter topic to the subscription. It basically catches any error while the subscriber handled a message. That way, you define max retries then it can failover to that topic.
Reply all
Reply to author
Forward
0 new messages