Queueing Pub/Sub Messages for Cloud Functions

Khari Kisile

unread,

Nov 23, 2020, 11:22:54 AM11/23/20

to Google Cloud Pub/Sub Discussions

I am using Pub/Sub messages to trigger cloud functions. The pub/sub messages are triggered by cloud storage uploads. Each cloud function then starts a Compute Engine instance. I am using Pub/Sub because of its queueing capabilities. I want to limit the number of instances that are started at a time by each pub/sub message. If the instance limit is reached, I want to queue the incoming pub/sub messages to wait until the instance limit is no longer being hit. Is this possible with Pub/Sub?

Jun Lu

unread,

Nov 26, 2020, 8:09:23 PM11/26/20

to Google Cloud Pub/Sub Discussions

Hi,

Google Cloud Pub/Sub does not implement any logic for delayed delivery, and messages are delivered as fast as possible [1].

Since Cloud Functions will be triggered by messages published to Pub/Sub topics in the same Cloud project as the function [2], it does not seem that you can queue the publishing messages to the topic for your purpose.

[1] https://blog.doit-intl.com/deduplication-delayed-messaging-and-fifo-with-pub-sub-b4b4373820a9#5fd6

[2] https://cloud.google.com/functions/docs/calling/pubsub

Kir Titievsky

unread,

Jan 27, 2021, 1:48:41 PM1/27/21

to Khari Kisile, Google Cloud Pub/Sub Discussions

Khari, There is nothing in Pub/Sub itself that can help you directly. You'll need the function itself to know how many instances are running and whether it is allowed to create any more. You can either do this by calling the GCE API or keeping and querying a counter in say Datastore. Once a function knows it has exceeded the quota it can either sleep and retry for up to 9 minutes or sleep and fail the call for Pub/Sub to re-delivery the message and cause a retry.

On Mon, Nov 23, 2020 at 11:22 AM Khari Kisile <khari....@gmail.com> wrote:

I am using Pub/Sub messages to trigger cloud functions. The pub/sub messages are triggered by cloud storage uploads. Each cloud function then starts a Compute Engine instance. I am using Pub/Sub because of its queueing capabilities. I want to limit the number of instances that are started at a time by each pub/sub message. If the instance limit is reached, I want to queue the incoming pub/sub messages to wait until the instance limit is no longer being hit. Is this possible with Pub/Sub?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/abd953cc-9867-4e2b-83d5-210382afb814n%40googlegroups.com.