Google Cloud Pub/Sub Multiple Notifications for same object

[Heta Saraiya]

Hi,

I am new to gcp and not sure if there is a way to do this so I need some suggestions.

I currently have a system in which for a topic I receive notifications when a new object is created in some folder. After this I have a cloud function which will get triggered when a new notification is generated in that topic. This function gets data from object and changes multiple object in cloud storage.

The issue is I am getting multiple notifications for same object and this will create multiple instances of cloud functions. This is not feasible for my system. I cannot have more than one instance of cloud function operating on same object as it will be changing other object and will create race conditions. 

I am looking for a way to get just one notification for one object/ acknowledge the topic that the notification is received don't send it again. The other option is to trigger directly from object storage using cloud storage trigger but the problem is that this will trigger for all objects in bucket rather than just for objects in specific folder.

I am open to all ideas.

Thank you for your help,

Heta

[George (Cloud Platform Support)]

This is to be expected. Your functions are typically invoked once for each incoming event. However, Cloud Functions does not guarantee a single invocation in all cases because of differences in error scenarios. You may find related detail on the "Cloud Functions Execution Environment" documentation page.

Cloud Pub/Sub assigns a unique `message_id` to each message, which can be used to detect duplicate messages received by the subscriber. You may implement your own solution, for instance save message_id in Memcache, then, prior to processing each message, you may check that its message_id is different from the ones on the list.

[Heta Saraiya]

The 'message_id' is not available in cloud function payload

Cloud Functions triggered from a Cloud Pub/Sub topic will be sent events conforming to the PubsubMessage type, with the caveat that publishTime and messageId are not directly available in the PubsubMessage. Instead, you can access publishTime and messageId via the event ID and timestamp properties of the event metadata. 

 So can the id in that be used (will it be always unique for same message)?

On Wednesday, August 21, 2019 at 10:18:29 AM UTC-7, George (Cloud Platform Support) wrote:

This is to be expected. Your functions are typically invoked once for each incoming event. However, Cloud Functions does not guarantee a single invocation in all cases because of differences in error scenarios. You may find related detail on the "Cloud Functions Execution Environment" documentation page.

Cloud Pub/Sub assigns a unique `message_id` to each message, which can be used to detect duplicate messages received by the subscriber. You may implement your own solution, for instance save message_id in Memcache, then, prior to processing each message, you may check that its message_id is different from the ones on the list.