Cloud Functions Triggers for Pub/Sub

[Roxana Ioana Mirel]

Hello, 

I have a question regarding the Cloud Function triggers for Pub/Sub. 

The Pub/Sub ensures at least once delivery, but doesn't guarantee that a message is sent only once, in some cases it might not: "Typically, Cloud Pub/Sub delivers each message once and in the order in which it was published. However, messages may sometimes be delivered out of order or more than once." 

Was the Cloud Function Pub/Sub Trigger implemented so that it deals with that? Couldn't find that information, only something about Dataflow Pub/Sub IO: "You can achieve exactly once processing of Cloud Pub/Sub message streams using Cloud Dataflow PubsubIO. PubsubIO de-duplicates messages on custom message identifiers or those assigned by Cloud Pub/Sub."

Thanks!

[George (Cloud Platform Support)]

Delivery is one type of event that may be used to trigger a cloud function. There are others, independent of the duplicate delivery aspect. How would you like the trigger to be implemented? Would you like a message to be sent once, even if delivery happened actually multiple times? More detail on the considered setup that your question refers to would help make things clearer. 

[Roxana Ioana Mirel]

The Cloud Function is triggered when a message is published to a Pub/Sub topic so event is: google.pubsub.topic.publish. I'd like it to be triggered once even if the delivery happened more times? Would I have to take care of this idempotence?

[George (Cloud Platform Support)]

Publishing occurs once, delivery possibly repeated. There is no reason to worry about multiple publishing. Which other events would be of interest for your app?

[Laurens Vijnck]

Dear George,

Does this imply that we have to implement our own de-duplication logic in the Cloud Function? For my case, i'm using a Cloud Function to merge a partial result calculated by Dataflow, with an already existing entry in Datastore. If it is possible that the Cloud Function is triggered twice, due to the at-least-once semantics, I'll be combining the partial result twice thereby throwing correctness out of the window. Insights are very much appreciated.

Op dinsdag 8 mei 2018 18:49:22 UTC+2 schreef George (Cloud Platform Support):

[yka...@google.com]

Hello Laurens,

As mentioned here, background functions are invoked at least once. The way Functions is set up internally is a "at least once" delivery system meaning you should expect duplicate invocations. Multiple attempts could be made to execute the function. So to answer your concern, it would be something you have to guard against in your implementation. Here are some tips on making functions idempotent.