Duplicate PubSub message with different message id

Tinshuk Singh

unread,

Feb 18, 2021, 7:46:16 AM2/18/21

to Google Cloud Pub/Sub Discussions

Hi Team,

I have a publisher which published a message to the topic however the same message was received multiple times to subscriber with the different pubsub message id. Checked all possible scenario where publisher can send the same message multiple times but that's not the case. Can someone please confirm if this is an expected edge scenario behavior of pubsub?

Thanks,

Tinshuk

Ben Warrick

unread,

Feb 18, 2021, 7:59:32 AM2/18/21

to Tinshuk Singh, Google Cloud Pub/Sub Discussions

Three scenarios come to mind. First, PubSub is an at-least-once delivery system, so it can deliver a message more than once. Second, I would ask how you're acknowledging the messages, but since the message ID is different, that's probably not this issue.

So if the message ID is different, that makes me think maybe the publisher legitimately created multiple messages.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/05fab618-b829-415e-badc-966ae834a4fen%40googlegroups.com.

Kir Titievsky

unread,

Feb 18, 2021, 9:27:52 AM2/18/21

to Ben Warrick, Tinshuk Singh, Google Cloud Pub/Sub Discussions

Tinshuk,

To add to Ben's response: Because you are seeing the same message with different IDs, it means it was published by the client multiple times. This will typically happen if the success response of a publish request is not properly handled (e.g. the callback is never invoked) or if the publish request terminates with a deadline_exceeded error or when a client reaches the maximum request time (configured by the client). In the deadline_exceeded case the service may have responded to you with a "success" but the response got stuck somewhere in the network.

To confirm, look at the request error rates in Monitoring. You can also force this behavior to be handled more explicitly on the client by changing the retry settings. You may want to set very long deadlines if your network connection is struggling so that the client does not terminate the connection prematurely. In the end, it's not possible to eliminate such duplication entirely but you can certainly reduce the rate.

I'd love to find out about your use case here. If you'd be so kind, you can respond to me directly.

Kir, Product Manager, Pub/Sub

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/CAJyQTEu4%3Dummh8R7yzLMPudou_Yi-_V4vvTgZph%2B4nW8NbGAcw%40mail.gmail.com.