Issues with multiple messages being published to same pub sub topic

74 views

Skip to first unread message

A Prabin Sarab

unread,

Dec 10, 2021, 10:51:03 AM12/10/21

to pubsub-discuss

Hi
I am trying to publish more than 50k messages to a topic within 2-3 mins, but on the subscriber side, it's being duplicated with a different message publish ids. Message content is the same but it's getting listened to multiple times on the subscriber side.

Instead of 50k messages, I have got around 7 lakh messages on the subscriber side.

Waiting for any suggestions to solve this problem.

Kir Titievsky

unread,

Dec 10, 2021, 11:14:30 AM12/10/21

to A Prabin Sarab, pubsub-discuss

Hi. A possible explanation of this is that your publisher calls are failing frequently and your client retries the publish requests. I can't say much without seeing your code and monitoring, but a first sanity check here would be to make sure you are handling Publish time outs explicitly to detect this problem (see samples here). A common cause of timeouts like this is that you are publishing too fast. Try your publisher code with flow control settings. As a rule of thumb, a very safe starting point would be limiting your publisher to 10MBps.

--
You received this message because you are subscribed to the Google Groups "pubsub-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/3cdb94ae-e6ef-4d72-b195-d24f27c25cedn%40googlegroups.com.