Google PubSub: Some messages are massively delayed

Klaus Gamauf

unread,

Jul 13, 2018, 9:05:04 AM7/13/18

to Google Cloud Pub/Sub Discussions

Hey,

we are using PubSub and we are quite happy with it except for one thing: Some messages that we publish in PubSub are not delivered to any consumer for hours (once we had a case where messages were delayed for more than 24h). To be precise I'm talking about the "Pub/Sub Subscription - Oldest Unacknowledged Message" metric (see attached screenshots).

As you can see in the attached screenshots, the age of the oldest unacknowledged message increases, while the number of messages in PubSub is more or less stable (this is especially good visible when looking at the last hour only) - the insertion rate into PubSub at the time of the screenshot was ~250 messages per second.

To give you some context on how we use PubSub: We publish messages through a nodejs application and consume them on the other end again in a nodejs application using a pull subscription, both the publish and the consume part are autoscaled instance groups running in the same project in GCE.

Any idea what this could be caused by? Any idea what we could do to fix this? Could it be a bug in PubSub itself?

Thanks,

Klaus

stackdriver.1h.png

stackdriver.1h.unack.png

stackdriver.png

stackdriver.unack.png

Katayoon (Cloud Platform Support)

unread,

Jul 13, 2018, 4:51:48 PM7/13/18

to Google Cloud Pub/Sub Discussions

Hi Klaus,

I recommend that you create a private case in the Issue Tracker and provide us your Project ID, Topic Name, Subscriber ID there, so that we would be able to dig into the issue.

Klaus Gamauf

unread,

Jul 25, 2018, 3:39:58 PM7/25/18

to Google Cloud Pub/Sub Discussions

Hey,

thanks for you help. I created a issue in the tracker - looking forward to getting this resolved.

Best regards,

Klaus

Kir Titievsky

unread,

Jul 26, 2018, 3:29:10 PM7/26/18

to Klaus Gamauf, Google Cloud Pub/Sub Discussions

Klaus, thanks for the email and submitting the support case. Please let me know on this thread if this is not resolved in the next few days. There are many reasons this kind of thing might happen so it might be worth for us to work though this live.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/c57af4e9-b8c4-48cf-89c5-467777c0f7a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.