GCP Pub/Sub Pull api median latency suddenly increase.

[Peng Yu]

Hello,

So i'm experiencing some issue with Pub/Sub. I'm using Dataflow to pull from pub/sub and doing some data processing and write to another pubsub.

The job works fine for a few hours and suddenly went on stale. Meanwhile from the google api monitoring metrics, the pubsub.subscribe.pull operation is experiencing significant delay. median delay jumps to 20+ seconds. In the api quota section, there are some exceed api quota error, and the detailed error info shows current usage green and 7 day usage exceeded.  I tried to specify a different region in my dataflow job, which runs into the same issue(pull latency jumps to 20+ second). I'm wondering what could be the real issue. 

Is it really just api quota? If it is, then how is it computed, if i throttle my consumer, would the problem go away? 

[Samar Amara]

Hi

Google Groups are for general discussion. For individual issue troubleshooting, please open a support ticket or an issue on Public Issue Tracker https://cloud.google.com/support/docs/issue-trackers providing the steps to reproduce for our investigation.

[Kir Titievsky]

Peng,  I'm with Samar on this -- given that this is a relatively complex and specific thing this should be a support issue.  Some general things to look at:

1. Are you building up a backlog while this is happening? It's possible that pull requests hang because there is no data to send.  I

2. Are you seeing a log of expired acknowledgements? This could cause Pub/Sub to slow message delivery. 

You can find more details here: https://cloud.google.com/pubsub/docs/monitoring

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e43f0917-9979-4c8e-8009-0197be10bc8en%40googlegroups.com.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Kir Titievsky]

This blog post is also very handy in diagnosing subscriber latency issues: https://medium.com/google-cloud/google-cloud-pub-sub-how-to-monitor-the-health-of-your-subscription-for-optimal-end-to-end-latency-540b868e7929