PubSub Pull Subscriber stops receiving messages randomly

[Roxana Ioana Mirel]

Hello,

I have a GKE cluster with three nodes, which are being subscribers to the same PubSub subscription 

and polling continuously for new messages.  

I am using the Asynchronous Pull mechanism for my subscribers. These are running in a NodeJS Express application.

subscription.on("message", message => messageHandler(message));

async function messageHandler(message) {

  try {

    await doSomeProcessing();

    message.ack();

  } catch (error) {

    if (error instanceof DuplicateRowError) {

      message.ack();

    } else {

      console.log(`Message ${message.id} was not acknowledged. Waiting to be sent again.`);

    }

  }

}

Things go well most of the time, but from time to time the subscribers stop receiving messages. Redeploying the application doesn't help and they recover by themselves after aprox 30min- 1 hour, when the pending messages are being sent. The quota has not been exceeded and subscription.on("error", error ..) does not get triggered, the StackDriver logs also don't show any errors.

Any idea what might cause this pb?

[Sam (Google Cloud Support)]

Hi Roxanne,

It seems like you are dealing with message flow control issue [1], where the subscriber client might be slower in processing and acknowledging messages than Cloud Pub/Sub is in sending them. In your case, this might have led to a backlog of messages to one client because it doesn’t have the capacity to handle the influx of messages, thereby, being unable to send its messages quickly enough to the second client it gets stuck.

To resolve this, I suggest that you use the flow control feature of the subscriber to control the rate at which it receives messages. Follow the example provided in this documentation to try out the flow control feature [1]. You can also reference this troubleshooting guide for future reference [2][3]. Hope that helps. Let me know how that works for you.

[1] https://cloud.google.com/pubsub/docs/pull#message-flow-control

[2] https://cloud.google.com/pubsub/docs/troubleshooting#create

[3] https://cloud.google.com/pubsub/docs/push#app-engine-standard-endpoints

[Roxana Ioana Mirel]

Hi! 

Thank you for your reply. 

I was thinking it can be a flow control issue, so I've been logging the time between the arrival of the message at the subscriber and the time the subscriber acknowledges it and that is usually not more than 1 second. Nevertheless, I'll try the flow control is the problem persists. 

Thanks!