What causing unreachable errors in Google PubSub?

[Alexander Andryichenko]

i'm running application which consists of Google Cloud Functions, triggered by PubSub Topics, so basically theu're communicating to each other via Google PubSub.

The problem is, it can struggle sometimes and show delays when publishing message up to 9s or more. I checked the Metrics Explorer and found out that when high delays it shows errors like:

 - unreachable_5xx_error_500 

 - unreachable_no_response

 - internal_rejected_error 

 - unreachable_5xx_error_503 

 - url_4xx_error_429

Code example of publishing message inside Google Cloud Function:

 const {PubSub} = require('@google-cloud/pubsub');

 const pubSubClient = new PubSub();

 async function publishMessage() {

    const topicName = 'my-topic';

    const dataBuffer = Buffer.from(data);

    const messageId = await pubSubClient.topic(topicName).publish(dataBuffer);

    console.log(`Message ${messageId} published.`);

 }

 publishMessage().catch(console.error);

Code example of function triggered by PubSub Topic:

exports.subscribe = async (message) => {

  const name = message.data

    ? Buffer.from(message.data, 'base64').toString()

    : 'World';

  console.log(`Hello, ${name}!`);

}

And i think this errors causing delays. I didn't find anything on this on the internet, so i hope you can explain what causing this errors and why and probably can help with this.

[Abdel (Cloud Platform Support)]

The 5xx error message implies that Cloud Functions failed to handle the push request from Pub/Sub.

1. You CF has a configuration option called ´Maximum function instances´ (which can be found by navigating to CF > Advanced Options) which is the max # number of function instances that can run concurrently. If Pub/Sub is trying to invoke more instances than allowed (limited by this upper bound) CF will return a 5xx (i.e. server error) back to Pub/Sub. The workaround is to set  ´Maximum function instances´ to 0 (or leave empty) which will allow your function instances to scale upwards as needed!

2. CF ´PS trigger´ has a unique configuration (not available in other triggers such as HTTP) called ´Retry on failure´ (can be found under Advanced Options.) When enabled, ´Retry on failure´ will retry failed function runs up to and including 7 days. Should there be any momentary downtime from CF, the function will be retried and once the retry is successful, the client (PS) will get a 2xx response. Pub/Sub won´t even known about the server downtime since no 5xx will be returned at all!

The above 2 tweaks should address your issue. Let me know if you have any additional questions.

[Alexander Andryichenko]

1. All ´Maximum function instances´ is set to 0 or unset (handled by Google)

2. Retry on failure flag is also set

Still facing same issues.

[Gurkomal (Cloud Platform Support)]

Hi Alexander,

unreachable_5xx_error_500 and url_4xx_error_429 seems to be messages returned by the endpoint and Pub/Sub is just relaying this message back. We would recommend opening a Cloud Functions Public Issue as this does not seem to be an issue within Cloud Pub/Sub.

[Nick Pomfret]

Alexander.  I'm having a similar issue: https://github.com/googleapis/nodejs-pubsub/issues/959

https://issuetracker.google.com/issues/154260226