Getting StatusCode.Unavailable messages when subscribing from a Google Cloud instance

Beekey Cheung

unread,

Oct 18, 2017, 7:21:18 PM10/18/17

to Google Cloud Pub/Sub Discussions

Today I started getting a lot of these messages when trying to subscribe from Google Pub Sub:

Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 248, in _blocking_consume
self._policy.on_exception(exc)
File "/usr/local/lib/python3.5/dist-packages/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 135, in on_exception
raise exception
File "/usr/local/lib/python3.5/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 234, in _blocking_consume
for response in response_generator:
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 348, in __next__
return self._next()
File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 342, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>

This only happens when attempting to run the consumer on a Google Cloud instance. I attempted running this on my development box and managed to receive messages from the topic.

This was working yesterday and the instance on Google Cloud hasn't been modified since then.

Restarting the process does not help.

Is there any other information I can provide?

Kir Titievsky

unread,

Oct 19, 2017, 11:02:27 AM10/19/17

to Beekey Cheung, Google Cloud Pub/Sub Discussions

Beekey, Did anything change with your firewall settings by chance?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/b45c416f-9d26-4b2f-bf03-c35cd16387cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

Simon D. Kim

unread,

Oct 19, 2017, 4:58:55 PM10/19/17

to Google Cloud Pub/Sub Discussions

We are also experiencing this issue with the below error log since yesterday:

Oct 19, 2017 8:51:02 PM com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1 onFailure

WARNING: Terminated streaming with exception

io.grpc.StatusRuntimeException: UNAVAILABLE: The service was unable to fulfill your request. Please try again. [code=8a75]

at io.grpc.Status.asRuntimeException(Status.java:526)

at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)

at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)

at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)

at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)

at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)

at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/b45c416f-9d26-4b2f-bf03-c35cd16387cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kir Titievsky

unread,

Oct 19, 2017, 5:10:42 PM10/19/17

to Simon D. Kim, Google Cloud Pub/Sub Discussions

Simon, Beekey: we expect these errors occasionally when bidirectional connections are broken. The client library would try to rebuild the connection. Are you not able to consume any messages at all? Could you give me an idea of the rate at which you are seeing these?

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/b45c416f-9d26-4b2f-bf03-c35cd16387cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/821ddc45-abbc-470c-a53a-07cf7f165c69%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Simon D. Kim

unread,

Oct 19, 2017, 5:15:12 PM10/19/17

to Kir Titievsky, Google Cloud Pub/Sub Discussions, j...@bitfinder.co, Charlie Lee

Hi Kir.

Thanks for your response.

As far as we know, we are not able to consume any message at all at an application that shows this behavior. It seems like the rate at which this happens for us is 1~2 minutes as shown below:

Oct 19, 2017 8:55:35 PM com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1 onFailure