io.grpc.StatusRuntimeException: UNAVAILABLE: The service was unable to fulfill your request. Please

[Manoj Santhakumaran]

Hello All,

Recently I started integrating Google Pub/Sub Cloud to one of the applications, where in my application is a subscriber. At first everything worked great.
When I moved to UAT and deployed in UAT Vms, I started seeing below error time-to-time.
Most of the times application is able to recover from this error after sometime. But if this error is coming at the time of starting up of JVM, I noticed that the JVM got exited.
If you have any solution/suggestions/clue, please help me out. Any help is much appreciated!!

Oct 27, 2017 2:41:04 AM com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1 onFailure
WARNING: Terminated streaming with exception
io.grpc.StatusRuntimeException: UNAVAILABLE: The service was unable to fulfill your request. Please try again. [code=8a75]
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

java -version
openjdk version "1.8.0_102"
OpenJDK Runtime Environment (build 1.8.0_102-b14)
OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)

uname -a
Linux <hostname> 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

Thank you very much.
Manoj

[George (Cloud Platform Support)]

Hello Manoj, 

In such situations it is recommendable to implement exponential back off for your client, as documented on the "Truncated Exponential Backoff" page. 

You may check the solution proposed in the "Pub/Sub StreamingSubscriberConnection : Terminated streaming with exception" github thread, quite relevant in your case. 

[Kir Titievsky]

Manoj,

Thanks for the message.  George is right, but hold off on taking action -- we want to look deeper into whether this is a client library bug.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/93dd6394-f286-4412-afe1-ec5e1fa4379e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Manoj Santhakumaran]

Hi George, Kir,

Thank you so much for your quick reply. Much appreciated!!

BTW, I'm using "google-cloud-pubsub" version "0.24.0-beta". Now I see that the latest version is "0.26.0-beta", which I'm going to test over this weekend.

As George pointed out, I checked the "Truncated exponential backoff", isn't it implicit in Java client library?

[Kir Titievsky]

Manoj, yes -- this should already be handled by the client library.  I'll plan to either post an updated to this thread or a bug on GitHub repo.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e1293ed1-6e68-4c9e-8f74-9f467d84a12c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kir Titievsky]

Manoj, I've confirmed that 0.26.0-beta of the library has improved support for retries in this kind of scenario. Would you be so kind as to update and report back on whether the problem persists with the new version?

[Manoj Santhakumaran]

Hi Kir,

Thank you very much for your update !! Much appreciated !!

BTW, I tested with "0.26.0-beta" over the weekend and my finding is that this exception is no longer happening with the latest version.
And I'm happy that my application logs look very clean after this upgrade :-)

So in short, as you said the latest version "0.26.0-beta" had the fixes for my issue reported earlier.

Once again thank you very much for your prompt replies!!
Manoj

On Wednesday, 1 November 2017 08:51:32 UTC-7, Kir Titievsky wrote:

Manoj, I've confirmed that 0.26.0-beta of the library has improved support for retries in this kind of scenario. Would you be so kind as to update and report back on whether the problem persists with the new version?