[grpc-java 0.14.0] gRPC calls not working after a certain period.

[Taehyun Park]

I'm using grpc java on Android and I found a very weird issue. After a certain period a ManagedChannel no longer works.

When this issue happens, grpc calls stopped working so I had to close and restart my app.

I instantiated a ManagedChannel when there is no cached channel then cache it until the number of active channels is 0. My app worked fine and didn't have a problem when it's launched. but all grpc calls stopped working after a certain period. The app wasn't closed but it was in a backstack.

I searched a similar issue in grpc issue tracker on github but I'm not sure if https://github.com/grpc/grpc-java/issues/1636 and https://github.com/grpc/grpc-java/issues/1648 are the issue I'm having.

[Eric Anderson]

On Sat, Jun 25, 2016 at 4:36 AM, Taehyun Park <gold.d...@gmail.com> wrote:

I'm using grpc java on Android and I found a very weird issue. After a certain period a ManagedChannel no longer works.

Was that after a period of inactivity? Were you on good WiFi (one that you trust), bad WiFi, or cellular?

I instantiated a ManagedChannel when there is no cached channel then cache it until the number of active channels is 0. My app worked fine and didn't have a problem when it's launched. but all grpc calls stopped working after a certain period. The app wasn't closed but it was in a backstack.

I searched a similar issue in grpc issue tracker on github but I'm not sure if https://github.com/grpc/grpc-java/issues/1636 and https://github.com/grpc/grpc-java/issues/1648 are the issue I'm having.

When discussing keepalive, the general assumption is the network misbehaved (which is not uncommon on mobile). Keepalive is only going to be active when an RPC is outstanding. That means that it will need to be combined with channel idleness to close TCP connections after inactivity.

https://github.com/grpc/grpc-java/issues/1972

https://github.com/grpc/grpc-java/issues/1276

Both are planned for 1.0 in order to give a full solution to this sort of issue (assuming that failures are due to poor networks). Idleness in general is useful. With it you really don't have much reason to cache the channel like you were. You could create the channel eagerly (which starts IDLE) and the channel can release resources when there is a period of inactivity.

[Taehyun Park]

All devices are connected to a router which has a good wifi connection. 3~4 hours of inactivity causes this issue. It could be less but I didn't check when the channel got unusable. The app was in a back stack as testing devices became locked and it turned off the screen due to inactivity. My internet connection is pretty stable so I'm not sure if this issue is due to internet connectivity. When the channel got unusable, all the requests never return a response.

I will implement a feature that can release resources when there is a period of inactivity as you suggested. 

Thank you for your rapid response!

[Taehyun Park]

I've confirmed that a period of inactivity is about 60 minutes. I added an idle timer which release a channel after a period of inactivity but a newly created channel is still unusable.

Below is the logcat once the channel is terminated and created.

06-26 13:41:47.612 28807-28891/com.test I/ManagedChannelImpl: [{0}] Terminated

06-26 14:39:01.128 28807-28917/com.test I/ManagedChannelImpl: [{0}] Created with target {1}

 I re-created a channel with the same config after a certain period of inactivity (an hour) but the newly created channel still doesn't make a request to a server. I installed a packet capturing app and my app didn't any make a request to a server after this happens. Could you guide me what I need to do to fix this issue?

  public static synchronized ManagedChannelImpl get() throws RuntimeException {

   if (_interceptor == null) {

     throw new IllegalArgumentException("A Non-null ClientInterceptor must be provided");

   }

   if (channel == null || channel.isShutdown() || channel.isTerminated()) {

     channel = OkHttpChannelBuilder.forAddress(BuildConfig.HOST, BuildConfig.PORT)

         .sslSocketFactory(provideSSLSocketFactory())

         .intercept(_interceptor)

         .compressorRegistry(CompressorRegistry.getDefaultInstance())

         .decompressorRegistry(DecompressorRegistry.getDefaultInstance())

         .build();

   }

   return channel;

 }

  static SSLSocketFactory provideSSLSocketFactory() {

    try {

     CertificateFactory cf = CertificateFactory.getInstance("X.509");

     InputStream caInput =

         AndroidContext.getContext().getResources().openRawResource(R.raw.api_rancam_com);

     Certificate ca;

     try {

       ca = cf.generateCertificate(caInput);

     } finally {

       caInput.close();

     }

     String keyStoreType = KeyStore.getDefaultType();

     KeyStore keyStore = KeyStore.getInstance(keyStoreType);

     keyStore.load(null, null);

     keyStore.setCertificateEntry("ca", ca);

     String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
     TrustManagerFactory tmf = TrustManagerFactory.getInstance(tmfAlgorithm);

     tmf.init(keyStore);

      SSLContext context = SSLContext.getInstance("TLS");

      context.init(null, tmf.getTrustManagers(), null);

     return context.getSocketFactory();

   } catch (Exception e) {

     Timber.e(e, "Failed to create SSLSocketFactory");

     return null;

   }

 }

  public static void release() {

    try {

     if (channel != null) {

       Timber.d("Releasing channel");

       channel.shutdown();

       channel = null;

       Timber.d("Channel released");

     }

   } catch (Exception e) {

     Timber.e(e, "Failed to release channel");

     channel = null;

   }

 }

Above is the code that I use to create and release a ManagedChannel.

Thank you in advance.

[Taehyun Park]

Sorry for confusions. The logcat was not the same device and the timer I added didn't work properly. Releasing resources by shutting down a channel when there is a period of inactivity solved my issue.

On Sunday, June 26, 2016 at 1:06:06 AM UTC+9, Eric Anderson wrote:

[Eric Anderson]

Whew. I saw your email this morning but didn't have time to reply saying I had so clue what could cause that. Glad it's working for you, and you can look forward to built-in idleness.

And an hour makes sense for the tcp connection to (silently) break with a home network, since there is a NAT involved.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/2b885e4d-abef-4658-a6e9-45c9a4cd967e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Anderson]

FYI: 3.0.0-pre1 includes support for auto-idling Channels (and keepalive for OkHttp). So you may be able to simplify your code now.

[Taehyun Park]

Thank you for this update. I updated to 1.0.0-pre2 and got rid of my own timer that releases a created channel after a certain period. 

It sends a GO_AWAY packet when the status is changed to IDLE. Do I have to release this channel or is it safe to reuse this channel? If I can reuse the channel, it looks like I no longer need to release the channel.

It would be really great if there's a documentation about managing channels as well.

On Saturday, July 30, 2016 at 2:22:29 AM UTC+9, Eric Anderson wrote:

FYI: 3.0.0-pre1 includes support for auto-idling Channels (and keepalive for OkHttp). So you may be able to simplify your code now.

On Sun, Jun 26, 2016 at 1:01 PM, Eric Anderson <ej...@google.com> wrote:

Whew. I saw your email this morning but didn't have time to reply saying I had so clue what could cause that. Glad it's working for you, and you can look forward to built-in idleness.

And an hour makes sense for the tcp connection to (silently) break with a home network, since there is a NAT involved.

On Jun 26, 2016 12:29 PM, "Taehyun Park" <gold.d...@gmail.com> wrote:

Sorry for confusions. The logcat was not the same device and the timer I added didn't work properly. Releasing resources by shutting down a channel when there is a period of inactivity solved my issue.

On Sunday, June 26, 2016 at 1:06:06 AM UTC+9, Eric Anderson wrote:

On Sat, Jun 25, 2016 at 4:36 AM, Taehyun Park <gold.d...@gmail.com> wrote:

I'm using grpc java on Android and I found a very weird issue. After a certain period a ManagedChannel no longer works.

Was that after a period of inactivity? Were you on good WiFi (one that you trust), bad WiFi, or cellular?

I instantiated a ManagedChannel when there is no cached channel then cache it until the number of active channels is 0. My app worked fine and didn't have a problem when it's launched. but all grpc calls stopped working after a certain period. The app wasn't closed but it was in a backstack.

I searched a similar issue in grpc issue tracker on github but I'm not sure if https://github.com/grpc/grpc-java/issues/1636 and https://github.com/grpc/grpc-java/issues/1648 are the issue I'm having.

When discussing keepalive, the general assumption is the network misbehaved (which is not uncommon on mobile). Keepalive is only going to be active when an RPC is outstanding. That means that it will need to be combined with channel idleness to close TCP connections after inactivity.

https://github.com/grpc/grpc-java/issues/1972

https://github.com/grpc/grpc-java/issues/1276

Both are planned for 1.0 in order to give a full solution to this sort of issue (assuming that failures are due to poor networks). Idleness in general is useful. With it you really don't have much reason to cache the channel like you were. You could create the channel eagerly (which starts IDLE) and the channel can release resources when there is a period of inactivity.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

[Taehyun Park]

I had to restore my own timer which releases a channel because auto-idle still caused the same problem. The grpc calls still didn't go through after a certain period so I guess releasing a channel is the only way if I don't want to use keepalive.

On Saturday, July 30, 2016 at 2:22:29 AM UTC+9, Eric Anderson wrote:

FYI: 3.0.0-pre1 includes support for auto-idling Channels (and keepalive for OkHttp). So you may be able to simplify your code now.

On Sun, Jun 26, 2016 at 1:01 PM, Eric Anderson <ej...@google.com> wrote:

Whew. I saw your email this morning but didn't have time to reply saying I had so clue what could cause that. Glad it's working for you, and you can look forward to built-in idleness.

And an hour makes sense for the tcp connection to (silently) break with a home network, since there is a NAT involved.

On Jun 26, 2016 12:29 PM, "Taehyun Park" <gold.d...@gmail.com> wrote:

Sorry for confusions. The logcat was not the same device and the timer I added didn't work properly. Releasing resources by shutting down a channel when there is a period of inactivity solved my issue.

On Sunday, June 26, 2016 at 1:06:06 AM UTC+9, Eric Anderson wrote:

On Sat, Jun 25, 2016 at 4:36 AM, Taehyun Park <gold.d...@gmail.com> wrote:

I'm using grpc java on Android and I found a very weird issue. After a certain period a ManagedChannel no longer works.

Was that after a period of inactivity? Were you on good WiFi (one that you trust), bad WiFi, or cellular?

I instantiated a ManagedChannel when there is no cached channel then cache it until the number of active channels is 0. My app worked fine and didn't have a problem when it's launched. but all grpc calls stopped working after a certain period. The app wasn't closed but it was in a backstack.

I searched a similar issue in grpc issue tracker on github but I'm not sure if https://github.com/grpc/grpc-java/issues/1636 and https://github.com/grpc/grpc-java/issues/1648 are the issue I'm having.

When discussing keepalive, the general assumption is the network misbehaved (which is not uncommon on mobile). Keepalive is only going to be active when an RPC is outstanding. That means that it will need to be combined with channel idleness to close TCP connections after inactivity.

https://github.com/grpc/grpc-java/issues/1972

https://github.com/grpc/grpc-java/issues/1276

Both are planned for 1.0 in order to give a full solution to this sort of issue (assuming that failures are due to poor networks). Idleness in general is useful. With it you really don't have much reason to cache the channel like you were. You could create the channel eagerly (which starts IDLE) and the channel can release resources when there is a period of inactivity.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

[Eric Anderson]

Catching up on old threads...

It sends a GO_AWAY packet when the status is changed to IDLE. Do I have to release this channel or is it safe to reuse this channel? If I can reuse the channel, it looks like I no longer need to release the channel.

The channel should have been safe to reuse. When the channel is IDLE, it should be in a very similar state to a newly-created channel.

On Thu, Aug 11, 2016 at 11:51 PM, Taehyun Park <gold.d...@gmail.com> wrote:

I had to restore my own timer which releases a channel because auto-idle still caused the same problem. The grpc calls still didn't go through after a certain period so I guess releasing a channel is the only way if I don't want to use keepalive.

That's unfortunate. What delay did you use for auto-idle? It sounds like you didn't enable OkHttp's keepalive, but that should really only help if the RPCs take a long time before the response arrives.

[nevil...@gmail.com]

Hi Eric,

I've also observed the same issue where after a while, my GPRC calls from Android start failing.

I lazily create my stub, so I've tried checking if stub is disconnected, and creating a new one in that case, but I'm still not winning. Is there some solution that you or anyone else could suggest?

Regards

[Eric Anderson]

On Sat, Sep 17, 2016 at 12:40 AM, <nevil...@gmail.com> wrote:

I lazily create my stub, so I've tried checking if stub is disconnected, and creating a new one in that case, but I'm still not winning. Is there some solution that you or anyone else could suggest?

The channel itself is lazy, so you could just create the channel eagerly. When the channel is disconnected, using the channel will automatically reconnect. The problem here is probably because the channel is "connected," but something is wrong with the TCP connection.