grpc-java: Register all channels / access root channel list

[Sam G]

Hey gRPC,

I am working in grpc-java and looking for a way to access all of my currently-open root channels.

I am aware of Channelz's existence, perhaps that would help? But I have so far been unable to find documentation to that effect (i.e. fetching all living channels, regardless of OkHttp/Netty implementation).

Some of our gRPC services make use of Google's Cloud APIs for Java and so they have their own ManagedChannel connections up to, say, Firestore and Cloud Logging.

When shutting down our server, it often closes the active connections as the JVM exits, rather than giving them time to exit gracefully (this could be some implementation problem on our side, but we have encountered other people with this same issue).

This isn't much of an issue in production, where servers are long-lived and rarely shut down with live traffic in play. However, in our testsuites, we are encountering this exception which is causing other problems, like unreported coverage (missing coverage lines for gRPC services):

Feb 21, 2019 8:05:57 PM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue

SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=11, target=firestore.googleapis.com:443} was not shutdown properly!!! ~*~*~*

    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.

java.lang.RuntimeException: ManagedChannel allocation site

at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)

at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)

at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)

at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:419)

at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:254)

at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:165)

at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:157)

at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:157)

at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:122)

at com.google.cloud.firestore.spi.v1beta1.GrpcFirestoreRpc.<init>(GrpcFirestoreRpc.java:121)

at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreRpcFactory.create(FirestoreOptions.java:80)

at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreRpcFactory.create(FirestoreOptions.java:72)

at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:509)

at com.google.cloud.firestore.FirestoreOptions.getFirestoreRpc(FirestoreOptions.java:315)

at com.google.cloud.firestore.FirestoreImpl.<init>(FirestoreImpl.java:76)

at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreFactory.create(FirestoreOptions.java:63)

at com.google.cloud.firestore.FirestoreOptions$DefaultFirestoreFactory.create(FirestoreOptions.java:56)

at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:497)

at io.[ REDACTED ].impl.FirestoreService.<init>(FirestoreService.kt:451)

        [... more frames ...]

at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)

at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)

at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)

at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)

at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)

This isn't specific to Firestore, this is just one example, although they are always related to Google Cloud APIs.

Is there a known way to avoid this issue, or at the very least, a way to query gRPC Java for any active channels that need to be shut down? Thank you in advance for your help.

(BTW I am posting here because the Firestore team did not seem to have an answer, and we have other places where this might occur, like our own use of our internal gRPC client within our gRPC server. So I am hoping there is a general gRPC-layer solution for gracefully closing active channels en-masse).

Thank you in advance for your help

Sam

[Sam G]

It appears the stacktrace I posted is written in white font. Apologies for that... here it is again, but this time, legibly:

[Carl Mastrangelo]

The warning you see is triggered by the garbage collector (via a WeakRef), so likely the Channel won't show up in Channelz when you see the message.  If you don't have references to these channels, then it won't be possible to shut them down directly.   If you can create them (such as in your tests), we provide some helper classes to auto create and shut them down such as GrpcServerRule.  

One other thing: you could (not saying it's a good idea), silence these logs for your tests.  I would strongly encourage you to file bugs to the projects that create the channels, (which is why we include the creation site stack trace), but mute them if you can't take any action in response to them.   

[Carl Mastrangelo]

You can see sample usage here:  https://github.com/grpc/grpc-java/blob/3a39b81cf54b3e93b06f8b7375f2bcbf0f712612/services/src/test/java/io/grpc/services/HealthStatusManagerTest.java#L48  Again, it works if you can pass in the Channel, but is less useful otherwise.

I see http://googleapis.github.io/gax-java/1.23.0/apidocs/com/google/api/gax/grpc/GrpcTransportChannel.html    as a possible TransportChannelProvider, but I haven't personally used it.   It seems like the right avenue to follow.  

I'm curious if this ends up working out.  

On Fri, Feb 22, 2019 at 2:00 PM Sam Gammon <s...@bloombox.io> wrote:

Thank you for your thoughtful response, that all makes sense.

Is there a code sample somewhere of the GrpcServerRule stuff? I’ll dig into the source and see what I can find. And does it shut down all channels, or just server side channels?

The exceptions usually come from Firestore channels, and I can see an interface in the service builder that accepts a TransportChannelProvider.

However I have never encountered the TransportChannelProvider and don’t know how I might go about providing my own, which might facilitate registering channels so they can be shut down.

Thank you again for your help.

Sam

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/dacef987-a86e-4b1b-9aa2-0d54c12eabb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-sam

sam gammon

founder and CEO, Bloombox

916.341.9482 // s...@bloombox.io

pgp: F6465D6FCE108644731B2A3AEABC7900D506BFF3

---

This message (and any associated files) may contain confidential and/or privileged information. If you are not the intended recipient or authorized to receive this for the intended recipient, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately and delete this message. Thank you for your cooperation.

[aj...@qlogic.io]

This has been fixed in latest version of clients. Can you upgrade and try again?

[Sam Gammon]

Thank you ajay, that’s great news. What do you mean by “fixed” exactly?

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9149b342-e3b8-4ef9-805d-0d1e6b26287c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.