GRPC client interceptors in Java and fault injection

[Christopher Meiklejohn]

As part of a research project we're working on at Carnegie Mellon University, we're using GRPC client interceptors to inject faults when the message contents contain certain keywords.  For this scenario, we're looking only at the unary API.

To achieve this, I've made a custom interceptor that throws an exception (or, creates a Status of a particular type like FAILED_PRECONDITION, and converting that to an exception) and throwing in the sendMessage callback, where I can access the message contents.  Before I throw the exception, I call delegate().halfClose() and *do not* call super.sendMessage.

The problem I'm running into is this: I'm seeing no problems locally, but when I try my code in the GitHub Actions environment, I'm still seeing the call be received at the remote service it's calling.  It's transient: I put my test in a loop and it's failing at different iterations of the loop.  

Should this be happening?  Any insights on how I can go about debugging exactly what is happening?  

Thanks,

Chris

[Eric Anderson]

On Mon, Nov 15, 2021 at 12:55 AM Christopher Meiklejohn <christopher...@gmail.com> wrote:

To achieve this, I've made a custom interceptor that throws an exception (or, creates a Status of a particular type like FAILED_PRECONDITION, and converting that to an exception) and throwing in the sendMessage callback, where I can access the message contents.

Interceptors should not throw. Generally interceptors should call Listener.onClose() instead. But that also means you must not pass the listener to `next`, because then it will also call onClose().

Before I throw the exception, I call delegate().halfClose() and *do not* call super.sendMessage.

You can have an RPC without a message. Streaming RPCs can work this way. You are creating a new RPC, and then gracefully closing it, but the RPC still exists.

Should this be happening?  Any insights on how I can go about debugging exactly what is happening?

You should delay calling `next` (which creates a new RPC) from the interceptor until you are certain the RPC will be needed (i.e., that you won't inject a failure).

The Client Cache Example is in a similar situation to you, as it needs to avoid sending an RPC if there is a cache hit. Your code will be a bit simpler, because you don't need CachingListener, but the interceptCall+ForwardingClientCall code is appropriate to look at. Unfortunately, the current code in master suffers from actually sending the RPC unnecessarily, so you can look at my branch that has sketched out a better approach (again, pay most attention to interceptCall+ForwardingClientCall):

https://github.com/ejona86/grpc-java/blob/caching-interceptor-avoid-rpcs/examples/android/clientcache/app/src/main/java/io/grpc/clientcacheexample/SafeMethodCachingInterceptor.java

Essentially, it 1) avoids calling `next` until it is certain it needs an RPC and 2) caches method calls until it has made its decision.

[Christopher Meiklejohn]

Hi Eric,

Thanks for your response. I was able to refactor my code matching the
example you provided. I can't thank you enough -- this was an
extremely useful example that helped me understand both the proper way
to do this and what was actually happening underneath it all.

Christopher