gRPC Metadata and multithreaded environment

[Alex]

Hello,

I'm trying to get my head around how to use request and response metadata while working with grpc stubs in multithreaded environment ( i.e. multiple threads hitting same stub at same time ), I found this example in one of gRPC tests:

   AtomicReference<Metadata> trailersCapture = new AtomicReference<Metadata>();

    AtomicReference<Metadata> headersCapture = new AtomicReference<Metadata>();

    stub = MetadataUtils.captureMetadata(stub, headersCapture, trailersCapture);

    assertNotNull(stub.emptyCall(EMPTY));

    // Assert that our side channel object is echoed back in both headers and trailers

    Assert.assertEquals(contextValue, headersCapture.get().get(METADATA_KEY));

    Assert.assertEquals(contextValue, trailersCapture.get().get(METADATA_KEY));

Looking into Metadata utils, I can see it creates an interceptor, which sets the state of headersCapture/trailersCapture upon headers coming:

public static ClientInterceptor newCaptureMetadataInterceptor(final AtomicReference<Metadata> headersCapture, final AtomicReference<Metadata> trailersCapture) {
    return new ClientInterceptor() {
        public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
            return new SimpleForwardingClientCall(next.newCall(method, callOptions)) {
                public void start(final Listener<RespT> responseListener, Metadata headers) {
                    headersCapture.set((Object)null);

                    trailersCapture.set((Object)null);
                    super.start(new SimpleForwardingClientCallListener(responseListener) {
                    public void onHeaders(Metadata headers) {
                        headersCapture.set(headers);
                        super.onHeaders(headers);
[...]
                    }

As far as I'm concerned, this might work well when stub is called by one thread, but because we share headersCapture state, it will get overwritten ( or it won't and I'm not seeing something? )

My main question is : How can i setup a stub, that will be thread-safe, and able to return correct metadata headers back to the each caller? 

[Carl Mastrangelo]

So, before going forward, here is the expected use:

• Metadata objects are cheap, make new ones per RPC call
  
• Metadata objects are mutable and not threadsafe. 
• Once you pass the Metadata to our API (ClientCall or ClientInterceptor), you must consider it untouchable.  You cannot modify it or any of the objects set on it any more.
• Once you get a Metadata object back from our API, it is yours.  We promise to never touch it again.  

In addition, ClientInterceptors are cheap to make too, so you can presumably make a new one per call.  Depending on your exact use, you can create a new one next to each stub call, which a reference to a Metadata in the scope of the closure.  That will make sure the metadata for each call doesn't interfere with other calls.

We use AtomicReference in that code mainly as a way to get an indirection.  Basically, a pointer.   Unless you have some performance concerns, making a new Interceptor each time would be the right thing.