keyManager is the method you are looking for on SslContext. And if you have a Netty server as well, then clientAuth is another method in SslContext that you want.
You only need to set the trustManager on the client if the server's certificate isn't signed by a normal CA (which has nothing to do with client certificates). Since client certificates are commonly signed by a "fake" CA, then it would be common to specify trustManager on the server-side to verify the client certificates.
You might have a look at our
client auth test, and you should be aware of
a bug when using tcnative and Netty where the clientAuth() method didn't work. Jetty ALPN works fine. That bug has been fixed upstream, but grpc-java isn't yet compatible with that new version; we expect to be for the 0.13 release scheduled for two weeks. Only the server-side was broken.