Thank you, Eric. Well put and easy/clear to understand!
I guess it was my naive understanding of "defense-in-depth" that I tried to add additional layers of authentication on top of TLS with a digital signature. Is there no real benefit to a HMAC + TLS vs something like a JWT + TLS? In today's world, using TLS is a no-brainer, but that only covers the server-side part of it all, client/user authentication is still something I want to make sure I get right. I believe having a HMAC would prevent someone from making anything other than replay requests (which I can guard against using a nonce), but a stolen token would enable an attacker to form any sort of request on behalf of that user! I understand that the likelihood of being able to read the message/token when transporting via TLS is unlikely, but as there are vulnerabilities uncovered time and time again, layering my protection seems like a good approach.
I think I've thought up of a way to get this working (in the Go library at least):
Create a general message proto that includes your basic JWT headers (e.g. exp, jti, nbf, etc.) and make it a property on all messages used for requests to the RPC service. When creating a request, store this common message type into the context.Context used to send the request so that a pluggable Credential implementation can pull the information from the context.Context and create a JWT metadata header out of it. Server side, you'd verify the JWT headers match what's in the request and verify the JWT signature. I think this would achieve the same level of protection, as long as the implementation of the nonce is done properly.
Thank you for the information!
Prateek M.