Using self-signed certificate for mTLS connection in C#

[umi...@gmail.com]

Hi everyone!

I have a self-signed certificate (with its private key as well), provided on both server and client sides, and I want to use that certificate to encrypt the connection in a mutual TLS way. Here is the code overview.

// Getting the cert and PEM formats...
byte[] certBytes = ReadCertificateBytes(...);
X509Certificate2 cert = new X509Certificate2(secretBytes, string.Empty, X509KeyStorageFlags.Exportable);

string certPEMFormat = "-----BEGIN CERTIFICATE-----\n";
certPEMFormat += Convert.ToBase64String(containerCert.Export(X509ContentType.Cert), Base64FormattingOptions.InsertLineBreaks);
certPEMFormat += "\n-----END CERTIFICATE-----";

string privateKeyPEMFormat = "-----BEGIN RSA PRIVATE KEY-----\n";
privateKeyPEMFormat += ConvertPrivateKeyToPEM(cert);
privateKeyPEMFormat = "-----END RSA PRIVATE KEY-----\n";

var keypair = new KeyCertificatePair(certPEMFormat, privateKeyPEMFormat);

// Client side
SslCredentials clientCredentials = new SslCredentials(certPEMFormat, keypair);
var channel = new Channel(grpcEndpoint, channelCredentials);

...

// Server side
SslServerCredentials serverCredentials = new SslServerCredentials(new[] { keypair }, certPEMFormat, false);
Server server = new Server(someChannelOptions)
{
   // Create the default implementation
   Services = { ProtocolTypes.ExecutionService.BindService(executionServiceImp) },
   // Using 0.0.0.0 to hear on all interfaces
   Ports = { new ServerPort("0.0.0.0", port, serverCredentials) },
};
server.Start();




By setting things like this, these are the errors I get.

On client side:

E0610 15:34:42.670602 0 ..\..\src\core\tsi\ssl_transport_security.cc:1229: Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

On server side:

E0610 15:34:42.076880 0 ..\..\src\core\tsi\ssl_transport_security.cc:1566: No match found for server name: SERVERNAME.

Looking around, I found an advice to pass these options to client's channel (override target name, so it matches the Subject field from X509 certificate)

// Client side
var channelOptions = new List<ChannelOption>()
{
    new ChannelOption(ChannelOptions.SslTargetNameOverride, "SUBJECT_STRING"),
    new ChannelOption(ChannelOptions.DefaultAuthority, "SUBJECT_STRING")
};
var channel = new Channel(grpcEndpoint, channelCredentials, channelOptions);

And now, there are no error printings on the server side ('No match found for server name'), but they still exist on the client side (CERTIFICATE_VERIFY_FAILED).

Certificate is already generated and provided so I can not change it. I really tried with different arguments, options, combinations, and what not, but I just can not establish this secure connection. Am I missing something crucial here, can I use one and only self-signed certificate with private key for mTLS (without any CA root certs), or maybe I am missing some flag here or something minor?

Thanks,
Ugi

[Jan Tattermusch]

Setting new ChannelOption(ChannelOptions.DefaultAuthority, "SUBJECT_STRING") seems unnecessary. It's fine to use ChannelOptions.SslTargetNameOverride but only for testing (do not use in production!).

Have you tried setting GRPC_VERBOSITY=debug and perhaps some traces (see https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md)?

You can use openssl command line tools to verify that the certificates are well formed and that the server/client pem can be verified using the CA cert.

A working mTLS example for C# is here: https://github.com/jtattermusch/grpc-authentication-kubernetes-examples

[umi...@gmail.com]

Thanks Jan for a quick replay.

Right, there is no need for seting DefaultAuthority, it was just a leftover from some comment I found online.

Verifying the certificates using openssl was all good (the right format, able to read all the info, cert/key matching was good). Pay attention that I do not have CA cert, I am using one and only self-signed cert (and its private key) on server side, client side, and as a CA cert. Something like these guys here [https://github.com/sandtable/ssl_grpc_example].

Tracing seems to be not yet implemented for Windows, but I will give it a try with setting the proper verbosity (debug).

[umi...@gmail.com]

These are the outputs with GRPC_VERBOSITY=DEBUG and GRPC_TRACE=all (I will put only the lines around the error):

I0611 17:27:35.427914 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\transport\connectivity_state.cc:164: SET: 014F9E94 pick_first: IDLE --> CONNECTING [connecting_changed] error=00000000 "No Error"
I0611 17:27:35.428916 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\transport\connectivity_state.cc:190: NOTIFY: 014F9E94 pick_first: 0150DAFC
I0611 17:27:35.428916 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src/core/ext/filters/client_channel/lb_policy/subchannel_list.h:354: [pick_first 014F9DA0] subchannel list 014FA0B0 index 0 of 2 (subchannel 0150D308): renewing watch: requesting connectivity change notification (from CONNECTING)
I0611 17:27:35.428916 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\transport\connectivity_state.cc:116: CONWATCH: 0150D384 subchannel: from CONNECTING [cur=CONNECTING] notify=014E142C
I0611 17:27:35.428916 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\ext\filters\client_channel\client_channel.cc:214: chand=014ECCE8: lb_policy=014F9DA0 state changed to CONNECTING
I0611 17:27:35.429913 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\ext\filters\client_channel\client_channel.cc:196: chand=014ECCE8: setting connectivity state to CONNECTING
I0611 17:27:35.429913 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\transport\connectivity_state.cc:164: SET: 014ECD2C client_channel: IDLE --> CONNECTING [lb_changed] error=00000000 "No Error"
I0611 17:27:35.429913 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\transport\connectivity_state.cc:116: CONWATCH: 014F9E94 pick_first: from CONNECTING [cur=CONNECTING] notify=0150DE1C
I0611 17:27:35.429913 21609880 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\iomgr\executor.cc:167: EXECUTOR (resolver-executor) [0]: step (sub_depth=1)
I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client read_server_certifi  - !!!!!!
I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client read_certificate_st  - !!!!!!
I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client verify_server_certi  - !!!!!!
E0611 17:27:35.440924 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:1229: Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
D0611 17:27:35.440924 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\security\transport\security_handshaker.cc:129: Security handshake failed: {"created":"@1560266855.441000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\security\transport\security_handshaker.cc","file_line":248,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I0611 17:27:35.441423 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\channel\handshaker.cc:212: handshake_manager 014D1000: error={"created":"@1560266855.441000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\security\transport\security_handshaker.cc","file_line":248,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"} shutdown=0 index=2, args={endpoint=00000000, args=00000000 {size=0: (null)}, read_buffer=00000000 (length=0), exit_early=0}
I0611 17:27:35.444923 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\channel\handshaker.cc:245: handshake_manager 014D1000: handshaking complete -- scheduling on_handshake_done with error={"created":"@1560266855.441000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\security\transport\security_handshaker.cc","file_line":248,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I0611 17:27:35.445423 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\lib\iomgr\timer_generic.cc:467: TIMER 014D1048: CANCEL pending=true

Everything looks fine till the client starts verifying the server certificate. Again, client and server use the same cert for CA root, public key, and private key.

[umi...@gmail.com]

Looking now at these three lines:

I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client read_server_certifi  - !!!!!!
I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client read_certificate_st  - !!!!!!
I0611 17:27:35.439927 0 T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\tsi\ssl_transport_security.cc:213:                 LOOP - TLS client verify_server_certi  - !!!!!!


Can it be that client reads server cert and then checks if that cert exists and can be found in its certificate store? Basically, since it is the same cert on boths sides, I can add it to the store before the communication starts...

[umi...@gmail.com]

Just to close the thread, the error was on my side.

When creating self-signed certificate, make sure that appropriate KeyUsage and BasicConstrains flags are set. Apperently, gRPC checks them, and if for example X509KeyUsageFlags.KeyCertSign is not set, the handshake protocol will fail.

Thanks,

Ugi