Certificates for Mutual TLS for in-network remote access of embedded server

[Kartik Aiyer]

Hello folks

We have implemented a gRPC server on our embedded linux based camera and have a couple of clients that are expected to run on both the camera itself (so local loopback connection) as well as on host computers that are on the same subnet as the camera. Both the server and clients are written in C++ and use the gRPC C++ API. I’m trying to use mutual TLS so that only clients written by us can connect to the server.

I setup a self-signed root CA and used it to sign a server certificate and client certificate (more than one client certificate since I have more than one client). I’m not sure what is the best way to setup the certificate. From what I understand that either the common name or subjectAlternativeNames will be used to verify a connection in addition to the signature with the root certificate.

Server

• I can setup the camera’s hostname to something that will match the common name in the certificate. Is this the recommended approach ?

Client

• I can’t set the hostnames of the clients, so I’m not sure what to put in the common name for the server to verify. Any recommendation here ?

Currently I’m using a subjectAlternativeName of IP:0.0.0.0 which allows me to make calls over local loopback but its not usable over the network.

I’m using SslCredentials and SslServerCredentials but I’m wondering if I should be using the TlsCredentials and TlsServerCredentials with some kind of of custom verification callback. I would appreciate any advice on setting up the certificates appropriately for the usecase I have described.

To summarize, the client is expected to connect to the cameras on the same network and be able to use the remote API.

Thanks
Kartik

​

[Kartik Aiyer]

The following script is what I used to generate the certificates

#!/bin/bash

rm *.pem

# Generate a self-signed CA certificate and private key
openssl req -x509 -newkey rsa:4096 -days 3650 -keyout ca-key.pem -out ca-cert.pem -nodes -subj "/C=US/ST=California/O=Motive/OU=Embedded/CN=sc1-grpc-ca"

# Display info on self-signed CA certificate
openssl x509 -in ca-cert.pem -noout -text

# Generate a server private key and CSR
openssl req -newkey rsa:4096 -keyout kt-cam-key.pem -out kt-cam-req.pem -nodes -subj "/C=US/ST=California/O=Motive/OU=Embedded/CN=sc1-kt-cam" -addext "subjectAltName = IP:0.0.0.0"

# Use the CA cert to sign the kt-cam server CSR
openssl x509 -req -in kt-cam-req.pem -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out kt-cam-cert.pem -days 3650 -extfile kt-cam-ext.cnf

# Generate a client private key and CSR for kt_iot
openssl req -newkey rsa:4096 -keyout kt-iot-key.pem -out kt-iot-req.pem -nodes -subj "/C=US/ST=California/O=Motive/OU=Embedded/CN=sc1-kt-iot" -addext "subjectAltName = IP:0.0.0.0"

# Use the CA cert to sign the kt-iot client CSR
openssl x509 -req -in kt-iot-req.pem -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out kt-iot-cert.pem -days 3650 -extfile kt-iot-ext.cnf

# Generate a client private key and CSR for kt_cli
openssl req -newkey rsa:4096 -keyout kt-cli-key.pem -out kt-cli-req.pem -nodes -subj "/C=US/ST=California/O=Motive/OU=Embedded/CN=sc1-kt-cli" -addext "subjectAltName = IP:0.0.0.0"

# Use the CA cert to sign the kt-cli client CSR
openssl x509 -req -in kt-cli-req.pem -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out kt-cli-cert.pem -days 3650 -extfile kt-cli-ext.cnf

All the extension files look like this

subjectAltName = IP:0.0.0.0

Thanks
Kartik

​

[Sachin Bharadwaj S]

Hi Kartik,

The below answer is assuming that your local mTLS is working but remote mTLS is failing. If this is not the case, clarify what is not working and are you getting any error logs.

It is not mandatory for the client's IP to be part of the client's cert.

But the server's cert needs to have the server's IP or hostname. This should match with how the client is going to connect to the server.

For the local client, 0.0.0.0 might work but for the remote clients running on a different machine, this server cert does not work.

How is the remote client connecting to the server?

1. If it is based on IP directly, add that IP as well in the server cert. You can add multiple IPs in SAN

2. If it is based on hostname, add the DNS name something like this: subjectAltName=DNS:<serverHostname>,IP:x.y.z.a

Let us know if it works

Regards,

Sachin

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/87e3c94e-2935-46c4-8bd7-fd407f9070d0n%40googlegroups.com.

[Kartik Aiyer]

HI Sachin,

Thanks a bunch for the quick response. Currently the camera (server) is running avahi and the uses mdns to find all cameras found on the network. I then use the IP address of a discovered camera to connect. The IP address of the camera or the clients are dynamic and are typically assigned by DHCP so I don’t think a static IP address would work. I can assign a hostname to the camera (server) though I cannot assign one to the clients.

I have few questions based on my ignorance of mTLS.
Assuming I could add the hostname of the camera to its server certificate’s common name, is this what the client will use in addition to the signature check to validate the server ?

Given that the client can be any machine, I only want to confirm that the client program written by me is the one that can connect to the server. Is there a way I can tell the server to only validate the signature of client supplied certificate and skip the host name or IP address checks ?

Finally, a question about subjectAlternateName. I’ve currently added the 0.0.0.0 address to both the server and client extension files. What does it mean to the client when it see a IP address of 0.0.0.0 in the server certificate’s subjectAlternativeName. Does it mean the client will accept the connection only if the server is on localhost ? Similarly what does it mean to the server when it sees 0.0.0.0 in the client’s subjectAlternativeName ?

Thanks a bunch for the help and I truly appreciate your time.

Here are logs of a remote call made from my computer to the camera both of which are on my home wifi.

Server

Sep 12 22:46:32 buildroot kt_cam[204]: E0912 22:46:32.284179389     208 ssl_transport_security.cc:1807] No match found for server name: 10.0.0.39.

Client

[2022-09-12 15:46:32.235] [info] Running on 10.0.0.39:50051
[2022-09-12 15:46:32.246] [info] Starting
[2022-09-12 15:46:32.410] [error] 14: failed to connect to all addresses
[2022-09-12 15:46:32.410] [info] Greeter received: RPC failed

The only error I get from the client when I call the stub is “failed to connect to all addresses” however on the server end I get the above error which says no match for server name. Little confused why it says that. P.S: 10.0.0.39 is the server (camera) IP address.

Thanks again for the help.

Kartik

​

[Kartik Aiyer]

Hi Sachin, et al,

A follow up question. Given that the server is at 10.0.0.39 and the client attempts to connect with its certificate using the IP address specifically, the server reports that "No match found for server name: 10.0.0.39". Does the server expect the client to connect using common name or an address specified in the servers subjectAlternativeName only ? i.e only connections attempts to addresses or DNS names specified in the certificate of the server are accepted ? Even if there server is listening on other addresses, the client cannot use those addresses directly to connect ? If I attach server hostname dynamically when an IP address is assigned to the camera then I'm guessing that would work provided the client uses the host name to connect, though I'm not sure how the client will translate the hostname to an actual address given that there is no DNS serer in the internal network. 

Thanks
Kartik

[Sachin Bharadwaj S]

Hi Kartik,

See my inline responses

Regards,

Sachin

On Tue, Sep 13, 2022 at 1:36 PM 'Kartik Aiyer' via grpc.io <grp...@googlegroups.com> wrote:

Hi Sachin, et al,

A follow up question. Given that the server is at 10.0.0.39 and the client attempts to connect with its certificate using the IP address specifically, the server reports that "No match found for server name: 10.0.0.39". Does the server expect the client to connect using common name or an address specified in the servers subjectAlternativeName only ?

I was expecting this kind of error: "No match found for server name: 10.0.0.39"

As I had mentioned earlier, this is expected from the server. 

There is a way to override this behavior by calling SetSslTargetNameOverride as discussed in this thread.

But, it is a bad security practice as described in the same thread.

If IP keeps changing, having hostname as DNS in the SAN field of server cert is the best option for you.

I will wait to see if any other grpc experts commenting on this.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/54b6f81b-bd13-431b-867a-e37603d6e902n%40googlegroups.com.

[Easwar Swaminathan]

You might be better off having two self-signed CAs. One for server, and one for the clients. This will give you some flexibility with certificate rotation. But given that your server and your clients are on the same network, this might add a lot of management complexity. You need to think about how you are going to keep the private keys of the self-signed CAs secure. 

But, stepping back, given that your clients and servers are on the same subnet, what other clients on the same network do you want *not* to connect to your server?

[Kartik Aiyer]

Oh man.. I hit the wrong button and I think I only replied to the author. Going to re-write what I sent

Thanks a bunch for your feedback Sachin and Eashwar

But, stepping back, given that your clients and servers are on the same subnet, what other clients on the same network do you want not to connect to your server?

Good question. I’m only trying to guard against clients that are not written by my company that are trying to probe the gRPC server API. I believe gRPC allows clients to probe the server API using reflection without a stub. If this can be disabled in our usecase then maybe I can proceed using InsecureChannels given that all the access is within a subnet. However, I’m not sure if this a good practice for a production. This is not running on the cloud or anything. If you have any suggestions about this I’m all ears.

If IP keeps changing, having hostname as DNS in the SAN field of server cert is the best option for you.

I don’t have a DNS server running in the local network and so the server advertises over mDNS and I have written a library that my client uses to perform service discovery. I read this in the docs which mention a name resolution plugin. I could wrap my service discovery code into a plugin but I can’t see any samples of this in C++ and am not sure what class to override or how to install the plugin into a channel. My guess is if I provide an mDNS name resolution plugin installed on any *.local name then, the client will be able to find the device and will supply the service name in the peer info to the server (_src-grpc._tcp.local). I can add that name to the server’s CN and I’m guessing that will be sufficient for the server to validate the Client’s connection attempt.

If that works and the server accepts the clients connection parameters, then the next step would be to get the server to validate the clients certificate. I see that I can tell the server to require a client certificate but not verify which is instead delegated to my server code. However, I’m again not sure how to do this verification. Is it done in every call or is there an object that I have to attach to the server or the server builder ? There is mention of a grpc_auth_context which looks like some C API but I’m not sure how to attach it to server.

Does the above sound like a more reasonable approach. If so, any suggestions on where I can look to start experimenting ?

Thanks again for all the help.

Regards
Kartik

​

[Sachin Bharadwaj S]

On Wed, Sep 14, 2022 at 6:58 PM 'Kartik Aiyer' via grpc.io <grp...@googlegroups.com> wrote:

Oh man.. I hit the wrong button and I think I only replied to the author. Going to re-write what I sent

Thanks a bunch for your feedback Sachin and Eashwar

But, stepping back, given that your clients and servers are on the same subnet, what other clients on the same network do you want not to connect to your server?

Good question. I’m only trying to guard against clients that are not written by my company that are trying to probe the gRPC server API. I believe gRPC allows clients to probe the server API using reflection without a stub. If this can be disabled in our usecase then maybe I can proceed using InsecureChannels given that all the access is within a subnet. However, I’m not sure if this a good practice for a production. This is not running on the cloud or anything. If you have any suggestions about this I’m all ears.

If IP keeps changing, having hostname as DNS in the SAN field of server cert is the best option for you.

I don’t have a DNS server running in the local network and so the server advertises over mDNS and I have written a library that my client uses to perform service discovery. I read this in the docs which mention a name resolution plugin. I could wrap my service discovery code into a plugin but I can’t see any samples of this in C++ and am not sure what class to override or how to install the plugin into a channel. My guess is if I provide an mDNS name resolution plugin installed on any *.local name then, the client will be able to find the device and will supply the service name in the peer info to the server (_src-grpc._tcp.local). I can add that name to the server’s CN and I’m guessing that will be sufficient for the server to validate the Client’s connection attempt.

If that works and the server accepts the clients connection parameters, then the next step would be to get the server to validate the clients certificate. I see that I can tell the server to require a client certificate but not verify which is instead delegated to my server code. However, I’m again not sure how to do this verification. Is it done in every call or is there an object that I have to attach to the server or the server builder ? There is mention of a grpc_auth_context which looks like some C API but I’m not sure how to attach it to server.

I think you are mixing two things. Just to clarify, the issue of IP/DNS missing in server cert leading to error message is not related to 

grpc_ssl_client_certificate_request_type

 If your intention is to verify the client cert on the server side, you need to pass the CA which has signed the client certificate. And you need to pass GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY

grpc library running in your application will use openssl library (most likely) to validate the incoming client certificate against the CAs which you have passed to grpc.

You might need to explore more on SslServerCredentialsOptions 

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/baf54fb4-011e-460e-b4be-c2fd103a72ben%40googlegroups.com.

[Kartik Aiyer]

Just wanted to close the loop here. 

I went with using the `SetSslTargetNameOverride` when the client makes the call. Both sides present their certificates and they share the root CA so signature verification succeeds on both ends. The Server uses the client specified Traget name override and compares against its comman name and succeeds. I understand that this is not the best approach from a security standpoint but given that the server and clients are running on a local network and are assigned IP addresses via DHCP, I can't think of another way to work. 

One thing I did notice, is that when the camera boots up and launches the server, it takes about 30 seconds before SSL verifications succeed. i.e the camera boots up launches the gRPC server and client call attempts keep failing certificate verification until about 30 -40 seconds after bootup. Does anyone know what other OS resources need to come up for the SSL verifications to proceed ? This was not the case when using insecure credentials on both ends. 

Regards

Kartik