TLS with Google Cloud ESP for GRPC endpoint

490 views
Skip to first unread message

b...@soliduslink.com

unread,
Mar 6, 2018, 5:50:36 PM3/6/18
to Google Cloud Endpoints
Hi,

We are hosting our grpc server with ESP at the moment with no encryption. We need to add a encryption layer over the communication to meet our security standard. Have tried to run grpc with TLS but with no success. Could some one give me some suggestions how to do it correctly?

Thanks a lot!

Best,
Baojun

Lizan Zhou

unread,
Mar 6, 2018, 7:39:49 PM3/6/18
to b...@soliduslink.com, Google Cloud Endpoints
Hi Baojun,

You can enable TLS between your gRPC client and ESP by Enabling SSL. Currently the upstream TLS with gRPC is not supported (#346).

If you are running ESP and gRPC server in same VM or same Pod, the suggested way to make it secure is to use Unix Domain Socket between ESP and your gRPC server.

Best,
Lizan





--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6ee88452-755c-4f03-925c-df07cde6a100%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Baojun Xu

unread,
Mar 7, 2018, 4:26:52 AM3/7/18
to Lizan Zhou, Google Cloud Endpoints, Jun Li
Hi Lizan,

Thanks a lot answering. Yes we are running ESP and gRPC server on the same VM.

I am not sure if I understood you correctly. Do you mean to use Unix Domain Socket between ESP and gRPC server to secure the channel between ESP and gRPC? Our purpose is to secure the channel between ESP and client. 

Best,
Baojun

On Wed, Mar 7, 2018 at 1:39 AM, Lizan Zhou <zli...@google.com> wrote:
Hi Baojun,

You can enable TLS between your gRPC client and ESP by Enabling SSL. Currently the upstream TLS with gRPC is not supported (#346).

If you are running ESP and gRPC server in same VM or same Pod, the suggested way to make it secure is to use Unix Domain Socket between ESP and your gRPC server.

Best,
Lizan





On Tue, Mar 6, 2018 at 2:50 PM <b...@soliduslink.com> wrote:
Hi,

We are hosting our grpc server with ESP at the moment with no encryption. We need to add a encryption layer over the communication to meet our security standard. Have tried to run grpc with TLS but with no success. Could some one give me some suggestions how to do it correctly?

Thanks a lot!

Best,
Baojun

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6ee88452-755c-4f03-925c-df07cde6a100%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

Lizan Zhou

unread,
Mar 7, 2018, 3:07:25 PM3/7/18
to Baojun Xu, Google Cloud Endpoints, Jun Li
If you just want to secure the channel between your gRPC client and ESP, then just follow this guide: Enabling SSL.

What error have you seen with this?

On Wed, Mar 7, 2018 at 1:26 AM Baojun Xu <b...@soliduslink.com> wrote:
Hi Lizan,

Thanks a lot answering. Yes we are running ESP and gRPC server on the same VM.

I am not sure if I understood you correctly. Do you mean to use Unix Domain Socket between ESP and gRPC server to secure the channel between ESP and gRPC? Our purpose is to secure the channel between ESP and client. 

Best,
Baojun
On Wed, Mar 7, 2018 at 1:39 AM, Lizan Zhou <zli...@google.com> wrote:
Hi Baojun,

You can enable TLS between your gRPC client and ESP by Enabling SSL. Currently the upstream TLS with gRPC is not supported (#346).

If you are running ESP and gRPC server in same VM or same Pod, the suggested way to make it secure is to use Unix Domain Socket between ESP and your gRPC server.

Best,
Lizan





On Tue, Mar 6, 2018 at 2:50 PM <b...@soliduslink.com> wrote:
Hi,

We are hosting our grpc server with ESP at the moment with no encryption. We need to add a encryption layer over the communication to meet our security standard. Have tried to run grpc with TLS but with no success. Could some one give me some suggestions how to do it correctly?

Thanks a lot!

Best,
Baojun

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

Baojun Xu

unread,
Mar 7, 2018, 6:00:00 PM3/7/18
to Lizan Zhou, Google Cloud Endpoints, Jun Li
We were not able to make the grpc calls from the client following the guide. What kind of protocol does esp use to serve the ssl port 443? Do you maybe have some code snippet for the grpc client that works with the guide?

Lizan Zhou

unread,
Mar 7, 2018, 6:23:19 PM3/7/18
to Baojun Xu, Google Cloud Endpoints, Jun Li
Port 443 serves HTTP/2 and HTTP/1.1 over SSL.

Which language are you using? A snippet from gRPC guide should just work. Note the server_name have to match your certificate has, and the root certificate should be trusted (you may need "GRPC_DEFAULT_SSL_ROOTS_FILE_PATH" env var to configure it).

Let me know what error are you seeing. Email me directly if you're not comfortable to share at public group.

Thanks,
Lizan



Thomas Barnekow

unread,
Apr 2, 2020, 6:31:22 AM4/2/20
to Google Cloud Endpoints
Hi,

I am having the same problem and somehow can't figure out how this works based on the piecemeal examples provided. Unfortunately, I could not find any working example showing the "whole thing". The examples provided are also sometimes using obsolete docker commands.

Is there any working example, if possible for a Java gRPC server (C# is also fine; other languages are OK, too, but I am most familiar with Java and C#)? I would like to host a Java server on Google Compute Engine as described here?

Regards, Thomas
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--

Wayne Zhang

unread,
Apr 2, 2020, 2:54:31 PM4/2/20
to Thomas Barnekow, Google Cloud Endpoints
Hi Thomas,

Sorry, we don't have a tutorial for setting TLS in ESP to secure clients and ESP for a grpc service. 

Most likely the problem is with TLS setup.   You can double check the followings:

1) is the server cert name match with the DNS name your client is calling?  If you use IP,  is your cert name has that IP address?  
If your cert name doesn't match with the name in your grpc channel creation in the client code,  you need to use host_override.  Here is a c++ code for that.

2) you can enable grpc trace by defining these two environments in your client host.

GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

Thanks

-Wayne


To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/604c1501-92ac-436f-a0fb-45d6277eed54%40googlegroups.com.

Thomas Barnekow

unread,
Apr 2, 2020, 4:16:17 PM4/2/20
to Google Cloud Endpoints
Hi Wayne,

Thanks a lot for your reply. Let me comment on your checks:

Regarding your first point, I don't use a DNS name but the IP address of the GCE VM. The server cert name is "/O=OrgName/OU=Servers/CN=35.240.160.100", where 35.240.160.100 is the IP address (not the real one in this example).

Here's the client C# code I am using for setting up the channel: 

const string host = "35.240.160.100";
const int port = 443;
string caCertPem = File.ReadAllText("Certs\\ca.cer");
var sslCredentials = new SslCredentials(caCertPem);
var channel = new Channel(host, port, sslCredentials);

The CA certificate contained in the ca.cer PEM file corresponds with the CA key used to sign the server-side certificate.

Regarding your second point on tracing, I will try that out. 

Thanks again, Thomas
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

Thomas Barnekow

unread,
Apr 2, 2020, 5:20:58 PM4/2/20
to Google Cloud Endpoints
Hi Wayne,

I have tried it again after having set the tracing-related environment variables (set the variables, deleted and re-ran the docker containers). However, I can't see anything in the logs (and I might be looking in the wrong place).

Firstly, I ran "sudo docker logs [container-name]" for both the esp container and the rpc server container. While I get the same error message on the client as before, the logs are silent on errors. The GRPC server only tells me that it is listening on port 50051. Further, the Logs Viewer doesn't tell me anything either. So, the question is where I'd look for errors. I am just starting with the Google Cloud Platform.

Regards, Thomas

On Thursday, April 2, 2020 at 8:54:31 PM UTC+2, Wayne Zhang wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

Wayne Zhang

unread,
Apr 2, 2020, 5:38:55 PM4/2/20
to Thomas Barnekow, Google Cloud Endpoints
Let us just enable grpc environment variables in the client host,  after set them, run your client code again, the detail log will dump to stdout or stderr.  what language is your grpc client?

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6e7d6706-39be-4987-a29d-5ee2382a07d3%40googlegroups.com.

Thomas Barnekow

unread,
Apr 3, 2020, 8:14:44 AM4/3/20
to Google Cloud Endpoints
Hi Wayne,

Setting the gRPC environment variables on the client produced a log that helped me identify and fix the issue! :-)

My client is written in C# and targets .NET Framework 4.7.2 (because I can't use .NET Core in my application). The problem was that the SSL handshake failed because the X.509 server certificate (nginx.crt) did not contain the subjectAltName with the IP address of the GCE VM. My nginx.crt only contained that IP address in the common name (CN) as described in the how-to guide, in which there was no mention of the subjectAltName being required. The interesting part is that it was not required when self-hosting the server on my local machine.

Using a self-signed certificate with the CN and subjectAltName containing the IP address of the GCE VM did not fully solve the problem, however. I had to create a certificate signing request (CSR) and sign it with my root CA while making sure the subjectAltName made it into the certificate (nginx.crt), which turned out to be not so easy. But once I had a proper server certificate signed by my root CA, which I also used in setting up the client channel, it worked!

Here is how I created the server certificate on my Windows 10 machine:

set ipAddress=[GCE VM IP Address]
set subject=/O=DokuMate/OU=Servers/CN=%ipAddress%
set subjectAltNameConfig=subjectAltName = IP:%ipAddress%
echo %subjectAltNameConfig% > extfile.cfg

openssl req -newkey rsa:2048 -keyout Certs\nginx.key -nodes -out Certs\Requests\nginx.csr -subj "%subject%" -addext "%subjectAltNameConfig%"

openssl x509 -req -extfile extfile.cfg -in Certs\Requests\nginx.csr -CA Certs\ca.cer -CAkey Certs\ca.key -passin pass:1111 -days 365 -set_serial 01 -out Certs\nginx.crt

This is what the server certificate with the subjectAltName now looks like (using "openssl x509 -in nginx.crt -text -noout"):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: DC = com, DC = dokumate, CN = DokuMate Root CA
        Validity
            Not Before: Apr  3 10:59:23 2020 GMT
            Not After : Apr  3 10:59:23 2021 GMT
        Subject: O = DokuMate, OU = Servers, CN = [GCE VM IP Address]
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    [Bytes removed]
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                IP Address:[GCE VM IP Address]
    Signature Algorithm: sha256WithRSAEncryption
         [Bytes removed]

There is still one thing that is strangely different from my self-hosted server. I used to verify the root (meaning CA) certificate of the server certificate by comparing its thumbprint with the thumbprint of the CA certificate. Strangely enough, while everything works, the thumbprints are different. Not sure whether there is still something wrong with my certificates.

Here is how I create the channel for my C# client to make it work:

const string host = "[GCE VM IP Address]";
const int port = 443;

string caCertPem = File.ReadAllText("Certs\\ca.cer");

string clientCertPem = File.ReadAllText("Certs\\client.cer");
string clientKeyPem = File.ReadAllText("Certs\\client.key");
var keyCertificatePair = new KeyCertificatePair(clientCertPem, clientKeyPem);

var sslCredentials = new SslCredentials(caCertPem, keyCertificatePair);

var channel = new Channel(host, port, sslCredentials);

The above is for mutual TLS. I can obviously also leave out the keyCertificatePair from the SslCredentials constructor and it also works in this case.

Thanks again for your help.

Regards, Thomas
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

Thomas Barnekow

unread,
Apr 3, 2020, 8:51:02 AM4/3/20
to Google Cloud Endpoints
I've now found out why the certificate thumbprints were different, so that is solved as well. I had recreated the CA certificate in the attempt to create a server certificate (nginx.crt) with a subjectAltName and did not replace the previous certificate in my Trusted Root Certification Authorities store. When retrieving the root certificate of the server certificate, this always produced the outdated CA certificate contained in the certificate store rather than the new one used to sign the server (nginx.crt) and client certificates.

Wayne Zhang

unread,
Apr 3, 2020, 1:54:06 PM4/3/20
to Thomas Barnekow, Google Cloud Endpoints
I am glad to hear that every works for you now.  Congratulations.  Thanks.  -Wayne

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
--
--
Baojun Xu

SolidusLink AG
Rütistrasse 16
8952 Schlieren

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/880489c7-9547-4d85-ac1e-7918e3f3ac83%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages