Proxyless gRPC services in Istio mesh

[Wesley Hartford]

I'm using the xDS support in grpc-java to build a sample project with proxyless client and server in Kotlin which participate in an Istio service mesh. My project is available here: https://github.com/wfhartford/kotlin-grpc-xds

I got everything working without too much trouble, but I'm a little concerned that things might not be working as intended.

What's working correctly:

• Client and server can communicate,
• Client requests are round-robin load-balanced across multiple service instances.

What doesn't seem right:

• A server interceptor reports that ServerCall.getSecurityLevel() returns NONE,
  
• When I configure Istio to enforce STRICT mTLS via a namespace wide PeerAuthentication resource, the client's connection to the server fails with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1]

Is this the expected behavior, or have I missed something?

Thanks for any insight you might have.

Wesley

[sanjay...@google.com]

On Wednesday, May 17, 2023 at 11:07:43 AM UTC-7 Wesley Hartford wrote:

...

What doesn't seem right:

• A server interceptor reports that ServerCall.getSecurityLevel() returns NONE,
  

Seems right when you are using InsecureChannelCredentials i.e. plaintext.

 

• When I configure Istio to enforce STRICT mTLS via a namespace wide PeerAuthentication resource, the client's connection to the server fails with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1]

You will have to modify the client code to use XdsCredentials as described in https://github.com/grpc/grpc-java/tree/master/examples/example-xds#run-the-example-with-xds-credentials . I am assuming the server is using XdsServerCredentials. 

[Wesley Hartford]

Thanks for getting back to me, Sanjay. As far as I can tell, my client and server are both using the appropriate Xds credentials:

The client code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
  Grpc.newChannelBuilder(targetUrl, XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()

The server code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45

  XdsServerBuilder.forPort(8443, XdsServerCredentials.create(InsecureServerCredentials.create()))

The insecure credentials provided to both a fallback, and it looks like the sample you linked is doing the same thing. I'm not sure why, but I'm guessing that the secure connection is failing and it is falling back to insecure. Based on the example you linked, the only other requirement is that the GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by the istio sidecar; kubectl describe pod shows that both the client and server containers have two environment variables injected:

      GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT:  true
      GRPC_XDS_BOOTSTRAP:                      /etc/istio/proxy/grpc-bootstrap.json

There are only two warning lines being logged from both the client and the server:

14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' for channel '[id: 0xba433026]'

Do you know of anything else I might be missing that is required for a secure connection?

Thanks,

Wesley

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/3e513ace-30e7-4a7d-8023-dde3a904be3cn%40googlegroups.com.

[sanjay...@google.com]

On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:

Thanks for getting back to me, Sanjay. As far as I can tell, my client and server are both using the appropriate Xds credentials:

The client code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
  Grpc.newChannelBuilder(targetUrl, XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()

The server code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45

  XdsServerBuilder.forPort(8443, XdsServerCredentials.create(InsecureServerCredentials.create()))

The insecure credentials provided to both a fallback, and it looks like the sample you linked is doing the same thing.

Yes, you are right - Xds credentials are being correctly used so that's not an issue.

I'm not sure why, but I'm guessing that the secure connection is failing and it is falling back to insecure.

No, that cannot be the case. As described here https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api fallback credentials are *not* meant to be used when secure connection fails. If you suspect that's happening can you provide some evidence (logs etc)?

 

Based on the example you linked, the only other requirement is that the GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by the istio sidecar; kubectl describe pod shows that both the client and server containers have two environment variables injected:

      GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT:  true
      GRPC_XDS_BOOTSTRAP:                      /etc/istio/proxy/grpc-bootstrap.json

The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) is required not just for security but overall XDS support. Based on your observations in the first email it looks like client and server are able to communicate based on xds configuration so that part must be working. Just to confirm your client did use the "xds:///" prefix to access your service, right? 

If you are certain that XDS is working for load balancing and plaintext communication but breaks only when mTLS is enabled it might be useful to enable debug logging on both client and server side (for the io.grpc.xds package and below). 

Also you can check the contents of /etc/istio/proxy/grpc-bootstrap.json esp. the certificate_providers part and confirm that the 3 files provided in the config have valid certificate material. Let me know if you need help debugging that part.

 

There are only two warning lines being logged from both the client and the server:

14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' for channel '[id: 0xba433026]'

I doubt this is an issue. Just to confirm you see these messages also when mTLS is not enabled, right?

[Sanjay Pujare]

(adding grpc.io group back)

On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca> wrote:

Hi,

My suggestion that the connection was falling back to insecure was not evidence based, I'm still trying to wrap my head around how all this is working.

Okay.

 

The target address on the client side is using the xds:/// prefix.

I've enabled trace level logging on the io.grpc.xds logger but I'm not seeing any additional log messages, have I missed something? I'm using slf4j and logback and have the SLF4JBridgeHandler installed.

The code uses Java util logging so you can just something like -Djava.util.logging.config.file=/logging.properties in your Java invocation command line and enable the logger for io.grpc.xds  .

 

The grpc-bootstrap.json file seems reasonable, though I don't know just what it all means (I've attached the content). The three pem files referenced in certificate_providers point to real files containing apparently valid PEM content.

You've mentioned a couple times enabling vs. disabling mTLS, are you referring to some specific setting on the client and/or server, or in istio somewhere? My understanding has been that with the Xds server and channel, both will use mTLS unless I specifically set the mtls mode to DISABLE in a PeerAuthentication resource, which I haven't done. I've experimented with mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple as not enabling mTLS somewhere?

In your first post you said "I got everything working without too much trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got the impression that you got everything working in plaintext (without enabling mTLS) and then you enabled mTLS via Istio which is when you saw connection problems. I was just referring to the same - you enable mTLS in Istio security policy.

I suggest couple of additional tests to troubleshoot this:

- instead of using Xds Channel and Server credentials use the Tls Channel and Server credentials with the same files provided by Istio (under /var/lib/istio/data). You can check out the doc at https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html and https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html . Note that even if you are using XDS for load balancing, service discovery you can use Tls credentials for security.

- if you can try a Golang (or C++) example with your Istio setup and verify mTLS is working with those examples.

Hope that helps.

 

Thanks again,

Wesley

--

You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com.

[Wesley Hartford]

I've attached a copy of the log files with xds logging set to trace for an execution of the client and server with istio's mtls mode set to STRICT and PERMISSIVE. My interpretation of these logs is:

In PERMISSIVE mode, neither client nor server is trying to use any type of TLS; they're both using plain text and can interact just fine. I was under the impression that PERMISSIVE mode would use mTLS, but reading the istio docs, it's a little ambiguous, so I guess this is a correct behaviour.

In STRICT mode, it looks like the server is using the supplied TLS key material to accept mTLS connections, but it doesn't look like the client is making any attempt to use TLS.

I'm working on your other suggestions, but thought I should update you with the logs before spending too much time on them.

[sanjay...@google.com]

I looked at the logs and I can confirm that the client is not using mTLS because Istio didn't provide the right configuration. Let me explain:

From your server-mtls-strict.log I see this 

      "transportSocket": {
        "name": "envoy.transport_sockets.tls",
        "typedConfig": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
          "commonTlsContext": {
            "combinedValidationContext": {
              "defaultValidationContext": {
              },
              "validationContextCertificateProviderInstance": {
                "instanceName": "default",
                "certificateName": "ROOTCA"
              }
            },
            "tlsCertificateCertificateProviderInstance": {
              "instanceName": "default",
              "certificateName": "default"
            }
          },
          "requireClientCertificate": true
        }
      },
      "name": "inbound-mtls"

In the Listener resource received (see timestamp 16:47:54.070)

However I don't see a similar security configuration on the client side - which is a bit complex I admin. In the client-mtls-strict.log I see a Cluster resource response:

{
  "versionInfo": "2023-05-26T16:47:41Z/749",
  "resources": [{
    "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
    "name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local",
    "type": "EDS",
    "edsClusterConfig": {
      "edsConfig": {
        "ads": {
        }
      },
      "serviceName": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local"
    }
  }],
  "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255",
  "controlPlane": {
    "identifier": "{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}"
  }
}

which has no security/mTLS configuration. I would expect a config similar to the transportSocket snippet I described above.

So the error is on the Istio side: whether a bug in Istio or a user-error I can't tell. I hope you are able to figure it out?

[Xavier XYZ]

Hi, I have exactly the same issue, did you find any solution ? 

Thank you

[Xavier XYZ]

I finally find the solution after some debugging hours.

Istio provides by default a RSA key, when the grpc-java lib is waiting for a PKCS#8 key.

Add environment variable `PKCS8_KEY=true` to the istio proxy solved the issue.

Hope this help.