Two endpoints in a cluster, one must use SNI, the other not

[Dino Chiesa]

I would like to configure a cluster , type STRICT_DNS, with 2 upstream lb_endpoints, both of which will use TLS. One of these requires SNI and the other does not. 

client ---(http)---> envoy ---(https) ---> {upstream1, upstream2}

The configuration below shows what I am trying.  When envoy proxies to the endpoint that does not require SNI, it all works. When envoy proxies the request to the endpoint that requires SNI, the TLS handshake with the upstream fails. 

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE 268435610:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO

dns logging shows the DNS lookup for both endpoints has succeeded. The DNS names resolve to different IPv4 addresses.

Is it possible to configure a cluster to allow this?   Any hints?

#### config snip

    load_assignment:
      cluster_name: service1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:

                address: requires-sni.example.com
                port_value: 443
        - endpoint:
            address:
              socket_address:
                address: no-sni-needed.example.io
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context:
            trusted_ca:
              filename: /usr/local/etc/openssl/cert.pem

[Dino Chiesa]

Related question: 

Is there a configuration for Envoy that tells it to use the hostname of the endpoint to use for  TLS negotiation , whether SNI or not? 

In other words, I would like to configure a cluster with multiple endpoints, and one or more of them uses SNI, and I want the TLS to those upstreams to be negotiated (SNI or otherwise) based on the upstream endpoint hostname.  Is such a configuration possible, and if so, please show me. 

If I am asking a ridiculous question, please let me know. 

[Yan Avlasov]

Strictly speaking I do not think you can do something like this now. The upstream TLS context https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#extensions-transport-sockets-tls-v3-upstreamtlscontext has the "sni" configuration parameter, but it will not work if your cluster has multiple endpoints with different hostnames. You can make it work by setting the SNI in the upstream TLS context to requires-sni.example.com and since your other host does not care about SNI it should just ignore it (I assume this).

I also suggest opening an Issue on GitHub to add support for your use case.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/0db777ef-ebd1-4145-b9d5-85b74c4755acn%40googlegroups.com.

[Dino Chiesa]

I tried specifying just the sni but that didn't work either. 

WHAT DID WORK, is specifying weighted_clusters in the route, and placing each upstream in a different cluster. 

            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  auto_host_rewrite: true
                  weighted_clusters:
                    clusters:
                    - name: cluster1
                      weight: 1
                    - name: cluster2
                      weight: 1
...

  clusters:
  - name: cluster1
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    dns_refresh_rate: 90s
    load_assignment:
      cluster_name: cluster1

      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:

                address: endpoint.example.io
                port_value: 443
          load_balancing_weight: 1

    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context:
            trusted_ca:
              filename: /usr/local/etc/openssl/cert.pem

        sni: endpoint.example.io
  - name: cluster2
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    dns_refresh_rate: 90s
    load_assignment:
      cluster_name: cluster2

      endpoints:
      - lb_endpoints:
        - endpoint:

            hostname: "requires-sni.example.com"

            address:
              socket_address:
                address: "requires-sni.example.com"
                port_value: 443

          load_balancing_weight: 1

    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context:
            trusted_ca:
              filename: /usr/local/etc/openssl/cert.pem

        sni: requires-sni.example.com

[Yan Avlasov]

Glad to hear it worked. Please file an Issue on GitHub for non-working configuration.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/408179f2-2438-4699-bac4-643161c0791cn%40googlegroups.com.