Simpler listener configuration for multiple domains with TLS support

[Shams Imam]

tl;dr What is the simplest way to provide a mapping of domains to certificates and avoid generating repeated (one that can easily be templated by the domain+cert combination) filter chain configurations in the Envoy listener config?

Details:

The example for configuring multiple domains suggests configuring multiple filter chains. For example, using the examples in this document for two custom domains we have something like in our listener configurations:

filter_chains:
- filter_chain_match:
    server_names: ["foo.example.com"]
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/foo/cert.pem" }
          private_key: { filename: "/path/foo/key.pem" }
  # other configuration
  ...
- filter_chain_match:
    server_names: "bar.example.com"
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/bar/cert.pem" }
          private_key: { filename: "/path/bar/key.pem" }
  # other configuration (repeat of what we have for foo.example.com
  ...
# repeat for every other domain

When configuring listeners for many domains (in the order of thousands) this can become quite cumbersome as there is one entry per domain name with repeated associated configuration on other options and yields a very large configuration file (in the order of MBs). In addition, anytime a domain name gets added or removed, the configuration has to be regenerated (for LDS). Separately, use of CommonTlsContext means reloading the certificate needs an Envoy hot-restart. 

The Envoy documentation of Certificate Management says:

Secret Discovery Service referenced certificates. By using SDS, certificates can either be referenced as files (reloading the certs when the parent directory is moved) or through an external SDS server that can push new certificates.

We explored the use of SDS file as documented in this article:

 common_tls_context:
     alpn_protocols: ["h2", "http/1.1"]
     tls_certificate_sds_secret_configs:
         sds_config:
             path: /etc/envoy/sds.yaml

But, SDS yaml files can contain only a single entry, and attempting to configure multiple domain mapping fails with the following message:

[2021-03-16 23:40:35.661][909395][warning][config] [source/common/config/filesystem_subscription_impl.cc:43] Filesystem config update rejected: Unexpected SDS secrets length: 251

As a result, using SDS with the file-based approach means we need to generate additional files, one per domain, to support dynamic refresh of certificates on top of the filter chain listener configurations described above.

Question:

What is the better alternative for configuring listeners for multiple domains along with TLS cert configuration? Ideally, we avoid repeating filter chain configurations and generating multiple files for certificates.

Relevant feature in HAProxy:

We have previously used HAProxy for similar configurations where a common “listener” configuration can be provided for multiple domains with the per domain certificate lookup dynamically configured via an external table (crt-list option). This yields a much smaller configuration file, in the order of KBs when everything is combined.

Thanks in advance for your help.

[Yuchen Dai]

There are several issues

>> When configuring listeners for many domains (in the order of thousands) this can become quite cumbersome as there is one entry per domain name with repeated associated configuration on other options and yields a very large configuration file (in the order of MBs)

ECDS helps a bit, by referring to the network filter(http_conn_manager in your case) and any http filter's config by name. 

>> Separately, use of CommonTlsContext means reloading the certificate needs an Envoy hot-restart. 

Not sure what is restarting the Envoy. When you add/delete domains in this listener config through grpc LDS, the inplace listener update should be executed. Thus no envoy restart is expected. The connections of other domains are not drained.

>> Multi cert per SDS config

I don't have a good amount of security contexts. The smarter people on SDS might have some ideas. 

-Yuchen

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/0404014e-b972-410f-8507-26f1a29882c8n%40googlegroups.com.

[Shams Imam]

> ECDS helps a bit, by referring to the network filter(http_conn_manager in your case) and any http filter's config by name.

I was unable to find an example, but reading what little I was able to find about it looks promising. Do you have an idea what the example would look like for the dummy configuration with foo.example.com and bar.example.com I have above? A link to a good reference would also be much appreciated.

> Not sure what is restarting the Envoy. When you add/delete domains in this listener config through grpc LDS, the inplace listener update should be executed. Thus no envoy restart is expected. The connections of other domains are not drained.

We use file-based LDS, the restart is not for adding/removing domains. The restart is required for picking up updates to the SSL certificates, e.g. that extends the expiry date in a new certificate at the same path. Based on the documentation and also empirical evidence, Envoy does not support reloading of new certificates when using CommonTlsContext.

Regards,

Shams.

[Shams Imam]

Cross posted in https://github.com/envoyproxy/envoy/issues/16174 in the hopes of getting more responses.

Regards,

Shams.