Load Balancing and Connection pooling in Envoy Proxy

[Prateek Jain]

Hi folks,

We are building an edge proxy using Envoy which dynamically routes traffic over upstream clusters. We have a cluster discovery service which defines active healthy cluster hosts over which traffic can be re-routed to. We have following strategies to tackle this design:

1. We can use Strict DNS route type for service discovery. In this case, DNS resolver service will call our cluster discovery service at regular intervals and fetch active healthy hosts for the cluster. This way, Envoy will get exposed to healthy hosts and can do traffic load balancing over these hosts.

2. We use xDS streaming APIs. In this case, envoy will create a streaming connection with the xDS service and the service API will push fresh cluster configuration at regular intervals.

Now we have a couple of open questions:

1. In case of xDS APIs, what is the behavior for connection management? Do envoy drains connections created with removed hosts? Is there any mechanism to set keep alive configuration for such connections?

2. Suppose at point T1, Envoy has a cluster configuration with hosts (h1, h2, h3). After next refresh Envoy gets a new configuration with (h1, h3). Envoy will drain allotted requests to h2 and route fresh traffic to h1 and h3. Now suppose in next refresh, h2 reappears in the configuration, will Envoy reuse earlier connections for traffic routing or will it re-create new connections to do so. Is there a way to setup keep alive configuration for such connections to avoid simultaneous connection close and open on same host.

3. Is there any mapping of downstream connection to upstream connection? We want to know about this because we have high number of low TPS clients and want to understand connection management behavior. 

Additional info:

Number of downstream clients : in range of 10k

Number of upstream services : 6

Average Number of hosts in each upstream service : 1000

Average TPS : 300k-400k

Please do let us know.

Thanks in advance.

[Prateek Jain]

Bumping this one again to get some attention.

Thanks,

Prateek Jain

[Stephan Zuercher]

1/2. Hosts removed from a cluster have their connections drained and closed by default. However, in the case of EDS, if active health checking is enabled and the host is in a healthy state it will remain in the cluster until health checks fail. You can disable that behavior with the drain_connections_on_host_removal option on the cluster. If the conditions are met to keep the host in the cluster post removal (EDS+active health checks+healthy+drain_connections_on_host_removal=false), the host's connections will remain open after removal and will continue to be used to serve requests until health checking fails and it is removed. There's no in-between state where the connections are kept open but left idle.

3. Downstream HTTP requests are individually mapped to a cluster (via route configuration) and load balanced across the hosts in that cluster based on the configured load balancer policy. Given sufficient request volume, multiple requests on a single downstream connection are as likely to be routed to different upstream hosts as multiple requests on separate connections. (With very few requests, you may notice effects related to thread-local state in load balancers.) Upstream connections are pooled per host. There are many configuration options on clusters (http idle timeouts, max requests per connection, circuit breakers and probably others) that alter the behavior around the connection pools.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CA%2BR-mqgXzJtkMH%2BQ4gGTw1H5tS5czsQddd6_gckzeG-nSx1FGA%40mail.gmail.com.