Redis is very slow though envoy

[Tomasz Chmielewski]

I have the following redis config on a busy-ish server (below).

Unfortunately, redis behaves slow, and I'm not sure how I can debug this.

1. Querying each redis server with redis-cli is reproducibly very fast, i.e.:

$ time echo HGETALL some:thing:here | redis-cli -h redis1.tld -p 6379

(empty list or set)

real    0m0.004s

user    0m0.004s

sys     0m0.001s

2. Querying through envoy is rarely fast, and mostly slow:

$ time echo HGETALL some:thing:here | redis-cli -h 127.0.0.1 -p 1999

(empty list or set)

real    0m0.357s

user    0m0.001s

sys     0m0.005s

$ time echo HGETALL some:thing:here | redis-cli -h 127.0.0.1 -p 1999

(empty list or set)

real    0m0.214s

user    0m0.001s

sys     0m0.005s

$ time echo HGETALL some:thing:here | redis-cli -h 127.0.0.1 -p 1999

(empty list or set)

real    0m0.005s

user    0m0.005s

sys     0m0.001s

How can I debug this further?

Config below:

static_resources:

 listeners:

 - name: redis_listener

   address:

     socket_address:

       address: 127.0.0.1

       port_value: 1999

   filter_chains:

   - filters:

     - name: envoy.filters.network.redis_proxy

       typed_config:

         "@type": type.googleapis.com/envoy.extensions.filters.network.redis_proxy.v3.RedisProxy

         stat_prefix: egress_redis

         settings:

           op_timeout: 5s

         prefix_routes:

           catch_all_route:

             cluster: redis_cluster

 clusters:

 - name: redis_cluster

   connect_timeout: 1s

   type: strict_dns

   lb_policy: MAGLEV

   load_assignment:

     cluster_name: redis_cluster

     endpoints:

     - lb_endpoints:

       - endpoint:

           address:

             socket_address:

               address: redis1.tld

               port_value: 6379

       - endpoint:

           address:

             socket_address:

               address: redis2.tld

               port_value: 6379

       - endpoint:

           address:

             socket_address:

               address: redis3.tld

               port_value: 6379

       - endpoint:

           address:

             socket_address:

               address: redis4.tld

               port_value: 6379

admin:

 access_log_path: "/dev/null"

 address:

   socket_address:

     address: 127.0.0.1

     port_value: 8001