Request timeouts for Envoy service mesh

[Yifan Mai]

Hi,

I'm trying to use the x-envoy-upstream-rq-timeout-ms HTTP header to set request timeouts in Envoy. I am having trouble getting this to work in a service mesh configuration, in which requests is sent to an egress Envoy listener, which forwards to an ingress Envoy listener, which forwards it to a target upstream server.

The reason for this is that the egress Envoy listener strips out the x-envoy-upstream-rq-timeout-ms, so the ingress Envoy listener never receives this header, which means that it will always use the default timeout. The egress Envoy listener adds a x-envoy-expected-rq-timeout-ms header, but this does not affect the ingress Envoy listener's behavior.

I have a test setup in which I have a very slow target upstream server running on port 9000, and I have the first (ingress) Envoy listener proxy port 30002 to port 9000 (the slow target server) and the second (egress) Envoy listener proxy port 30003 to port 30002 (the first Envoy listener). The whole config YAML file is below.

If I send a request to the first (ingress) Envoy listener proxy, the request succeeds:

curl localhost:30002 --header 'x-envoy-upstream-rq-timeout-ms: 20000'

But if I send a request to the second (egress) Envoy listener proxy, the request times out:

curl localhost:30003 --header 'x-envoy-upstream-rq-timeout-ms: 20000'

Is there a way to get header-based request timeouts to work in a service mesh configuration?

Regards,

Yifan

---

admin:

  access_log_path: /mnt/logs/envoy/access_admin.log

  address:

    socket_address:

      address: 0.0.0.0

      port_value: 30001

static_resources:

  listeners:

    - address:

        socket_address:

          address: 0.0.0.0

          port_value: 30002

      filter_chains:

      - filters:

        - name: envoy.http_connection_manager

          config:

            use_remote_address: true

            codec_type: auto

            stat_prefix: ingress_http

            route_config:

              name: target_slow

              virtual_hosts:

              - name: target_slow

                domains:

                - "*"

                routes:

                - match:

                    prefix: "/"

                  route:

                    cluster: target_slow

            http_filters:

            - name: envoy.router

              config: {}

    - address:

        socket_address:

          address: 0.0.0.0

          port_value: 30003

      filter_chains:

      - filters:

        - name: envoy.http_connection_manager

          config:

            use_remote_address: true

            codec_type: auto

            stat_prefix: ingress_http

            route_config:

              name: envoy_chain

              virtual_hosts:

              - name: envoy_chain

                domains:

                - "*"

                routes:

                - match:

                    prefix: "/"

                  route:

                    cluster: envoy_chain

            http_filters:

            - name: envoy.router

              config: {}

  clusters:

  - name: target_slow

    connect_timeout: 1s

    type: STATIC

    lb_policy: ROUND_ROBIN

    hosts:

    - socket_address:

        address: 127.0.0.1

        port_value: 9001

  - name: envoy_chain

    connect_timeout: 1s

    type: STATIC

    lb_policy: ROUND_ROBIN

    hosts:

    - socket_address:

        address: 127.0.0.1

        port_value: 30002

[Matt Klein]

What we do at Lyft is have our egress listeners configured to "use remote address." This picks up the localhost address of the sender, which is an internal address, so that trusted headers are not sanitized. See https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/header_sanitizing.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAOhh5XixNDiPtZmU8NAOWc03UBnzSj6M5Uj8kU0V6AKbTEHZuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Yifan Mai]

The use_remote_address was definitely a gotcha for us in that we did not realize at first that we needed to set the flag to true to get the header directives to work.

However, the problem I was originally asking about is the use case in which a request goes through through two Envoy proxies (an egress proxy and then an ingress proxy), in which case the header directive is stripped out by the egress proxy and never reaches the ingress proxy. Also, this stripping happens during the request timeout computation logic, and is separate from the header sanitizer logic. This is preventing us from getting the timeout header to work in the service mesh use case.

[Matt Klein]

Sorry I misunderstood. So you are wanting the timeout header to get proxied to the next hop so that hop picks up the timeout? Yeah, this is not supported today. I would open an issue on this for discussion.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAOhh5XixNDiPtZmU8NAOWc03UBnzSj6M5Uj8kU0V6AKbTEHZuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[rhernan...@gmail.com]

I am not sure if this functionality was ever formally request, so I'm adding this for posterity ...

I believe this Lua filter (added to an ingress listener), will propagate the timeout to upstream clusters:

          - name: envoy.lua

            typedConfig:

              '@type': type.googleapis.com/envoy.config.filter.http.lua.v2.Lua

              inline_code: |

                function envoy_on_request(request_handle)

                  local expected_timeout = request_handle:headers():get("x-envoy-expected-rq-timeout-ms")

                   if expected_timeout ~= nil then

                     request_handle:headers():add("x-envoy-upstream-rq-timeout-ms", expected_timeout)

                   else

                    request_handle:headers():add("x-envoy-upstream-rq-timeout-ms", "15000")           

                   end

                end

Caveat: Envoy will set `x-envoy-expected-timeout-ms: 15000` by default, so if the initial request does not set a timeout (or the egress route does not have a timeout configured), it will simply propagate 15 seconds to the upstream endpoint (which may not be desired).

[gus...@iterativo.do]

Hi,

I've been struggling with this same issue for several days.

What's the complete Filter configuration that you used to fix it? As I can only see a fraction of it.

Regards