Graceful termination in Kubernetes

[Mark Vincze]

Hey all,

I have a question about gracful termination when running Envoy in Kubernetes.

I've been running Envoy in Kubernetes to proxy to a set of upstreams based on an incoming query string argument. (I have a very simple setup, I'm running the normal envoyproxy/envoy docker image with a static yaml configuration.)

It's been working nicely, but there is one issue: whenever I do a rolling update release of a new version of my pod, I always see a couple of requests logged by the downstream as failed with 499 Client Closed Request, which I believe means that the pod running Envoy was stopped while there were still a couple of requests in flight.

I tried to alleviate this by adding the "usual" sleep command as a preStop hook of the pod:

        lifecycle:

          preStop:

            exec:

              command: ["/bin/sleep", "30"]

And this seems to indeed get rid of the 499 failures.

My question is: is this the proper solution? Or should Envoy by default capture the SIGTERM coming from Kubernetes, and not stop until the outstanding requests are drained out? Or this should be enabled somehow in the configuration?

Thanks,

Mark

[Harvey Tuch]

I'm not a k8s expert (or even user), but one option you have is to mark the Envoy as unhealthy, at which point it will drain, see https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/draining.html?highlight=drain. 

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/d199842f-5b92-470f-9d21-be53011102a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Vincze]

Hey Harvey,

Thanks for the info. I'm a bit confused by what "draining" would mean in this scenario.

The way this works in my setup is that I have 3 Envoy instances (called pods in Kubernetes), which have a Kubernetes load balancer in front of them.

And when I'm doing a rolling update, one pod at a time, Kubernetes removes the pod from the LB, and sends a SIGTERM to the container (in this case to the 'envoy' process). Then it waits 30 seconds, and if the process didn't exit by that time, then sends a SIGKILL. (So we don't have to worry about the incoming requests, because Kubernetes removes the pod from the LB at the beginning of the process, so it doesn't receive any new requests.)

What a typical problem can be (not just in Envoy, but with any service) is that when the SIGTERM is sent, there might be still requests in flight. And if the service doesn't wait for them to finish, but it terminates on the SIGTERM immediately, then those requests fail.

So this "/bin/sleep 30" command in the preStop hook is a usual trick to use when the service doesn't properly handle the SIGTERM this way, this way Kubernetes waits for 30 seconds before sending the SIGTERM.

What I don't know is: how does Envoy handle a SIGTERM signal? Does it stop the process immediately, or does it wait for the current requests to finish?

Thanks,

Mark

[Tony Allen]

Hey Mark,

It looks like SIGTERM is caught in server.cc:

  sigterm_ = dispatcher.listenForSignal(SIGTERM, [this, &hot_restart, &dispatcher]() {

    ENVOY_LOG(warn, "caught SIGTERM");

    shutdown_ = true;

    hot_restart.terminateParent();

    dispatcher.exit();

  }); 

It looks like the real work is in `terminateParent()` which is supposed to be a graceful termination. Following the code path, it looks like it ultimately results in the following block in the socket handler on the parent:

    case RpcMessageType::TerminateRequest: {

      ENVOY_LOG(info, "shutting down due to child request");

      kill(getpid(), SIGTERM);

      break;

    } 

My understanding of that is that envoy isn't waiting for the connections to drain. This is the same thing that occurs when we want to shutdown the server gracefully as well. You may also want to take a look this ticket regarding graceful shutdown:

https://github.com/envoyproxy/envoy/issues/1990

-Tony 

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CADXhE3KAzz0eL1bkAPzvigz0kPe-CSbJUd2_qmxKZNiEpk7XJA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Tony Allen Software Engineer

[Mark Vincze]

Hi Tony,

Thanks for the detailed answer!

>My understanding of that is that envoy isn't waiting for the connections to drain.

Then I think using the "/bin/sleep 30" preStop hook is indeed the proper workaround in Kubernetes.

Cheers,

Mark

[Harvey Tuch]

If you invoke the admin handler I point to to mark Envoy unhealthy, it will not accept any further requests but will continue to service in-flight requests. You can replace the SIGTERM step with this admin handler POST and then do a SIGTERM later, after the drain period.

[Mark Vincze]

Hi Harvey,

I'm not sure this would work properly in Kubernetes. As far as I know we don't have much control over how or when the SIGTERM is called, the only thing we can do is delay it by executing a command in the preStop hook which takes a lot of time (like /bin/sleep), which will make the SIGTERM be delayed by that duration.

So if we did something like this in the preStop hook:

        lifecycle:

          preStop:

            exec:

              command: ["curl", "http://localhost:9901/healthcheck/fail"]

After this command returns (this endpoint is not blocking until the draining is done, right?), Kubernetes would immediately send the SIGTERM, so this wouldn't make us wait for the drainining to finish. (And we don't have to worry about Envoy not accepting any further requests, because at this point Kubernetes has already removed the pod from the LB pool.)

If there was a way to actually block until the draining is done, that could be used instead of the more blunt /bin/sleep approach I guess.

(Or if Envoy would wait for the draining to finish after a SIGTERM, then we wouldn't have to write a custom hook, that would do the right thing by default.)

Cheers,

Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAA4W8ZnV1NK6%3D1GMWgs9ThzVGArLHM%2Boy5GQoxus2m9LmmTirg%40mail.gmail.com.

[Harvey Tuch]

Can't you just tell it to curl on that URL and then sleep for N seconds?

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CADXhE3LSqacqbNasd358zv2MQU-3uNZAGzG%3DNCJP01DENvPFYg%40mail.gmail.com.

[Mark Vincze]

Yeah, I guess I could have multiple commands there (or just run a shell script which does this), but at that point, does calling this endpoint gives us anything useful?

[Harvey Tuch]

It provides graceful drain; Envoy will issue an HTTP/2 GOAWAY for example, allowing the client to know to avoid sending further requests.