Container termination force pod termination?

[David Rosenstrauch]

I have a pod which runs a single container. The pod is being run under
a ReplicaSet (which starts a new pod to replace a pod that's
terminated).


What I'm seeing is that when the container within that pod terminates,
instead of the pod terminating too, the pod stays alive, and just
restarts the container in it. However I'm thinking that what would make
more sense would be for the entire pod to terminate in this situation,
and then another would automatically start to replace it.

Does this seem sensible? If so, how would one accomplish this with k8s?
Changing the restart policy setting doesn't seem to be an option. The
restart policy (e.g. Restart=Always) seems to apply only to whether to
restart a pod; the decision about whether to restart a container in a
pod doesn't seem to be configurable. (At least not that I could see.)

Would appreciate any guidance anyone could offer here.

Thanks,

DR

[David Rosenstrauch]

Was speaking to our admin here, and he offered that running a health
check container inside the same pod might work. Anyone agree that that
would be a good (or even preferred) approach?

Thanks,

DR

[Rodrigo Campos]

I don't think it is configurable.

But I don't really see what you are trying to solve, maybe there is another way to achieve it? If you are running a pod of a single container, what is the problem that the container is restarted when is appropriate instead of the whole pod?

I mean, you would need to handle the case where some container in the pod crashed or is stalled, right? The liveness probe will be done periodically, but until the next check is done, it can be hunged or something. So even if the whole pod is restarted, that problem is still there. And restarting the whole pod won't solve that. So probably my guess is not correct about what you are trying to solve.

So, sorry, but can I ask again what is the problem you want to address? :)

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Tim Hockin]

What Rodrigo said - what problem are you trying to solve?

The pod lifecycle is defined as restart-in-place, today. Nothing you
can do inside your pod, except deleting it from the apiserver, will do
what you asking. It doesn't seem too far fetched that a pod could
exit and "ask for a different node", but we're not going there without
a solid solid solid use case.

>> email to kubernetes-use...@googlegroups.com.
>> To post to this group, send email to kubernet...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

[David Rosenstrauch]

Well restarting the pod actually does have a better chance of fixing
whatever the issue is, rather than just restarting the container inside
of it. The pod might very well get restarted on a different machine.
If the machine the pod is running on is either down or hurting, then
just restarting the container won't really help you.

>> send an email to kubernetes-use...@googlegroups.com.

>> To post to this group, send email to

>> kubernet...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users

>> [1].
>> For more options, visit https://groups.google.com/d/optout [2].

>
> --
> You received this message because you are subscribed to the Google
> Groups "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to kubernetes-use...@googlegroups.com.

> To post to this group, send email to

> kubernet...@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>
>

> Links:
> ------
> [1] https://groups.google.com/group/kubernetes-users
> [2] https://groups.google.com/d/optout

[Tim Hockin]

If the machine is down, k8s will automatically move all pods off of it, anyway.

[David Rosenstrauch]

I'm trying to make sure that as I'm deploying new services on our
cluster, that failures/restarts get handled in a way that's most optimal
for resiliency/uptime.


I'm simplifying things a bit, but if a piece of code running inside a
container crashes, there's more or less 2 possibilities: 1) bug in the
code (and/or it's trying to process data that causes an error), or 2)
problems with the hardware/network (full disk, bad disk, network outage,
etc.) If the issue is #1, then it doesn't matter whether you restart
the container or the pod. But if the issue is #2, then restarting the
pod (i.e., on another host) would fix the problem, while restarting the
container probably wouldn't.

So I guess this is sort of alluding to a bigger question, then: does
k8s have any ability to detect if a host is having hardware problems
and, if so, avoid scheduling new pods on it, move pods off of it if
their containers are crashing, etc.

I've done a lot of work with big data systems previously and, IIRC,
Hadoop (for example) used to employ procedures to detect if a disk was
bad, if many tasks on a particular node kept crashing, etc., and it
would start to blacklist those. My thinking was that k8s worked
similarly - i.e., if all containers in a pod terminated unsuccessfully,
then terminate the pod; if a particular node is having many pods
terminated unsuccessfully, then stop launching new pods on there, etc.
Perhaps I'm misunderstanding / assuming incorrectly though.

Thanks,

DR

On 2017-10-27 4:35 pm, 'Tim Hockin' via Kubernetes user discussion and

[Rodrigo Campos]

(google groups is taking days when I use my non-gmail email, so I'm
sending via gmail again)

On Fri, Oct 27, 2017 at 6:17 PM, David Rosenstrauch <dar...@darose.net> wrote:
> I'm trying to make sure that as I'm deploying new services on our cluster,
> that failures/restarts get handled in a way that's most optimal for
> resiliency/uptime.
>
>
> I'm simplifying things a bit, but if a piece of code running inside a
> container crashes, there's more or less 2 possibilities: 1) bug in the code
> (and/or it's trying to process data that causes an error), or 2) problems

It can be a "random" issue (like a network burp, etc.), or also exceeding memory
limits of the container and gets restarted (that happens when X event is
processed in the container and uses tons of mem), etc, etc. Which will, most
probably, work if restarted.

> with the hardware/network (full disk, bad disk, network outage, etc.) If

As Tim said, network outage is handled (like if the node <--> master network is
not working) just fine. Full disk should be handled fine since a few versions of
kubernetes now, as there is accounting of inodes and space used by containers,
so it can be reclaimed too.

> the issue is #1, then it doesn't matter whether you restart the container or
> the pod. But if the issue is #2, then restarting the pod (i.e., on another
> host) would fix the problem, while restarting the container probably
> wouldn't.
>
> So I guess this is sort of alluding to a bigger question, then: does k8s
> have any ability to detect if a host is having hardware problems and, if so,
> avoid scheduling new pods on it, move pods off of it if their containers are
> crashing, etc.

I know of https://github.com/kubernetes/node-problem-detector, that I think
tries to solve exactly that. But I have not used that myself.

So I guess the answer is "yes" :-)



Thanks,
Rodrigo

[Rodrigo Campos]

On Fri, Oct 27, 2017 at 1:34 PM, David Rosenstrauch <dar...@darose.net> wrote:
> Was speaking to our admin here, and he offered that running a health check
> container inside the same pod might work. Anyone agree that that would be a
> good (or even preferred) approach?

Not sure what you mean, but IIUC, it won't get the pod scheduled to
another node.

[Tim Hockin]

On Fri, Oct 27, 2017 at 2:17 PM, David Rosenstrauch <dar...@darose.net> wrote:
> I'm trying to make sure that as I'm deploying new services on our cluster,
> that failures/restarts get handled in a way that's most optimal for
> resiliency/uptime.
>
>
> I'm simplifying things a bit, but if a piece of code running inside a
> container crashes, there's more or less 2 possibilities: 1) bug in the code
> (and/or it's trying to process data that causes an error), or 2) problems
> with the hardware/network (full disk, bad disk, network outage, etc.) If
> the issue is #1, then it doesn't matter whether you restart the container or
> the pod. But if the issue is #2, then restarting the pod (i.e., on another
> host) would fix the problem, while restarting the container probably
> wouldn't.

We automatically detect things like full disks and network outages and
remove or repair those nodes. Are you working around a known problem
or a hypothetical?

> So I guess this is sort of alluding to a bigger question, then: does k8s
> have any ability to detect if a host is having hardware problems and, if so,
> avoid scheduling new pods on it, move pods off of it if their containers are
> crashing, etc.

yes. Node Problem Detector detects a number of issues and responds.
GKE's NodeAutoRepair will automacially rebuild nodes when it detects
problems.

> I've done a lot of work with big data systems previously and, IIRC, Hadoop
> (for example) used to employ procedures to detect if a disk was bad, if many
> tasks on a particular node kept crashing, etc., and it would start to
> blacklist those. My thinking was that k8s worked similarly - i.e., if all
> containers in a pod terminated unsuccessfully, then terminate the pod; if a
> particular node is having many pods terminated unsuccessfully, then stop
> launching new pods on there, etc. Perhaps I'm misunderstanding / assuming
> incorrectly though.

We probably should have a crash-loop mode that kills the pod and lets
the scheduler re-assess. AFAIK, we don't do that today, but it hasn't
been a huge problem.

[Rodrigo Campos]

On Fri, Oct 27, 2017 at 05:17:48PM -0400, David Rosenstrauch wrote:
> I'm trying to make sure that as I'm deploying new services on our cluster,
> that failures/restarts get handled in a way that's most optimal for
> resiliency/uptime.
>
>
> I'm simplifying things a bit, but if a piece of code running inside a
> container crashes, there's more or less 2 possibilities: 1) bug in the code
> (and/or it's trying to process data that causes an error), or 2) problems

It can be a "random" issue (like a network burp, etc.), or also exceeding memory
limits of the container and gets restarted (that happens when X event is
processed in the container and uses tons of mem), etc, etc. Which will, most
probably, work if restarted.

> with the hardware/network (full disk, bad disk, network outage, etc.) If

As Tim said, network outage is handled (like if the node <--> master network is
not working) just fine. Full disk should be handled fine since a few versions of
kubernetes now, as there is accounting of inodes and space used by containers,
so it can be reclaimed too.

> the issue is #1, then it doesn't matter whether you restart the container or
> the pod. But if the issue is #2, then restarting the pod (i.e., on another
> host) would fix the problem, while restarting the container probably
> wouldn't.
>
> So I guess this is sort of alluding to a bigger question, then: does k8s
> have any ability to detect if a host is having hardware problems and, if so,
> avoid scheduling new pods on it, move pods off of it if their containers are
> crashing, etc.