Is there a way to dump Crash data on the Crashing POD before it dies.

[Vinoth Narasimhan]

Hi,

We have using kubernetes in GKE. And we running java based container in k8s.

Last week we have issue in one of our  applications that kept restarting.

While debugging we found that its killed by OOM.

Is there a way in k8s to dump all the JVM memory.threads,heap all other standard debugging message for the java app before it going to die. 

We don't know what went wrong inside the POD that makes container killed by OOM. If we have the java dump information before it dies it will be really useful for troubleshooting.

In the stackdriver logging we don't see much info about the error case.

[Matthias Rampke]

Raise your Kubernetes memory limit, or lower the JVM heap size. If the container gets OOM-killed there is nothing it can do to still dump something. By lowering the heap size you may be able to provoke an OutOfMemoryException within the JVM before it gets killed; with the right flags set it will do a heap dump then.

In general, when setting these limits, account for off-heap memory usage.

Make it heap dump onto a volumeMount (emptyDir or some network storage if you have it). If nothing else you can then `kubectl exec` in after it restarted to extract those files.

/MR

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Vinoth Narasimhan]

Thanks Matthias for your reply.

Can we add a "PreStop" hook on the POD before it going to die, to dump the heap to the emptyDir. Is this hook will execute before it crash ?

[EJ Campbell]

Perhaps one of the options here could be used?

  Configure Out Of Resource Handling

Configure Out Of Resource Handling Production-Grade Container Orchestration

-EJ

[Matthias Rampke]

No, the OOM killer is a last-resort action by the kernel because it is completely out of usable memory (for this container or globally). At this point, nothing and no one can interfere anymore, because to do so they would need memory that is not available until after the OOM killer.

/MR

[EJ Campbell]

Specifically:

Which would give you a chance to snapshot your process.

[Vinoth Narasimhan]

Thanks EJ .

We will try it.

[Vinoth Narasimhan]

Hi EJ,

The Eviction on the kubelet is meant for Node right ?.

Is this same is available for the PODs?