Java VM crash (because of maintenance migration?)

[Thomas Baldauf]

Hi!

This morning I woke up encountering my GCE app down as a result of a Java VM crash (see log output). The incident happened at 12:28 AM UTC. I'm running a Bitnami Tomcat Stack (from Cloud Launcher) and could see that there was also a maintenance event going on:

12:10 AM tomcatstack-1-vm instance migrated during Compute Engine maintenance

Could the Java VM crash be connected to the migration? If so, how can it be mitigated in the future? In the morning I first tried to restart the Tomcat server. I worked at first, but after a few requests there was another crash. Only since I restarted my GCE instance everything seems to be running smoothly again. 

Any help and insight into this is very much appreciated!

Thomas

########################################################################

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x00000000e03078b8, pid=16841, tid=0x00007ff09d183700

#

# JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)

# Problematic frame:

# C  0x00000000e03078b8

#

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

# If you would like to submit a bug report, please visit:

#   http://bugreport.java.com/bugreport/crash.jsp

#

---------------  T H R E A D  ---------------

Current thread (0x00007ff0d0551800):  JavaThread "ajp-apr-8009-AsyncTimeout" daemon [_thread_in_Java, id=16876, stack(0x00007ff09d083000,0x00007ff09d184000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00000000e03078b8

...

< to be continued on request >

########################################################################

[Faizan (Google Cloud Support)]

Hello Thomas,

By default, standard instances are set to live migrate, where Google Compute Engine automatically migrates your instance away from an infrastructure maintenance event, and your instance remains running during the migration.

With that said, this issue could be related to VM migration. Did you check the VM console output logs to confirm it was booted properly and all the services were running?

Faizan

[Thomas Baldauf]

Hi Faizan,

I requested the VM console output logs as described in your link, but I could only find the logs from my manual restart of the VM in the morning after the crash/migration. However, before restarting the VM I could see that Apache was running properly (returning a 503), but Tomcat obviously was down because of the Java VM crash. 

Thomas

[Faizan (Google Cloud Support)]

Hello Thomas, 

Can you send me your project ID and instance name through private message? I'll investigate further to see if this issue was related to migration event.

Faizan

[Faizan (Google Cloud Support)]

Hello Thomas,

I have investigated this further with my backend team and can confirm that the live migration was normal. We were not able to find any correlation between your application crash and live migration. This looks like more of an application specific error. As such, I would recommend monitoring your instance to see if it reoccurs. If it does, you can reach out through GCE public issue tracker providing details information including the logs for further investigation.

I hope that helps.

Faizan