Jenkins instance got erased after server reboot

[jack]

Hey guys, got a very weird case here... My jenkins master instance which was installed on a linux VM on our server got completely erased after a reboot (I ran a reboot command on that server).

When the server restarted, I noticed that my jenkins server url didn't work anymore, so I performed a java -jar jenkins.war to restart jenkins manually, but it actually asked me to enter an admin password to unblock jenkins, as if it was the very first time I installed a jenkins instance, and when i looked at the password, it was a new one, the old one doesn't work anymore.

Now what's even stranger is, after this reboot, I need to keep my ssh connection to the VM active in order to keep jenkins server on, at the moment I terminate my SSH session (closing the remote console), jenkins server is offline too. Before it was not like this, I can access jenkins server url anytime even if there's no remote session with the VM.

Although my entire jenkins server got wipped out, my /var/jenkins/workspace directory is still alive and all data inside is safe and sound.

What happened here? Why a simple VM reboot would wipe out my jenkins instance, and why would I need to keep the SSH remote session open to keep jenkins server online? Anyone experienced similar issue? Anyone knows the reason? I'd like to prevent this from happening again because impacts the workflow and takes time to all configure and debug again...

Thanks and cheers!

[Björn Pedersen]

Hi,

some simple things to check:

 * do you persist your VM state, or is the VM restarting  from  a clean snapshot?
 * You need to run java -jar...  detached, otherwise  closing the controling console (ssh-connection) will kill it.

Björn

[jack]

Thank you very much for your reply Björn.

The VM wasn't restarting from a clean snapshot, a simple reboot ought to preserve all data, as all my jenkins files are still on the server, so does the workspace directory.

Actually we took a closer look and I have found the origin of the error: I rebooted the VM because I attempted to permenantly set new JAVA_HOME globally in /etc/profile and /etc/environment. 

After reboot, these changes took effect and jenkins is offline; when I executed (detached or not) the jenkins.war, it gave me a new blank instance. When we tried to do a systemctl restart jenkins.service, it displays an error, saying that my jdk/jre folder is not safe because it is group and wolrd writable (because I obviously did a chmod 777 :/ ). So I restrainded the permission, jenkins service started again as a deamon by itself and all my jobs and configs are back ;)

Now just got one more question: where is that new blank jenkins instance I strated after the reboot (for which i also donwloaded all recommended plugins) ? I'd like to delete it... lol