Jenkins Backup and Restore

[gotvi...@gmail.com]

We have a Jenkins instance running and we are trying to setup a backup and restore mechanism in case of any mishaps.  We are looking for a full backup of the Jenkins and so decided to go with PeriodicBackup plugin.  With this plugin, the backup works fine.  But, we are trying to restore it in a completely new Jenkins setup and not able to restore.  When we pick the backup file that we want to restore from, it says restoring, but nothing happens.  Even after waiting out for over 30 mins, nothing is restored.  As a work around, I stopped the Jenkins service, renamed the 'JENKINS_HOME' (/var/lib/jenkins) folder to 'jenkins_backup' and created another folder called 'jenkins' in the same location.  In this folder, I copied all the files from the backup and then started Jenkins (NOTE: I didn't change the .owner file.  I'm using the .owner file of the new Jenkins only).  Jenkins came up and I was able to see all the configurations from the old one.  Everything was fine.  I let it like that overnight and in the morning when I came and tried to access it, Jenkins was way too slow and I was not able to save any new configuration.  When I try I'm getting 

java.io.ioexception: too many open files jenkins

error.  I restarted the jenkins and it's taking too long for the UI to come up.  Am I doing anything wrong here?  Any suggestions for my requirement?

P.S. - I tried SCMSync plugin and ThinBackup.  But, problem with these are that they don't do full backup.  But, with ThinBackup both backup and restore works fine.

[Rob Pickerill]

Hi

Have you taken a look through: https://wiki.jenkins.io/display/JENKINS/I%27m+getting+too+many+open+files+error and https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ - these give guidance on handling and increasing the open file limit so you can restore your backup.

Thanks

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Artur Szostak]

Just my opinion: but I would forget about the backup plugins in Jenkins for full production backups. They all do a poor job when it comes to a full backup for a production system that also needs high availability. The best that can be done at the moment is to run Jenkins on a file system or data store that is able to perform copy-on-write snapshots. From within Jenkins you will want to setup a simple job that triggers the copy-on-write snapshot when no other job is running. Then use your normal production backup solutions to backup the snapshot in the background.

Cheers

Artur

________________________________________
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of gotvi...@gmail.com <gotvi...@gmail.com>
Sent: 01 November 2018 21:36
To: Jenkins Users
Subject: Jenkins Backup and Restore

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-use...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com?utm_medium=email&utm_source=footer>.

[John Mellor]

I have had restore issues with *ALL* of the backup plugins that I have
tried. The problem is that they are using stupid mechanisms to copy
the filesystem that cannot correctly copy a symlink. This breaks the
LastSuccessfulBuild and LastUnsuccessfulBuild and LastFailedBuild
symlinks, causing lots of secondary issues.

Doing my own using cpio or rsync in a cronjob only messes up copying
very large files that exceed the archive limits (4GB for cpio for
instance).

[gotvi...@gmail.com]

I wish for a tool like Jenkins it had a backup and restore option built in.

[gotvi...@gmail.com]

Our cloud is Azure.  Do you have any suggestions for this?

On Monday, November 26, 2018 at 4:13:06 PM UTC-5, Artur Szostak wrote:

Just my opinion: but I would forget about the backup plugins in Jenkins for full production backups. They all do a poor job when it comes to a full backup for a production system that also needs high availability. The best that can be done at the moment is to run Jenkins on a file system or data store that is able to perform copy-on-write snapshots. From within Jenkins you will want to setup a simple job that triggers the copy-on-write snapshot when no other job is running. Then use your normal production backup solutions to backup the snapshot in the background.

Cheers

Artur

________________________________________
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of gotvi...@gmail.com <gotvi...@gmail.com>
Sent: 01 November 2018 21:36
To: Jenkins Users
Subject: Jenkins Backup and Restore

We have a Jenkins instance running and we are trying to setup a backup and restore mechanism in case of any mishaps.  We are looking for a full backup of the Jenkins and so decided to go with PeriodicBackup plugin.  With this plugin, the backup works fine.  But, we are trying to restore it in a completely new Jenkins setup and not able to restore.  When we pick the backup file that we want to restore from, it says restoring, but nothing happens.  Even after waiting out for over 30 mins, nothing is restored.  As a work around, I stopped the Jenkins service, renamed the 'JENKINS_HOME' (/var/lib/jenkins) folder to 'jenkins_backup' and created another folder called 'jenkins' in the same location.  In this folder, I copied all the files from the backup and then started Jenkins (NOTE: I didn't change the .owner file.  I'm using the .owner file of the new Jenkins only).  Jenkins came up and I was able to see all the configurations from the old one.  Everything was fine.  I let it like that overnight and in the morning when I came and tried to access it, Jenkins was way too slow and I was not able to save any new configuration.  When I try I'm getting
java.io.ioexception: too many open files jenkins
error.  I restarted the jenkins and it's taking too long for the UI to come up.  Am I doing anything wrong here?  Any suggestions for my requirement?

P.S. - I tried SCMSync plugin and ThinBackup.  But, problem with these are that they don't do full backup.  But, with ThinBackup both backup and restore works fine.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-users+unsubscribe@googlegroups.com>.

[jxpe...@godaddy.com]

Have you considered sending a PR to (or at least filing a Github issue for) the plugins in question?

 

It's probably an easy fix to handle symlinks

 

Best

Jeff

[jxpe...@godaddy.com]

The beauty of Jenkins is that it's NOT built-in, but can be easily added as a plugin (of which there are several).

[RAJENDRA PRASAD]

Hi ,

I have another solution, just document the list of plugins in old Jenkins server and instead of backup and restore plugins just reinstall plugins on new Jenkins server.

Thanks,

Rajendra

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com.

[gotvi...@gmail.com]

The plugin team's been unresponsive for ages.  No help there.

[gotvi...@gmail.com]

Tried out a bunch of them.  Unfortunately, none of them are good for a production scale.

[gotvi...@gmail.com]

I'm not sure what you are trying to say?

[Jeff Pearce]

What issues are you seeing that prevent them from being good for production scale?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f5625308-1145-4d77-b021-62be8a779592%40googlegroups.com.

[Mark Waite]

He's saying that if you don't need to backup and restore your job definitions (possibly they are all multibranch Pipeline jobs that are easy to recreate, or they are all defined with the Job DSL plugin so they will be automatically recreated or they are defined in a Docker image) and don't need to backup and restore your credential definitions (same reasons) and don't need to backup and restore other configuration settings (possibly because you're using Configuration as Code plugin to store your configuration in a source control system), then there are backup concepts that use a source control system as your backup system.

In your use case, I suspect you want to use a backup.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9d3fad8e-7bbe-4472-86dd-57ef3d0da960%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[gotvi...@gmail.com]

We are looking for a full backup of the JENKINS_HOME folder (automated).  We looked at SMC Sync plugin, ThinBackup, Backup and PeriodicBackup.  PeriodicBackup was the only one which offered both.  But, with this we are facing the following issues

1) The values that we set for "# of backups" and the "# of days to store backup" does not work.  It keeps creating more and more without deleting the previous ones.

2) The restore feature does not work.  I took one backup and tried to restore a new Jenkins instance.  It says "restoring backup" in the UI, but no matter how long we wait, it doesn't restore.  I had to manually extract the file to the JENKINS_HOME folder.

[gotvi...@gmail.com]

I'm relatively new to all these and I'm just now realizing there are this many ways to use Jenkins (sorry for being dumb).  YES!!! like you said, we want to use a backup mechanism as we don't have any of these in place.  Where can I find documentation or study materials for the stuff you mentioned here?  Would really appreciate if you can point me to some. 

[Artur Szostak]

Do you mean your Jenkins master is hosted on a VM in Azure or the build nodes?
I would not backup the build nodes. I would treat them as volatile and only focus on backing up the master. Unless you have a good reason for backing up the build nodes also.

As for the Jenkins master, I do not know much about Azure, but does it have any kind of copy-on-write snapshot mechanism that you could trigger from within the VM? If yes, then you could call that mechanism from within a periodic backup job within Jenkins, but make sure the job only executes when nothing else is running on Jenkins.

In our case we use Btrfs for the filesystem, so we call something like the following from within a Jenkins job:

sudo btrfs subvolume snapshot <jenkins_home> <snapshot_path>

To make sure the job only runs when nothing else is running we take advantage of the following plugin:

https://plugins.jenkins.io/build-blocker-plugin

and then just use '.*' for the regular expression pattern.
After the snapshot is complete, the snapshot itself is backed up to tape with a commercial solution. I believe our IT department does this on the block level.

Cheers

Artur

________________________________________
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of gotvi...@gmail.com <gotvi...@gmail.com>

Sent: 26 November 2018 22:42
To: Jenkins Users

Subject: Re: Jenkins Backup and Restore

Our cloud is Azure. Do you have any suggestions for this?

On Monday, November 26, 2018 at 4:13:06 PM UTC-5, Artur Szostak wrote:
Just my opinion: but I would forget about the backup plugins in Jenkins for full production backups. They all do a poor job when it comes to a full backup for a production system that also needs high availability. The best that can be done at the moment is to run Jenkins on a file system or data store that is able to perform copy-on-write snapshots. From within Jenkins you will want to setup a simple job that triggers the copy-on-write snapshot when no other job is running. Then use your normal production backup solutions to backup the snapshot in the background.

Cheers

Artur

________________________________________
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of gotvi...@gmail.com <gotvi...@gmail.com>
Sent: 01 November 2018 21:36
To: Jenkins Users
Subject: Jenkins Backup and Restore

We have a Jenkins instance running and we are trying to setup a backup and restore mechanism in case of any mishaps. We are looking for a full backup of the Jenkins and so decided to go with PeriodicBackup plugin. With this plugin, the backup works fine. But, we are trying to restore it in a completely new Jenkins setup and not able to restore. When we pick the backup file that we want to restore from, it says restoring, but nothing happens. Even after waiting out for over 30 mins, nothing is restored. As a work around, I stopped the Jenkins service, renamed the 'JENKINS_HOME' (/var/lib/jenkins) folder to 'jenkins_backup' and created another folder called 'jenkins' in the same location. In this folder, I copied all the files from the backup and then started Jenkins (NOTE: I didn't change the .owner file. I'm using the .owner file of the new Jenkins only). Jenkins came up and I was able to see all the configurations from the old one. Everything was fine. I let it like that overnight and in the morning when I came and tried to access it, Jenkins was way too slow and I was not able to save any new configuration. When I try I'm getting
java.io.ioexception: too many open files jenkins
error. I restarted the jenkins and it's taking too long for the UI to come up. Am I doing anything wrong here? Any suggestions for my requirement?

P.S. - I tried SCMSync plugin and ThinBackup. But, problem with these are that they don't do full backup. But, with ThinBackup both backup and restore works fine.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-use...@googlegroups.com>.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/0c58b4fa-4e8b-4c53-9ca9-b99397078fae%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-use...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7c46683a-fbfb-4d79-b470-10246e8999d6%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/7c46683a-fbfb-4d79-b470-10246e8999d6%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Artur Szostak]

None of the plugins performs a copy-on-write style full backup of $JENKINS_HOME on the master (I am assuming workspaces are not included or stored somewhere else). Why copy-on-write: because I want close to zero down time in my Jenkins master when performing the backup. I also see no evidence of any of the backup plugins applying a proper quiesce strategy.


________________________________________
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of Jeff Pearce <jxpe...@godaddy.com>
Sent: 27 November 2018 18:24
To: jenkins...@googlegroups.com

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f5625308-1145-4d77-b021-62be8a779592%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/f5625308-1145-4d77-b021-62be8a779592%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-use...@googlegroups.com>.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/588AA5F3-DCE6-4516-B0C7-7E2C161E3D5C%40godaddy.com<https://groups.google.com/d/msgid/jenkinsci-users/588AA5F3-DCE6-4516-B0C7-7E2C161E3D5C%40godaddy.com?utm_medium=email&utm_source=footer>.