|
We are using matrix jobs and also jobs which are running other jobs (Build Flow plugin). We enabled "Wait until Jenkins/Hudson is idle to perform a backup" in ThinBackup configuration, unfortunately, this has caused situation that our Jenkins instance is in quiet mode indefinitely (or until we abort the running jobs).
The problem occurs when a matrix parent job (or another parent job) is running, some of its child jobs are not running yet (but they have been put in the job queue), and at this moment ThinBackup forces Jenkins into quiet mode. Since the parent jobs will not finish until their children finish, and the child jobs will never be taken from the queue and executed (because of Jenkins quiet mode), Jenkins is stuck in quiet mode forever. The quiet mode cannot be cancelled, because ThinBackup immediately resumes the quiet mode.
There is obvious short-term "solution" - abort the stuck parent jobs, this will make ThinBackup to proceed with backup and finally cancel Jenkins quiet mode. Unfortunately, this requires restarting the affected jobs which is not acceptable (there are too many of them) and time consuming.
Since we are making backups every night and we need to run a lot of jobs, our situation in the past few days resulted in encountering Jenkins in quiet mode each morning and having to abort several jobs in order to get it back from quiet mode.
We have disabled "Wait until Jenkins/Hudson is idle to perform a backup" configuration option for now, but we do not feel comfortable with this "solution" because of potentially inconsistent backups.
|
