Jenkins hangs - core dump, then too many open files in the log

[Roman Chyla]

Hi,

We have recently moved to Jenkins (1.508) from Hudson and the service seems to be less stable. It happens that once in two weeks, Jenkins hangs. I can telnet to the Jenkins port, but it seems to just eat all input (black hole).

There is nothing in the log file (initially), two processed were showing 100% CPU. I tried jstack on them, and caused a java core dump on the second one of them - until then, one of our Jenkins tasks (which ran solr inside jenkins) was responding. Then it died.

So I restarted Jenkins, and jenkins log showed too many opened files.

INFO: Attempting to reconnect adsate-centos5

Exception in thread "ThinBackup Worker Thread thread" java.lang.Error: java.io.FileNotFoundException: /var/lib/jenkins/jenkins/ThinBackup Worker Thread.log (Too many open files)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork.createListener(AsyncPeriodicWork.java:76)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:50)

        at java.lang.Thread.run(Thread.java:722)

Caused by: java.io.FileNotFoundException: /var/lib/jenkins/jenkins/ThinBackup Worker Thread.log (Too many open files)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork.createListener(AsyncPeriodicWork.java:74)

        ... 2 more

Apr 23, 2013 1:30:57 PM hudson.triggers.SCMTrigger$Runner runPolling

SEVERE: Failed to record SCM polling for hudson.model.FreeStyleProject@28327b06[01-BEER-python]

java.io.FileNotFoundException: /var/lib/jenkins/jenkins/jobs/01-BEER-python/scm-polling.log (Too many open files)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)

        at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:433)

        at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468)

        at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

        at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

...

What can we do about this?

Thanks,

  roman