Re:Jenkins hangs - core dump, then too many open files in the log

[Kevin Fleming (BLOOMBERG/ 731 LEXIN)]

If by 'the Jenkins port' you mean the HTTP port that the Jenkins web interface listens on, then connecting to it with telnet will in fact produce no response at all, even if you type stuff, unless you type something that looks like an HTTP request.

If your Jenkins instance has too many files open at once, you need to determine how to increase the maximum number of open files on the system where it is running. If Jenkins is unable to open the files it needs, it is likely to hang and/or crash.

----- Original Message -----
From: jenkins...@googlegroups.com
To: jenkins...@googlegroups.com
At: Apr 23 2013 15:10:57

Hi,

We have recently moved to Jenkins (1.508) from Hudson and the service seems to be less stable. It happens that once in two weeks, Jenkins hangs. I can telnet to the Jenkins port, but it seems to just eat all input (black hole).

There is nothing in the log file (initially), two processed were showing 100% CPU. I tried jstack on them, and caused a java core dump on the second one of them - until then, one of our Jenkins tasks (which ran solr inside jenkins) was responding. Then it died.

So I restarted Jenkins, and jenkins log showed too many opened files.

INFO: Attempting to reconnect adsate-centos5

Exception in thread "ThinBackup Worker Thread thread" java.lang.Error: java.io.FileNotFoundException: /var/lib/jenkins/jenkins/ThinBackup Worker Thread.log (Too many open files)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork.createListener(AsyncPeriodicWork.java:76)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:50)

        at java.lang.Thread.run(Thread.java:722)

Caused by: java.io.FileNotFoundException: /var/lib/jenkins/jenkins/ThinBackup Worker Thread.log (Too many open files)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)

        at org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork.createListener(AsyncPeriodicWork.java:74)

        ... 2 more

Apr 23, 2013 1:30:57 PM hudson.triggers.SCMTrigger$Runner runPolling

SEVERE: Failed to record SCM polling for hudson.model.FreeStyleProject@28327b06[01-BEER-python]

java.io.FileNotFoundException: /var/lib/jenkins/jenkins/jobs/01-BEER-python/scm-polling.log (Too many open files)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:97)

        at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:90)

        at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:433)

        at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:468)

        at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

        at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

...

What can we do about this?

Thanks,

  roman

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mark Waite]

https://wiki.jenkins-ci.org/display/JENKINS/I%27m+getting+too+many+open+files+error suggests this may be due to a file descriptor leak.  That page also suggests ways you can assist with the diagnosis.

Since the stack trace seems to reference the thin backup plugin, you could consider disabling or removing that plugin to see if that resolves  the problem.

Mark Waite

---

From: Kevin Fleming (BLOOMBERG/ 731 LEXIN) <kpfl...@bloomberg.net>
To: jenkins...@googlegroups.com
Sent: Tuesday, April 23, 2013 1:15 PM
Subject: Re:Jenkins hangs - core dump, then too many open files in the log