java.lang.OutOfMemoryError: unable to create new native thread

[solid]

Hi Guys

Over the last few weeks I have been getting memory errors with jenkins. It first starts of as "pending—Waiting for next available executor" when attempting to execute a new job. I restart the jenkins daemon only to be confronted with the following error on the web GUI:

Apr 14, 2014 7:11:36 PM SEVERE hudson.triggers.SafeTimerTask run

Timer task hudson.model.Queue$MaintainTask@52d9051 failed
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:679)
	at hudson.model.Executor.start(Executor.java:479)
	at hudson.model.Queue$JobOffer.set(Queue.java:250)
	at hudson.model.queue.MappingWorksheet$ExecutorChunk.execute(MappingWorksheet.java:164)
	at hudson.model.queue.MappingWorksheet$ExecutorChunk.access$000(MappingWorksheet.java:112)
	at hudson.model.queue.MappingWorksheet$Mapping.execute(MappingWorksheet.java:313)
	at hudson.model.Queue.maintain(Queue.java:1048)
	at hudson.model.Queue$MaintainTask.doRun(Queue.java:1992)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:701)

The only way jenkins will function as normal is if I reboot the server. Jenkins will work fine for a number of days and then the problem will repeat itself.

I have increased the ulimit for the jenkins user to 8K soft and 16k hard. I have also increased the MaxPermSize and mx values in /etc/sysconfig/jenkins file:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=1024m"

Yet I am still getting out of memory errors. I am not sure what the cause is and how to remediate it.

free -m reveals that I have about 1800mb of free ram, so its definately not the the hardware.

Any guidence on how to resolve this problem would be much appreciated.

Regards

[Jenkins Usr]

[Björn Pedersen]

Am Montag, 14. April 2014 22:30:37 UTC+2 schrieb solid:

Hi Guys

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=1024m"

Well that's 2GB  allocated to jenkins. Which is not too bad, depending on the number of jobs and the discard-build settings.
(always under the assumption no builds are running on master)
There is a nice summary for fine tuning memory here:

https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-responsiveness-and-stability

And check this as well:
https://www.cloudbees.com/so-you-want-build-worlds-biggest-jenkins-cluster