java.io.IOException: error=12, Cannot allocate memory

[Shahaf Hart]

Hi,

Using Jenkins 1.547, I get a crash with an out of memory failure on a slave.  This occurs mostly when I run a larger number of jobs simultaneously.  It starts the job by fetching from git and fails giving the error below.  I don't understand this, as I have 24 GB Ram on my VM slave, but at the exact time it failed, only 16 GB out of that amount were used.

Thanks in advance,

Shahaf

Building remotely on ci-app01 in workspace /home/deploy/workspace/bdd tests 4

00:57:32 ERROR: Workspace has a .git repository, but it appears to be corrupt.

00:57:32 hudson.plugins.git.GitException: Error performing command: git rev-parse --is-inside-work-tree

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1283)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1253)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1249)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1065)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1075)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.hasGitRepo(CliGitAPIImpl.java:171)

00:57:32 at hudson.plugins.git.GitAPI.hasGitRepo(GitAPI.java:186)

00:57:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

00:57:32 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

00:57:32 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

00:57:32 at java.lang.reflect.Method.invoke(Method.java:622)

00:57:32 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:299)

00:57:32 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:280)

00:57:32 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:239)

00:57:32 at hudson.remoting.UserRequest.perform(UserRequest.java:118)

00:57:32 at hudson.remoting.UserRequest.perform(UserRequest.java:48)

00:57:32 at hudson.remoting.Request$2.run(Request.java:328)

00:57:32 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

00:57:32 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

00:57:32 at java.util.concurrent.FutureTask.run(FutureTask.java:166)

00:57:32 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

00:57:32 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

00:57:32 at java.lang.Thread.run(Thread.java:701)

00:57:32 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/deploy/workspace/bdd tests 4"): java.io.IOException: error=12, Cannot allocate memory

00:57:32 at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)

00:57:32 at hudson.Proc$LocalProc.<init>(Proc.java:244)

00:57:32 at hudson.Proc$LocalProc.<init>(Proc.java:216)

00:57:32 at hudson.Launcher$LocalLauncher.launch(Launcher.java:773)

00:57:32 at hudson.Launcher$ProcStarter.start(Launcher.java:353)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1272)

00:57:32 ... 22 more

00:57:32 Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory

00:57:32 at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)

00:57:32 at java.lang.ProcessImpl.start(ProcessImpl.java:81)

00:57:32 at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)

00:57:32 ... 27 more

00:57:32 Cloning the remote Git repository

00:57:32 Cloning repository sagit:sapi

00:57:32 ERROR: Error cloning remote repo 'origin'

00:57:32 hudson.plugins.git.GitException: Could not init /home/deploy/workspace/bdd tests 4

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$4.execute(CliGitAPIImpl.java:473)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:376)

00:57:32 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)

00:57:32 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)

00:57:32 at hudson.remoting.UserRequest.perform(UserRequest.java:118)

00:57:32 at hudson.remoting.UserRequest.perform(UserRequest.java:48)

00:57:32 at hudson.remoting.Request$2.run(Request.java:328)

00:57:32 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

00:57:32 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

00:57:32 at java.util.concurrent.FutureTask.run(FutureTask.java:166)

00:57:32 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

00:57:32 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

00:57:32 at java.lang.Thread.run(Thread.java:701)

00:57:32 Caused by: hudson.plugins.git.GitException: Error performing command: git init /home/deploy/workspace/bdd tests 4

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1283)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1253)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1249)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1065)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$4.execute(CliGitAPIImpl.java:471)

00:57:32 ... 12 more

00:57:32 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/deploy/workspace/bdd tests 4"): java.io.IOException: error=12, Cannot allocate memory

00:57:32 at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)

00:57:32 at hudson.Proc$LocalProc.<init>(Proc.java:244)

00:57:32 at hudson.Proc$LocalProc.<init>(Proc.java:216)

00:57:32 at hudson.Launcher$LocalLauncher.launch(Launcher.java:773)

00:57:32 at hudson.Launcher$ProcStarter.start(Launcher.java:353)

00:57:32 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1272)

00:57:32 ... 16 more

00:57:32 Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory

00:57:32 at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)

00:57:32 at java.lang.ProcessImpl.start(ProcessImpl.java:81)

00:57:32 at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)

00:57:32 ... 21 more

00:57:32 ERROR: null

[Mark Waite]

On Mon, Jan 5, 2015 at 2:58 AM, Shahaf Hart <sha...@seekingalpha.com> wrote:

Hi,

Using Jenkins 1.547, I get a crash with an out of memory failure on a slave.  This occurs mostly when I run a larger number of jobs simultaneously.  It starts the job by fetching from git and fails giving the error below.  I don't understand this, as I have 24 GB Ram on my VM slave, but at the exact time it failed, only 16 GB out of that amount were used.

The failure message is indicating that the git plugin failed to launch the external command line git program.  The IOException from the failure to launch seems to match the description in this answer on stackoverflow.com:

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0d91a671-9c2d-4bc9-bf15-e3666623848a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

On Mon, Jan 5, 2015 at 5:33 AM, Mark Waite <mark.ea...@gmail.com> wrote:

On Mon, Jan 5, 2015 at 2:58 AM, Shahaf Hart <sha...@seekingalpha.com> wrote:

Hi,

Using Jenkins 1.547, I get a crash with an out of memory failure on a slave.  This occurs mostly when I run a larger number of jobs simultaneously.  It starts the job by fetching from git and fails giving the error below.  I don't understand this, as I have 24 GB Ram on my VM slave, but at the exact time it failed, only 16 GB out of that amount were used.

The failure message is indicating that the git plugin failed to launch the external command line git program.  The IOException from the failure to launch seems to match the description in this answer on stackoverflow.com:

http://serverfault.com/questions/317115/jenkins-ci-cannot-allocate-memory

--

Thanks!

Mark Waite

[Daniel Beck]

That was my first guess as well. But it seems that the JRE defaults to using vfork though, which should not have this problem... right?

https://en.wikipedia.org/wiki/Fork_%28system_call%29#Vfork
http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/4645fadf8352/src/solaris/classes/java/lang/UNIXProcess.java.linux#l100

Of course a different JRE -- vendor or version -- may implement this differently, so switching to a different one, or finding out how to configure this, may be a possible solution.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtEJ5oLy-WS54AVtdwappYK7vg9994DG97hUVt_LUWQiaA%40mail.gmail.com.

[Stefan Thomasson]

Hi

If this only happens when a lot of simultaneous jobs are running on the same node you may want to check if you are reaching one of the limits set by ulimit, like threads per user.

http://stackoverflow.com/questions/5253072/java-unable-to-create-new-native-thread

//stefan

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CC49F372-F6A3-4CF0-95D9-4A39B841F445%40beckweb.net.