Cannot run program "git"

[Nicholas Mucci]

Over the last week some of my users have been getting "Internal Server
Error" popups when trying to review changes. The exception in the
logs is rather interesting:

2010-05-03 09:50:25,012::ERROR:
com.google.gerrit.server.cache.SelfPopulatingCache - Cannot lookup
com.google.gerrit.server
.patch.PatchListKey@e78bbb82 in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry
with key "com.google.gerrit.server.patch.PatchListKey@
e78bbb82".
at
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:
87)
at
com.google.gerrit.server.cache.SelfPopulatingCache.get(SelfPopulatingCache.java:
107)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
85)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
97)
at
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:
89)
at
com.google.gerrit.server.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFactory.java:
83)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(ChangeDetailFactory.java:
190)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
110)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
54)
at com.google.gerrit.server.rpc.Handler.to(Handler.java:56)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:
42)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
at
com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:
370)
at
com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:262)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:
820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:
216)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:
141)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:
93)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
63)
at
com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:
76)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
134)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.gerrit.server.http.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:
54)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:
122)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1115)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
361)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at
org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:
339)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
534)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:
403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Cannot run program "git" (in directory
"/data/gerrit/repositories/foobar-android/platform/frameworks/foobar/
foobar-src.git"): java.io.IOException: error=12, Cannot allocate
memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at
com.google.gerrit.server.patch.PatchListCacheImpl.exec(PatchListCacheImpl.java:
194)
at
com.google.gerrit.server.patch.PatchListCacheImpl.readPatchList(PatchListCacheImpl.java:
145)
at
com.google.gerrit.server.patch.PatchListCacheImpl.compute(PatchListCacheImpl.java:
104)
at com.google.gerrit.server.patch.PatchListCacheImpl.access
$000(PatchListCacheImpl.java:50)
at com.google.gerrit.server.patch.PatchListCacheImpl
$2.createEntry(PatchListCacheImpl.java:79)
at com.google.gerrit.server.patch.PatchListCacheImpl
$2.createEntry(PatchListCacheImpl.java:76)
at com.google.gerrit.server.cache.SelfPopulatingCache
$1.createEntry(SelfPopulatingCache.java:60)
at
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:
71)
... 50 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 59 more
2010-05-03 09:50:25.013:/code-review:WARN: Error in changeDetail
java.lang.NullPointerException
at
com.google.gerrit.server.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFactory.java:
84)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(ChangeDetailFactory.java:
190)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
110)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:
54)
at com.google.gerrit.server.rpc.Handler.to(Handler.java:56)
at
com.google.gerrit.server.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:
42)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
at
com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:
370)
at
com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:262)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:
820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:
216)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:
141)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:
93)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
63)
at
com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:
76)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
134)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.gerrit.server.http.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:
54)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:
129)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:
59)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:
122)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1115)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
361)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at
org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:
339)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
534)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:
403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:522)


My Gerrit instance (2.0.23) has 8GB of java heap space, and the box
its running on has 16GB RAM, and is only running Gerrit's postgresql
database in addition to Gerrit. There's plenty of free memory on the
box and plenty of free memory in the java heap. The only solution
I've seen so far for this is to restart Gerrit. I haven't seen anyone
else mention this, any ideas as to what could cause this problem?
Thanks!

-Nick

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Shawn Pearce]

Nicholas Mucci <nick....@gmail.com> wrote:
> Over the last week some of my users have been getting "Internal Server
> Error" popups when trying to review changes. The exception in the
> logs is rather interesting:
>
> 2010-05-03 09:50:25,012::ERROR:
> com.google.gerrit.server.cache.SelfPopulatingCache - Cannot lookup
> com.google.gerrit.server
> .patch.PatchListKey@e78bbb82 in "diff"
> net.sf.ehcache.CacheException: Could not fetch object for cache entry
> with key "com.google.gerrit.server.patch.PatchListKey@
> e78bbb82".

...

> Caused by: java.io.IOException: Cannot run program "git" (in directory
> "/data/gerrit/repositories/foobar-android/platform/frameworks/foobar/
> foobar-src.git"): java.io.IOException: error=12, Cannot allocate
> memory

...

> My Gerrit instance (2.0.23) has 8GB of java heap space, and the box
> its running on has 16GB RAM, and is only running Gerrit's postgresql
> database in addition to Gerrit. There's plenty of free memory on the
> box and plenty of free memory in the java heap. The only solution
> I've seen so far for this is to restart Gerrit. I haven't seen anyone
> else mention this, any ideas as to what could cause this problem?

What operating system?

The OS probably wants to reserve 8 GiB of memory for the new child
process being forked out of the JVM. Just in case that new child
writes every page. It won't, its going to write a couple of pages
at most before it exec's and discards its virtual memory. But the
OS doesn't know that, and is protecting itself by requiring enough
memory for the new child.

There isn't much you can do aside from boost swap and/or reduce
your JVM heap space size.

You might be able to upgrade your JVM and/or OS to ensure some newer
form of fork+exec is being used like spawn(), but I'm not sure the
Sun JVMs are really taking advantage of that yet.

FWIW, this is why I try to avoid forking stuff in Java. It always
ends up a disaster in the end.

We could try to do a code change in Gerrit where we fork and exec
some sort of fork-helper-daemon on startup, and then use that to
actually spawn off the git processes. Its a bit messy because we
need to use some sort of external program, like a Perl or Python
script, or a tiny C executable. Ick.

Long-term we really want to see JGit reach a point where its able
to do everything git diff-tree does for us, so we can avoid the
fork entirely. But its just not there yet.

[Nicholas Mucci]

I'm running on Ubuntu 8.04, with Java 1.6.0_12-b04. For right now, I
guess I can scale back my java heap to 6GB and see how that goes.
Under normal load from users, it doesn't take long for this problem to
manifest now.

I don't think I saw this before because I had 1GB of heap and 4GB RAM,
and just upgraded to 8GB heap, 16GB RAM last week. Immediately
afterward I noticed this problem.

I will see if we have a newer supported JVM for our Ubuntu servers.
I'll also let you know how scaling back to 6GB works out. Thanks!

-Nick

On May 3, 10:34 am, Shawn Pearce <s...@google.com> wrote:

[Nicholas Mucci]

Shawn,

Just to update, I scaled back the java heap from 8GB to a max of 6GB.
I haven't run into the "cannot allocate memory" exception since then.
Thanks for the suggestion!

-Nick