android builds failing with p=0.5

[Dave Dyer]

Android builds are currently failing about half the time.  Resubmitting the same build succeeds, eventually.
This is only annoying as long as the builds are "known good".   Is there some kind of server ID buried in
the build log?  It would be interesting to know if it's some particular server that needs to be rebooted.

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007df500000, 148897792, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 148897792 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/build5397136403480841141xxx/Launch/hs_err_pid10745.log
:transformClassesWithDexForRelease FAILED
:transformClassesWithDexForRelease (Thread[Daemon worker,5,main]) completed. Took 1 mins 1.231 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':transformClassesWithDexForRelease'.
> com.android.build.api.transform.TransformException: java.lang.RuntimeException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/usr/java/jdk1.7.0_67/bin/java'' finished with non-zero exit value 1

[Shai Almog]

Did you use android.includeGPlayServices=true ?
If so you should remove that

[Dave Dyer]

No, I've been using android.includeGPlayServices=false for quite some time,
and haven't changed my build settings recently.

[Shai Almog]

That is odd, we've made some server refreshes let me know if this is still happening.

[Dave Dyer]

No problem today.

[Dave Dyer]

And then again, another day, today my first build succeeded on the third attempt.  Same build all three times.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':transformClassesWithDexForRelease'.
> com.android.build.api.transform.TransformException: java.lang.RuntimeException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/usr/java/jdk1.7.0_67/bin/java'' finished with non-zero exit value 137

* Try:
Run with --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':transformClassesWithDexForRelease'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)

[Shai Almog]

That looks like a different error, any details?

[Dave Dyer]

Nothing that means anything to me.  It's clearly wrong for builds to be a probabilistic enterprise.
Maybe your servers are just overloaded. 

[Shai Almog]

They try to be consistent but gradle is a piece of .... Unfortunately that's the way Google chose to go so we have no choice.

[Dave Dyer]

When they do fail, the proximate complaint seems to be "out of memory" type errors.  Maybe if you fed your JVMs
a little better they would get more work done.

[Shai Almog]

Where do we stop? This is a build server.
Each server has MANY gigabytes of RAM just for compilation.

[Dave Dyer]

Try running your jvms with -m1g (or whatever the proper switch is these days).  Maybe that will help.
Or maybe there's another environmental problem.  The original error log clipping in this thread says
an allocation of 148M failed.  That seems rather large for a single request.

[Shai Almog]

Seriously?
Do you honestly think we didn't set MX properties on the VM's?

Gradle leaks like the titanic.

[Dave Dyer]

Sure, but maybe not high enough, else why are you getting malloc failures?

[Shai Almog]

Actually it's the exact opposite.