[JIRA] [p4-plugin] (JENKINS-31683) p4-plugin out of memory with large depots on 32 bit java

163 views
Skip to first unread message

ramia@hb-studios.com (JIRA)

unread,
Nov 20, 2015, 10:31:02 AM11/20/15
to jenkinsc...@googlegroups.com
Rami Alia created an issue
 
Jenkins / Bug JENKINS-31683
p4-plugin out of memory with large depots on 32 bit java
Issue Type: Bug Bug
Assignee: Unassigned
Components: p4-plugin
Created: 20/Nov/15 3:30 PM
Environment: Jenkins 1.634
p4-plugin 1.3.2
Priority: Blocker Blocker
Reporter: Rami Alia

p4 java out of memory exception, stack trace is below. I've worked around this on a couple of slaves by using java64, I'll probably investigate it as well and push up a fix if I track one down but thought I'd pass it on since it adds quite a bit of effort to getting a default jenkins machine master or slave working with decent size depots.

stack trace:
Nov 20, 2015 10:31:40 AM org.jenkinsci.plugins.p4.tasks.AbstractTask tryTask
SEVERE: P4 Task: attempt: 1
com.perforce.p4java.exception.ConnectionException: Java heap space
at com.perforce.p4java.impl.mapbased.rpc.packet.RpcPacketDispatcher.dispatch(RpcPacketDispatcher.java:225)
at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execMapCmdList(OneShotServerImpl.java:350)
at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execMapCmdList(OneShotServerImpl.java:199)
at com.perforce.p4java.impl.mapbased.server.Server.execMapCmdList(Server.java:5149)
at com.perforce.p4java.impl.mapbased.client.Client.reconcileFiles(Client.java:1593)
at org.jenkinsci.plugins.p4.client.ClientHelper.tidyClean(ClientHelper.java:358)
at org.jenkinsci.plugins.p4.client.ClientHelper.tidyAutoCleanImpl(ClientHelper.java:285)
at org.jenkinsci.plugins.p4.client.ClientHelper.tidyWorkspace(ClientHelper.java:245)
at org.jenkinsci.plugins.p4.tasks.CheckoutTask.task(CheckoutTask.java:97)
at org.jenkinsci.plugins.p4.tasks.AbstractTask.tryTask(AbstractTask.java:216)
at org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke(CheckoutTask.java:91)
at org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke(CheckoutTask.java:28)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2720)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:69)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.perforce.p4java.impl.mapbased.rpc.func.helper.MD5Digester.digestStream(MD5Digester.java:177)
at com.perforce.p4java.impl.mapbased.rpc.func.helper.MD5Digester.digestFileAs32ByteHex(MD5Digester.java:337)
at com.perforce.p4java.impl.mapbased.rpc.func.helper.MD5Digester.digestFileAs32ByteHex(MD5Digester.java:302)
at com.perforce.p4java.impl.mapbased.rpc.func.client.ClientSystemFileCommands.reconcileEdit(ClientSystemFileCommands.java:1681)
at com.perforce.p4java.impl.mapbased.rpc.func.client.ClientFunctionDispatcher.dispatch(ClientFunctionDispatcher.java:194)
at com.perforce.p4java.impl.mapbased.rpc.packet.RpcPacketDispatcher.dispatch(RpcPacketDispatcher.java:160)
... 21 more
Nov 20, 2015 11:13:25 AM org.jenkinsci.plugins.p4.tasks.AbstractTask tryTask

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

ramia@hb-studios.com (JIRA)

unread,
Nov 20, 2015, 10:32:01 AM11/20/15
to jenkinsc...@googlegroups.com
Rami Alia updated an issue
Change By: Rami Alia
p4 java out of memory exception, stack trace is below. I've worked around this on a couple of slaves by using java64, I'll probably investigate it as well and push up a fix if I track one down but thought I'd pass it on since it adds quite a bit of effort to getting a default jenkins machine master or slave working with decent size depots.

last jenkins console output:
P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f -m D:\jenkins\Swarm/...
 -
p4 reconcile -w -f -m D:\jenkins\Swarm/...

P4 Task: attempt: 1
P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
ERROR: P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
Archiving artifacts
Creating image galleries.Creating archived images gallery.Recording plot data
Finished: FAILURE

johannes.lothberg@starstable.com (JIRA)

unread,
May 27, 2016, 4:29:01 AM5/27/16
to jenkinsc...@googlegroups.com
Johannes Löthberg commented on Bug JENKINS-31683
 
Re: p4-plugin out of memory with large depots on 32 bit java

Can reproduce on Jenkins 2.1, p4-plugin 1.8.8. About to update to 1.8.9 to see if it's fixed there.

johannes.lothberg@starstable.com (JIRA)

unread,
May 27, 2016, 5:09:01 AM5/27/16
to jenkinsc...@googlegroups.com

Same issue with Jenkins 2.6 and p4-plugin 1.8.9.

pallen@perforce.com (JIRA)

unread,
Aug 25, 2016, 5:40:02 AM8/25/16
to jenkinsc...@googlegroups.com

The 1.4.4 release or P4 uses a more efficient method to sync and reconcile:
https://swarm.workshop.perforce.com/changes/19918

Please can you verify and update or close the issue as needed.

Thanks,
Paul

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

johannes.lothberg@starstable.com (JIRA)

unread,
Aug 25, 2016, 6:54:03 AM8/25/16
to jenkinsc...@googlegroups.com

Same problem on Jenkins 2.19 with P4 plugin version 1.4.4.

pallen@perforce.com (JIRA)

unread,
Aug 25, 2016, 11:15:02 AM8/25/16
to jenkinsc...@googlegroups.com

Please can you include (or update) the stack trace for 1.4.4

How many files are you potentially cleaning up (versioned/non-versioned files) and how much java heap memory does the slave have?

Thanks,
Paul

johannes.lothberg@starstable.com (JIRA)

unread,
Aug 26, 2016, 7:24:01 AM8/26/16
to jenkinsc...@googlegroups.com

No longer get a stack trace in the log, just "ERROR: P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space".

There are some 24 thousand files in the data directory that it tries to check out, and I'm not sure how to check the heap memory? It has 4G RAM at least.

aj1773@gmail.com (JIRA)

unread,
Sep 1, 2016, 5:18:02 PM9/1/16
to jenkinsc...@googlegroups.com

I just started seeing the same thing. I upgraded to 1.4.6, just to check the latest and greatest, and I'm still seeing the same thing. This is on a sync only execution.

pallen@perforce.com (JIRA)

unread,
Sep 2, 2016, 4:55:01 AM9/2/16
to jenkinsc...@googlegroups.com

Thank you for checking again, there was nothing specific in 1.4.6 release for the memory issue. We have a reproduction case here and it seems to be something in the p4java API, we are investigating and will let you know what we find.

araine@disruptorbeam.com (JIRA)

unread,
Oct 6, 2016, 2:30:02 PM10/6/16
to jenkinsc...@googlegroups.com

Any further progress on this issue? I'm encountering it on some of my builds when I attempt to switch to the official Perforce plugin, as opposed to the third-party perforce plugin. I've upgraded to the latest plugin (1.4.7), and I'm on latest Jenkins (2.7.4); the target box in the failure cases is a Windows box hosted on AWS. I'm happy to share any other relevant details that could help diagnose it. Thanks!

aj1773@gmail.com (JIRA)

unread,
Oct 7, 2016, 6:02:03 PM10/7/16
to jenkinsc...@googlegroups.com

I'd love to get an update on this, too. We're seeing it more and more often - I don't know if it's because the size of our stream is getting larger, or if there's something else that is degrading it. It's a big problem if I have to manually log into the jenkins machine and get the latest revision myself every time, which is the only workaround I've managed to find so far.

sid@inxile.net (JIRA)

unread,
Oct 10, 2016, 9:22:01 PM10/10/16
to jenkinsc...@googlegroups.com

I'm running into the same problem. Have a depot with a large number of files. Honestly I don't think we have any need to reconcile? I've only checked out the files I need and just need to submit what's checked out. Maybe i'm missing something but I feel like this should be an option on whether to do that step.

pallen@perforce.com (JIRA)

unread,
Oct 11, 2016, 4:52:06 AM10/11/16
to jenkinsc...@googlegroups.com

With version 1.4.7 or later, can someone please let me know where the plugin runs out of memory? For example: during the 'p4 reconcile -w' or a later stage. Another stack trace would be helpful.

In addition please let me know the number of files and size in the scope of the Perforce workspace:
p4 sizes -s //client_name/...

stalle+jenkins-ci@gmail.com (JIRA)

unread,
Oct 11, 2016, 9:02:04 AM10/11/16
to jenkinsc...@googlegroups.com

I'm pretty sure this happened on 1.4.7 for me, but the slave seems to have been restarted and the logs are gone. Isn't it from 1.4.8 or later we should be checking? That's were the new p4java is according to the change logs.
Is there a way to get the stack trace in the normal console output? For me it just prints the exception message.

pallen@perforce.com (JIRA)

unread,
Oct 11, 2016, 9:55:02 AM10/11/16
to jenkinsc...@googlegroups.com

1.4.4 added streaming support for reconcile (should significantly reduce the memory) and yes 1.4.8 uses an updated p4java. So anything between the 1.4.4 and 1.4.8 would be useful.
Unfortunately the slave can run out of memory at any point and sometimes it's not able to get a stacktrace, however the console log might suggest where it failed.

stalle+jenkins-ci@gmail.com (JIRA)

unread,
Oct 11, 2016, 11:18:03 AM10/11/16
to jenkinsc...@googlegroups.com

This is all I got in console output

P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f /home/run/jenkins/workspace/jobXXX/... -
p4 reconcile -w -f /home/run/jenkins/workspace/jobXXX/...

P4 Task: attempt: 1
P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
Finished: FAILURE

Maybe the lack of a stacktrace has to do with it being a pipeline job.

p4 sizes -s ./...
./... 16649 files 2476092266 bytes 

This is failed on a 64-bit debian server with 4Gb memory. But it does not always fail.

pallen@perforce.com (JIRA)

unread,
Oct 13, 2016, 6:10:02 AM10/13/16
to jenkinsc...@googlegroups.com
Paul Allen started work on Bug JENKINS-31683
 
Change By: Paul Allen
Status: Open In Progress

stalle+jenkins-ci@gmail.com (JIRA)

unread,
Dec 19, 2016, 1:42:03 AM12/19/16
to jenkinsc...@googlegroups.com
Staffan Forsell edited a comment on Bug JENKINS-31683
This is all I got in console output
{noformat}

P4 Task: cleaning workspace to match have list.
... p4 reconcile -w -f /home/run/jenkins/workspace/jobXXX/... -
p4 reconcile -w -f /home/run/jenkins/workspace/jobXXX/...

P4 Task: attempt: 1
P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
Finished: FAILURE
{noformat}


Maybe the lack of a stacktrace has to do with it being a pipeline job.
{noformat}

p4 sizes -s ./...
./... 16649 files 2476092266 bytes
{noformat}

This
is failed on a 64-bit debian server with 4Gb memory . But , but it does not always fail.

pallen@perforce.com (JIRA)

unread,
Dec 19, 2016, 5:29:01 AM12/19/16
to jenkinsc...@googlegroups.com

A possible work around is to use the 'Force clean' option, this avoids the reconcile command and should use less memory.

Enabling parallel sync should speed sync time and my use less memory in the Java Heap. If you are using slaves then parallel sync will require the 'p4' executable installed on the slave.

stalle+jenkins-ci@gmail.com (JIRA)

unread,
Jan 9, 2017, 10:24:04 AM1/9/17
to jenkinsc...@googlegroups.com

We switched force sync a while ago and it actually faster than the smart algorithm so well keep using it for now. So I will be adding more info to this case.

heiko.nardmann@itechnical.de (JIRA)

unread,
Feb 6, 2017, 4:43:02 AM2/6/17
to jenkinsc...@googlegroups.com

Looks like I got that this morning, too:

09:54:57 p4 reconcile -w -f -e -d D:\ci\workspace\some_job/...
09:54:57 
09:57:36 P4 Task: attempt: 1
09:57:39 P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space
09:57:39 ERROR: P4 Task: failed: com.perforce.p4java.exception.ConnectionException: Java heap space

I do not see a Java stack trace.

Jenkins v2.21, P4 plugin v1.4.12.

Jenkins runs on CentOS 7.2.1511 (64Bit). The slave system is Windows Server 2008 R2 Standard (64Bit). I'm not sure whether the Windows Java (jre1.8.0_121) is 32Bit or 64Bit. DependencyWalker says x86, so probably this is 32Bit.
Is 64Bit recommended?

The failed job is a freestyle job.

Acc. to /proc/.../maps the Java executable is

/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111-2.6.7.2.el7_2.x86_64/jre-abrt/bin/java

So this is the following package (see rpm -qif ...):

Name        : java-1.7.0-openjdk-headless
Epoch       : 1
Version     : 1.7.0.111
Release     : 2.6.7.2.el7_2
Architecture: x86_64
Install Date: Tue 15 Nov 2016 03:01:46 PM CET
Group       : Development/Languages
Size        : 95555015
License     : ASL 1.1 and ASL 2.0 and GPL+ and GPLv2 and GPLv2 with exceptions and LGPL+ and LGPLv2 and MPLv1.0 and MPLv1.1 and Public Domain and W3C
Signature   : RSA/SHA256, Wed 27 Jul 2016 01:39:38 PM CEST, Key ID 24c6a8a7f4a80eb5
Source RPM  : java-1.7.0-openjdk-1.7.0.111-2.6.7.2.el7_2.src.rpm
Build Date  : Wed 27 Jul 2016 01:07:19 PM CEST
Build Host  : worker1.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
URL         : http://openjdk.java.net/
Summary     : The OpenJDK runtime environment without audio and video support
Description :
The OpenJDK runtime environment without audio and video

What is not clear to me: is this an issue on the server or on the slave agent?

Ahh ... forget last question: acc. to slave agent error log:

Feb 06, 2017 9:57:37 AM org.jenkinsci.plugins.p4.tasks.AbstractTask tryTask
SEVERE: P4 Task: attempt: 1
com.perforce.p4java.exception.ConnectionException: Java heap space
	at com.perforce.p4java.impl.mapbased.rpc.packet.RpcPacketDispatcher.dispatch(RpcPacketDispatcher.java:225)
	at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execMapCmdList(OneShotServerImpl.java:350)
	at com.perforce.p4java.impl.mapbased.rpc.OneShotServerImpl.execStreamingMapCommand(OneShotServerImpl.java:415)
	at org.jenkinsci.plugins.p4.client.ClientHelper.tidyClean(ClientHelper.java:486)
	at org.jenkinsci.plugins.p4.client.ClientHelper.tidyAutoCleanImpl(ClientHelper.java:404)
	at org.jenkinsci.plugins.p4.client.ClientHelper.tidyWorkspace(ClientHelper.java:352)
	at org.jenkinsci.plugins.p4.tasks.CheckoutTask.task(CheckoutTask.java:120)
	at org.jenkinsci.plugins.p4.tasks.AbstractTask.tryTask(AbstractTask.java:215)
	at org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke(CheckoutTask.java:114)
	at org.jenkinsci.plugins.p4.tasks.CheckoutTask.invoke(CheckoutTask.java:26)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:62)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space

Hmm ... I do not see any special memory limit settings inside jenkins-slave.xml on the slave machine.

How do I see the limits which are active for the slave agent? The node System Properties view is not helpful either (or I am blind - chronical Monday morning blindness).

pallen@perforce.com (JIRA)

unread,
Feb 6, 2017, 5:49:04 AM2/6/17
to jenkinsc...@googlegroups.com

I wouldn't worry about Monday morning blindness - I have only just got back from a weeks skiing, so it will take me a while to get plugged back in.

When you start the slave agent you can add arguments to the JVM -Xmx1024m (e.g. 1GB for the heap); this all depends on how you start the slave, the StackOverflow link discusses some of the options:
http://stackoverflow.com/questions/5936519/how-to-give-jenkins-more-heap-space-when-it%C2%B4s-started-as-a-service-under-windows

An alternative is to use the ForceClean option (it might be slower, but potentially uses less memory than reconcile).

stalle+jenkins-ci@gmail.com (JIRA)

unread,
Mar 6, 2017, 7:43:02 AM3/6/17
to jenkinsc...@googlegroups.com
Staffan Forsell edited a comment on Bug JENKINS-31683
We switched force sync a while ago and it actually faster than the smart algorithm so well keep using it for now. So I will won't  be adding more info to this case.
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

ethan.w.thornburg@gmail.com (JIRA)

unread,
Mar 29, 2017, 8:07:03 AM3/29/17
to jenkinsc...@googlegroups.com

Is there any other resolutions for this?

-Xmx1025m in jenkins.xml didn't resolve the issue for me.

 

Using force clean does, but my builds all take a very long time now.

 

 

ethan.w.thornburg@gmail.com (JIRA)

unread,
Mar 29, 2017, 8:08:01 AM3/29/17
to jenkinsc...@googlegroups.com
Ethan Thornburg edited a comment on Bug JENKINS-31683
Reply all
Reply to author
Forward
0 new messages