[JIRA] [multi-branch-project-plugin] (JENKINS-35460) Multibranch Plugin: branch indexing causes java heap out of space error

itaisanders@gmail.com (JIRA)

unread,

Jun 8, 2016, 5:03:01 AM6/8/16

to jenkinsc...@googlegroups.com

Itai Sanders created an issue

Jenkins /

JENKINS-35460

Multibranch Plugin: branch indexing causes java heap out of space error

Issue Type:	Bug
Assignee:	Matthew DeTullio
Components:	multi-branch-project-plugin
Created:	2016/Jun/08 9:02 AM
Environment:	master runs on Windows Server 2012 Jenkins 2.8
Priority:	Major
Reporter:	Itai Sanders

we are migrating our jobs to the multibranch pipelines, about a dozen or more.
all of them work fine, but one repository wouldn't pass the branch indexing process. the process gets stuck and eventually fails with the following log message:
??Setting origin to https://git.....
Fetching origin...
Finished: NOT_BUILT??

after double and triple checking all our configurations and repositories, we couldn't find any difference between this repository and the others. I went through the Jenkins Log and found out that whenever this particular branch indexing fails the log prints out a java heap out of memory error.
it doesn't show the full message on every such error, so I can only assume the first one of those is the relevant full stack trace:

 
                                                    Jun 08, 2016 11:31:22 AM SEVERE hudson.model.Executor finish1
Executor threw an exception
java.lang.OutOfMemoryError: Java heap space
	at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)
	at org.eclipse.jgit.internal.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:610)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:587)
	at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:550)
	at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:507)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:194)
	at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:448)
	at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:762)
	at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:363)
	at org.eclipse.jgit.transport.TransportHttp$SmartHttpFetchConnection.doFetch(TransportHttp.java:783)
	at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:301)
	at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:291)
	at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:247)
	at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:160)
	at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
	at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1138)
	at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)
	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.fetch(JGitAPIImpl.java:678)
	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:174)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:146)
	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:294)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:157)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:122)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
 
                                                

again - we have no idea what makes the difference. the repository itself isn't bigger than the others, we don't have special characters in our branch names (only alphanomerics . - _ and such. no spaces and no escaped chars) and all the repositories are hosted in Assembla.

running this same job as a regular pipeline job works. it's only the branch indexing that causes this problem. so my guess is that the heap space should be sufficient, but something somehow throws the indexing process into an infinite recursion (or something of the kind).

Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

itaisanders@gmail.com (JIRA)

unread,

Jun 8, 2016, 5:04:05 AM6/8/16

to jenkinsc...@googlegroups.com

Itai Sanders updated an issue

Jenkins /

JENKINS-35460

Multibranch Plugin: branch indexing causes java heap out of space error

Change By:	Itai Sanders

we are migrating our jobs to the multibranch pipelines, about a dozen or more.
all of them work fine, but one repository wouldn't pass the branch indexing process. the process gets stuck and eventually fails with the following log message:

?? {noformat}
Setting origin to https://git.....
Fetching origin...
Finished: NOT_BUILT ??
{noformat}

after double and triple checking all our configurations and repositories, we couldn't find any difference between this repository and the others. I went through the Jenkins Log and found out that whenever this particular branch indexing fails the log prints out a java heap out of memory error.
it doesn't show the full message on every such error, so I can only assume the first one of those is the relevant full stack trace:

{noformat}

again - we have no idea what makes the difference. the repository itself isn't bigger than the others, we don't have special characters in our branch names (only alphanomerics . - _ and such. no spaces and no escaped chars) and all the repositories are hosted in Assembla.

running this same job as a regular pipeline job works. it's only the branch indexing that causes this problem. so my guess is that the heap space should be sufficient, but something somehow throws the indexing process into an infinite recursion (or something of the kind).

Add Comment

itaisanders@gmail.com (JIRA)

unread,

Jun 8, 2016, 7:36:01 AM6/8/16

to jenkinsc...@googlegroups.com

Itai Sanders commented on

JENKINS-35460

Re: Multibranch Plugin: branch indexing causes java heap out of space error

just an update on our situation that might shed some light on this:

I went to the .git dirs inside the cache dir and compared the on of the problematic repo to one of the others. I found that some files were missing, probably files that should be created in the initial fetch that was broken in mid process. I copied FETCH_HEAD, logs/, objects/ and refs/ from my own local (working copy) .git of the same repo into the jenkins cache one and ran another branch indexing. I guess it recognized the files and skipped the initial fetch straight to the update process since everything seems to work now.
I made sure that it now recognizes new branches and everything seems to work fine.

to validate this I the deleted the cached dir and the branch indexing indeed crashed again like before, leaving behind a half built dir. I copied back the above file and dirs and everything started working normally again.