StackOverflowError at startup

[Eguess]

Hi list,

I'm having the following problem with our jenkins. It started to
happen recently, but i was unable to link it with any particular
jenkins version. Currently we are having about 500 jobs, most of them
are git based, configured as free projects and using ivy as a
dependency manager. All jobs are running on a single instance - we are
not using any clustering yet.

The probelm i have is that at the startup I'm getting a
StackOverflowError.
I attached the log file below. After the stack trace for the error
ends, according to logs server is trying to trigger all jobs.

I spent quite a lot of time to try to single out a project that would
be a reason for such behavior, thinking that may be some circular
dependency is triggering it, but I was unable to do so.

I did come up with scenario how to workaround that error and get the
server working:
When i see this error i kill the process and then I'm disabling all
project via editing of config.xml files for all of them (from jobs
folder):

vi -c "argdo %s/<disabled>false/<disabled>true/ge | update " */
config.xml
Then i restart the server and I'm usually seeing quite huge build
queue.
I'm cleaning it by canceling all builds and then without restarting
the server I'm reverting config.xml files to enable all jobs
After that I'm reloading configuratiion from disk and everything
works.

Any ideas are much appreciated.

Thanks,
Eugene


Exception:

Jun 15, 2011 12:08:16 PM hudson.model.Hudson$5 onTaskFailed
SEVERE: Failed Finalizing set up
java.lang.StackOverflowError
at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:157)
at java.io.File.getCanonicalPath(File.java:559)
at sun.security.provider.PolicyFile.canonPath(PolicyFile.java:
1849)
at java.io.FilePermission$1.run(FilePermission.java:186)
at java.io.FilePermission$1.run(FilePermission.java:183)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:183)
at java.io.FilePermission.<init>(FilePermission.java:249)
at
sun.net.www.protocol.file.FileURLConnection.getPermission(FileURLConnection.java:
198)
at
sun.net.www.protocol.jar.JarFileFactory.getPermission(JarFileFactory.java:
125)
at
sun.net.www.protocol.jar.JarFileFactory.getCachedJarFile(JarFileFactory.java:
95)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:
52)
at
sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:
104)
at
sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:
132)
at
org.apache.ivy.util.url.BasicURLHandler.openStream(BasicURLHandler.java:
129)
at org.apache.ivy.util.XMLHelper.parse(XMLHelper.java:118)
at org.apache.ivy.util.XMLHelper.parse(XMLHelper.java:96)
at org.apache.ivy.util.XMLHelper.parse(XMLHelper.java:86)
at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser
$Parser.parse(XmlModuleDescriptorParser.java:238)
at
org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser.parseDescriptor(XmlModuleDescriptorParser.java:
102)
at
org.apache.ivy.plugins.parser.ModuleDescriptorParserRegistry.parseDescriptor(ModuleDescriptorParserRegistry.java:
86)
at
org.apache.ivy.plugins.parser.AbstractModuleDescriptorParser.parseDescriptor(AbstractModuleDescriptorParser.java:
48)
at hudson.ivy.IvyBuildTrigger
$1.doInIvyContext(IvyBuildTrigger.java:380)
at org.apache.ivy.Ivy.execute(Ivy.java:358)
at
hudson.ivy.IvyBuildTrigger.recomputeModuleDescriptor(IvyBuildTrigger.java:
377)
at
hudson.ivy.IvyBuildTrigger.getModuleDescriptor(IvyBuildTrigger.java:
264)
at
hudson.ivy.IvyBuildTrigger.buildDependencyGraph(IvyBuildTrigger.java:
449)
at
hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:
195)
at hudson.model.Project.buildDependencyGraph(Project.java:169)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:89)
at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3503)
at
hudson.ivy.IvyBuildTrigger.setModuleDescriptor(IvyBuildTrigger.java:
410)
at
hudson.ivy.IvyBuildTrigger.recomputeModuleDescriptor(IvyBuildTrigger.java:
377)
at
hudson.ivy.IvyBuildTrigger.getModuleDescriptor(IvyBuildTrigger.java:
264)
at
hudson.ivy.IvyBuildTrigger.buildDependencyGraph(IvyBuildTrigger.java:
449)
at
hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:
195)
at hudson.model.Project.buildDependencyGraph(Project.java:169)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:89)
at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3503)
at
hudson.ivy.IvyBuildTrigger.setModuleDescriptor(IvyBuildTrigger.java:
410)
at
hudson.ivy.IvyBuildTrigger.recomputeModuleDescriptor(IvyBuildTrigger.java:
377)
at
hudson.ivy.IvyBuildTrigger.getModuleDescriptor(IvyBuildTrigger.java:
264)
at
hudson.ivy.IvyBuildTrigger.buildDependencyGraph(IvyBuildTrigger.java:
449)
at
hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:
195)
at hudson.model.Project.buildDependencyGraph(Project.java:169)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:89)
at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3503)
at
hudson.ivy.IvyBuildTrigger.setModuleDescriptor(IvyBuildTrigger.java:
410)
at
hudson.ivy.IvyBuildTrigger.recomputeModuleDescriptor(IvyBuildTrigger.java:
377)
at
hudson.ivy.IvyBuildTrigger.getModuleDescriptor(IvyBuildTrigger.java:
264)
at
hudson.ivy.IvyBuildTrigger.buildDependencyGraph(IvyBuildTrigger.java:
449)
at
hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:
195)
at hudson.model.Project.buildDependencyGraph(Project.java:169)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:89)
at hudson.model.Hudson.rebuildDependencyGraph(Hudson.java:3503)
at
hudson.ivy.IvyBuildTrigger.setModuleDescriptor(IvyBuildTrigger.java:
410)
at
hudson.ivy.IvyBuildTrigger.recomputeModuleDescriptor(IvyBuildTrigger.java:
377)
at
hudson.ivy.IvyBuildTrigger.getModuleDescriptor(IvyBuildTrigger.java:
264)
at
hudson.ivy.IvyBuildTrigger.buildDependencyGraph(IvyBuildTrigger.java:
449)
at
hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:
195)
at hudson.model.Project.buildDependencyGraph(Project.java:169)
at hudson.model.DependencyGraph.<init>(DependencyGraph.java:89)

...

[Jan Seidel]

Hi Eguess,

this is way out of my scope of expertise but I made some research on
your problem.
My suspicion is that your are trying to use a cannonical path that
refers to a raw path (e.g. /../foo/ -> /foo or /../../ -> /).

A few things cross my mind here:
a) a plugin freaks out. I have some funny issues myself with the
promote build. It requires a comparable "special treatment" to bring
it back on track as you have to. As it looks like does another plugin
occasionally interfere with it (collabnet).
b) The file systems security goes wacko:

at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:157)
at java.io.File.getCanonicalPath(File.java:559)
at sun.security.provider.PolicyFile.canonPath(PolicyFile.java:
1849)
at java.io.FilePermission$1.run(FilePermission.java:186)
at java.io.FilePermission$1.run(FilePermission.java:183)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:183)

at java.io.FilePermission.<init>(FilePermission.java:249) ...
1) the canonical path is treated different than raw paths. Has
the system been modified? New policies?
2) has your system reached a critical mass? One thing I found
on my research was that too many symbolic links can give you a hard
time.
c) or something completely simple: Do you eventually have a job that
uses canonical paths and one entry has a trailing slash?
d) do you use a repository or other storage solution where you save
your configuration? I had a VERY annoying experience with our SVN
properties that modified the config files slightly. Jenkins did not
take offense in it but some plugins did. Simply opening the
configuration of the jobs in question - the char did not show up here
- and hitting the save button fixed it until next time the system
checked versus the repository.

I hope this may help you somehow to get a new point of view. Even if I
may be totally wrong :)

Take care
Jan

> sun.net.www.protocol.file.FileURLConnection.getPermission(FileURLConnection.j...

[Sven Strittmatter]

Hi list,

my Jenkins also starts with StackOverflowError. You can see it here
http://www.weltraumschaf.de:8080/

I do not have a lot of Projects or files in the projects. The problem
occured on the last update via apt-get some days ago.

regards
Sven

[Vojtech Juranek]

Hi,
IMHO some strange project dependecy (cycle dependecy?)

[Eguess]

On Jun 17, 5:39 am, Vojtech Juranek <vjura...@redhat.com> wrote:
> Hi,
> IMHO some strange project dependecy (cycle dependecy?)
>

I tried to perform a kind of a binary search by disabling projects and
checking when the exception will go away, but as soon as i'm getting
close to just few projects affecting the startup, the system behavior
becomes totally inconsistent, so i cannot pull out one or more at
fault.

Another confusing thing is that if it would be something that Jan has
mentioned I would think that the problem have to reappear every time
configurations files are loaded (i.e. when server interacts with the
file system), but it is not the case. It happens only when the server
starts, not when the configuration is reloaded from the disk.

[Vojtech Juranek]

> On Jun 17, 5:39 am, Vojtech Juranek <vjura...@redhat.com> wrote:
> > Hi,
> > IMHO some strange project dependecy (cycle dependecy?)
>
> I tried to perform a kind of a binary search by disabling projects and
> checking when the exception will go away, but as soon as i'm getting
> close to just few projects affecting the startup, the system behavior
> becomes totally inconsistent, so i cannot pull out one or more at
> fault.

I think switch off ivy build triggers should help.

I don't have experince with Ivy plugin and now I look into the code for the
first time, but it seems to me, that's a bug in Ivy plugin

Ivy calls
Hudson.getInstance().rebuildDependencyGraph();
which starts following cycle:
getModuleDescriptor() -> recomputeModuleDescriptor() -> setModuleDescriptor()
-> hudson.model.Hudson.rebuildDependencyGraph()

resulting in StackOverflowError.
I created a JIRA for it [1], please watch it and eventually provide details

>It happens only when the server
> starts, not when the configuration is reloaded from the disk.

this is strange, dependecy graph is build during the start-up as well as

[1] https://issues.jenkins-ci.org/browse/JENKINS-10025

[Eguess]

>
> I don't have experince with Ivy plugin and now I look into the code for the
> first time,  but it seems to me, that's a bug in Ivy plugin
>
> Ivy calls
> Hudson.getInstance().rebuildDependencyGraph();
> which starts following cycle:
> getModuleDescriptor() -> recomputeModuleDescriptor() -> setModuleDescriptor()
> -> hudson.model.Hudson.rebuildDependencyGraph()
>
> resulting in StackOverflowError.
> I created a JIRA for it [1], please watch it and eventually provide details
>
> >It happens only when the server
> > starts, not when the configuration is reloaded from the disk.
>
> this is strange, dependecy graph is build during the start-up as well as
>
> [1]https://issues.jenkins-ci.org/browse/JENKINS-10025

Cool, thank you so much - we will wait for the verdict then...