warnings plugin

Nord, James

unread,

Feb 9, 2012, 11:32:27 AM2/9/12

to jenkins...@googlegroups.com

Hi,

We have the warnings plugin enabled – which looks for javac and javadoc warnings in the maven2 type build.

All is fine until I ran a release. The warnings plugin took about 9.5 hours (yes nine and a half) to parse the console to find the javadoc warnings!?

A build of deploy site-deploy with the same configuration takes a few minutes.

The main difference is the maven jobs use -Dmaven.test.redirectTestOutputToFile=true (and as such the console was large ~700MB)

However the javac error parsing completed in a few minutes…

(<consoleLogParsers>

<string>Java Compiler</string>

<string>JavaDoc</string>

</consoleLogParsers>

)

The parser found 161 javadoc warnings – so not that many :-P

Stack trace from the job is as follows

"Executor #0 for xxxxxxxx: executing yyyycommit #5" Id=63 Group=main RUNNABLE

at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)

at java.util.regex.Pattern$Curly.match0(Pattern.java:4158)

at java.util.regex.Pattern$Curly.match(Pattern.java:4132)

at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)

at java.util.regex.Pattern$Branch.match(Pattern.java:4502)

at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)

at java.util.regex.Pattern$Branch.match(Pattern.java:4500)

at java.util.regex.Pattern$Start.match(Pattern.java:3408)

at java.util.regex.Matcher.search(Matcher.java:1198)

at java.util.regex.Matcher.find(Matcher.java:591)

at hudson.plugins.warnings.parser.RegexpParser.findAnnotations(RegexpParser.java:81)

at hudson.plugins.warnings.parser.RegexpLineParser.parse(RegexpLineParser.java:82)

at hudson.plugins.warnings.parser.ParserRegistry.parse(ParserRegistry.java:273)

at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:278)

at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:244)

at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:338)

at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)

at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)

at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)

at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:942)

at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)

at hudson.model.Run.run(Run.java:1400)

at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:476)

at hudson.model.ResourceController.execute(ResourceController.java:88)

at hudson.model.Executor.run(Executor.java:175)

Is there something massively different between how javadoc and javac warnings are tracked – I’m unclear why it took so long.

Monitoring showed there was plenty of heap available and although this is on a vm – it pretty much had one core of a X5690 dedicated to this parsing (Jenkins itself has 2 vcpus – and the whole system was mostly idle.)

Jenkins 1.424.2 with 3.26 of the warnings plugin.

/James

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

Ullrich Hafner

unread,

Feb 10, 2012, 8:04:20 AM2/10/12

to jenkins...@googlegroups.com

The parsing depends on the size of the file to read. Each line is matched with a regular expression. The complexity of these expressions is quite different for different parsers.

The Java parser has an optimization step: it using a string matcher for each line that detects the existence of the word "warning". Only lines containing that text will be matched with the regular expression afterwards. I don't remember the syntax of JavaDoc warnings, is there a similar string that can be used?

And 750MB is a really large log file :-) As a workaround you can try to write the output of the javadoc processing to a different file.

Ulli

Nord, James

unread,

Feb 10, 2012, 10:11:07 AM2/10/12

to jenkins...@googlegroups.com

Hi Ulli,

I tried to find the offending lines in order to get a sample – but in doing this I also went to look at the full detail of the error in the source file and was met with the following:

Copying the source file '[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' from the workspace to the build folder '/jenkinsdir/data/jobs/CMS/jobs/vcm/jobs/vcm-app/jobs/commit/builds/2012-02-08_15-04-06/workspace-files/1fb0ab88.tmp' on the Hudson master failed.

Seems that the path is relative, however an absolute path is required when copying the sources.

Is the file 'ClassificationMediaDataCreator.java' contained more than once in your workspace?

Is the file '[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' a valid filename?

If you are building on a slave: please check if the file is accessible under '$HUDSON_HOME/[job-name]/[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java'

If you are building on the master: please check if the file is accessible under '$HUDSON_HOME/[job-name]/workspace/[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java'

hudson.util.IOException2: remote file operation failed: [INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java at hudson.remoting.Channel@2f891745:cabci-cf-rh53-x64-10

at hudson.FilePath.act(FilePath.java:754)

<snip>

Caused by: java.io.FileNotFoundException: [INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java (No such file or directory)

at java.io.FileInputStream.open(Native Method)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

<snip>

I wonder if there are multiple reasons then why this is slow – trying to find files that don’t exist by recursively looking form them – and regexp parsing?

As this was a release build the files will be contained twice in the workspace – once for the initial checkout and once for the subsequent release checkout.

What I see in the console for javadoc issues (maven2 build) is

[INFO] [INFO] Generating "JavaDocs" report.

[INFO] [ERROR] Error fetching link: http://someserver/maven_sites/somepath/package-list. Ignored it.

[INFO] Loading source files for package a.b.c...

[INFO] Loading source files for package x.y.z...

[INFO] Constructing Javadoc information...

[INFO] Standard Doclet version 1.6.0_22

[INFO] Building tree for all the packages and classes...

[INFO] Generating /path/to/some/file.html...

[INFO] Building index for all the packages and classes...

[INFO] Generating /path/to/some/file.html.

[INFO] Building index for all classes...

[INFO] Generating /path/to/some/file.html.

[INFO] Note: Custom tags that could override future standard tags: @todo. To avoid potential overrides, use at least one period character (.) in custom tag names.

[INFO] Note: Custom tags that were not seen: @todo

[INFO] 3 warnings

[INFO] [WARNING] Javadoc Warnings

[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java:65: warning - @return tag has no arguments.

[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ProgramClassificationAssociationDataGatherer.java:101: warning - @param argument "em2" is not a parameter name.

[INFO] [WARNING] /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/PublishClassificationDataCreator.java:30: warning - @return tag has no arguments.

Note the [WARNING] in this case is inside the [INFO] as the maven invocation performing javadoc is the release maven process.

So for this version of javadoc it looks like “.*\w\.java: warning –“ – so again I would think that “warning” could be used the same as javadoc (although 1.7 and pre 1.6 vms may be different and it may be different still if you don’t use maven?)

The large log was actually a mistake on my part – as I forgot to add “-Dmaven.test.redirectTestOutputToFile=true” and the tests run with debug log enabled J

So I can fix my immediate issue by sorting out my own mess – but I thought there may be some other issue lurking in the background.

/James

Ullrich Hafner

unread,

Feb 13, 2012, 5:25:52 AM2/13/12

to jenkins...@googlegroups.com

Hi James,

you are right, this is another reason that parsing is slow. (See JENKINS-10097, JENKINS-10596)

Currently, when the parser detects relative names it guesses absolute paths. The current implementation is quite dumb and very slow for large projects. A volunteer tried to implement a faster solution for this problem, however that code still is not ready for a public release.

Ull

Reply all

Reply to author

Forward