Processing of 2016 and Modified never completes

[Steve Chernyak]

Hello,

I'm using the maven plugin. When using 1.4.2, I see all of the downloads complete, but when it gets around to processing the 2016 and Modified, it appears to hang. I tried building 1.4.3-SNAPSHOT, but the build fails to pass it's tests for apparently the same reason:

[INFO] Checking for updates
[INFO] NVD CVE requires several updates; this could take a couple of minutes.
[INFO] Download Started for NVD CVE - 2014
[INFO] Download Started for NVD CVE - 2015
[INFO] Download Started for NVD CVE - 2016
[INFO] Download Complete for NVD CVE - 2016  (4977 ms)
[INFO] Download Started for NVD CVE - Modified
[INFO] Processing Started for NVD CVE - 2016
[INFO] Download Complete for NVD CVE - 2015  (6715 ms)
[INFO] Download Complete for NVD CVE - 2014  (7848 ms)
[INFO] Download Complete for NVD CVE - Modified  (3926 ms)
[INFO] Processing Started for NVD CVE - 2015
[INFO] Processing Complete for NVD CVE - 2015  (16710 ms)
[INFO] Processing Started for NVD CVE - 2014
[INFO] Processing Complete for NVD CVE - 2014  (31289 ms)
[INFO] Processing Started for NVD CVE - Modified
[WARN] Unable to update Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.

This is slightly different than what I'm seeing when running a build with 1.4.2. With 1.4.2 I don't see the warning, it just sits there "processing" 2016 and Modified for hours.

Is there a way to get additional information on what's happening with these? Can I get around the problem by manually loading the data somehow?

Thanks

Relevant versions:

$ uname -a
Linux 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

$ mvn --version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-36-generic", arch: "amd64", family: "unix"
steve@Latitude-E6400:~/Development/Projects/DependencyCheck$
 

[Jeremy Long]

By using the `mvn verify -X` you can get additional details about the error. However, I am pretty sure you are running into some null pointer exceptions - which I just ran into today (I'm guessing due to new contents in the NVD that appear to have a few NULL values that I had not previously accounted for. In reality, the previous versions also ran into the exact same issue - the newer version has stronger (better) error handling in place to show when there are problems.

I will be publishing an updated version shortly that will resolve these bugs.

--Jeremy

--
You received this message because you are subscribed to the Google Groups "Dependency Check" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dependency-check+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Chernyak]

Looks like it's failing to process references with missing source like this:

<vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
  <vuln:reference href="http://www.securityfocus.com/bid/92630/references" xml:lang="en"/>
</vuln:references>

I duplicated the nvd cve parse test with the 2016 file and get the following error:

java.lang.NullPointerException
    at org.owasp.dependencycheck.dependency.Reference.compareTo(Reference.java:145)
    at org.owasp.dependencycheck.dependency.Reference.compareTo(Reference.java:28)
    at java.util.TreeMap.compare(TreeMap.java:1290)
    at java.util.TreeMap.put(TreeMap.java:538)
    at java.util.TreeSet.add(TreeSet.java:255)
    at org.owasp.dependencycheck.dependency.Vulnerability.addReference(Vulnerability.java:110)
    at org.owasp.dependencycheck.data.update.nvd.NvdCve20Handler.endElement(NvdCve20Handler.java:205)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
    at javax.xml.parsers.SAXParser.parse(SAXParser.java:328)
    at org.owasp.dependencycheck.data.update.nvd.NvdCve_2_0_HandlerTest.testParseWithMissingReferenceSources(NvdCve_2_0_HandlerTest.java:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:156)
    at mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
    at mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:37)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

java.lang.AssertionError: Exception thrown during parse of 2016 CVE version 2.0?

    at org.owasp.dependencycheck.data.update.nvd.NvdCve_2_0_HandlerTest.testParseWithMissingReferenceSources(NvdCve_2_0_HandlerTest.java:78)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.

[Steve Chernyak]

Ignore the line numbers in the Reference class. I added a try/catch block to get the debugger to stop on the npe.

[Jeremy Long]

The error with references was patched in this commit: https://github.com/jeremylong/DependencyCheck/commit/bcd6634d8af1303de08d00e46f07c5bb5b131672#diff-cfffe619a8a05677b2f46366836b488a

--Jeremy

On Sun, Sep 4, 2016 at 11:47 PM, Steve Chernyak <steve.c...@gmail.com> wrote:

Ignore the line numbers in the Reference class. I added a try/catch block to get the debugger to stop on the npe.

--

You received this message because you are subscribed to the Google Groups "Dependency Check" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dependency-check+unsubscribe@googlegroups.com.