CPEs not being found (e.g., commons-io, commons-lang, velocity, lucene, h2, more)

[Mike Yawn]

I'm having an issue similar (perhaps identical) to the one posted a while back about CPEAnalyzer failing to find CPEs for identifiers (despite what seems to be plenty of evidence).    I tried posting on that thread but my post was deleted, so I"ll post separately.

When I run dependency-check, I get output that shows no vulnerabilities.  Digging into this, the likely reason seems to be that very few dependencies have a CPE associated with them.  

As a test, I created a sample project and gave it nearly all the dependencies of dependency-check-core (since that;'s an easy comparison for anyone with the checker software installed.   I deleted only the dependency-check-utils dependency.   When I run check, 23 dependencies are found, but only 4 of them have CPEs associated, and all 4 of those are low confidence (commons-compress, commons-collections, jsoup, and javamail are the 4 that are found).

I took Hans' suggested from the other thread and added the suggested timestamp fallback to 'now'; after adding that code I ran purge and update-only; then another check but the results were the same.  

It seems we don't have index entries that I would expect us to have.  I changed the logging level of some of the logger output in CPEAnalyzer and added some of my own; here for example is the logging where it tries to find the CPE for commons-io.   I would expect that to be in the CPE database, but it doesn't appear in any IndexEntry:

vendor search: commons-io http://commons.apache.org/proper/commons-io/
product search: commons-io
Trying to find CPE for commons-io http://commons.apache.org/proper/commons-io/ commons-io
Search string product:(commons\-io) AND vendor:(commons\-io http\:\/\/commons.apache.org\/proper\/commons\-io\/) search returns 11 11 1.2654142
Verifying entry: IndexEntry{vendor=commons_wikis_project, product=commons_wikis}
Verifying entry: IndexEntry{vendor=not_yet_commons_ssl_project, product=not_yet_commons_ssl}
Verifying entry: IndexEntry{vendor=io-socket-ssl, product=io-socket-ssl}
Verifying entry: IndexEntry{vendor=apache, product=commons-compress}
Verifying entry: IndexEntry{vendor=apache, product=commons-jelly}
Verifying entry: IndexEntry{vendor=apache, product=commons_fileupload}
Verifying entry: IndexEntry{vendor=apache, product=commons-httpclient}
Verifying entry: IndexEntry{vendor=apache, product=commons_collections}
Verifying entry: IndexEntry{vendor=apache, product=commons_beanutils}
Verifying entry: IndexEntry{vendor=apache, product=commons_email}
Verifying entry: IndexEntry{vendor=apache, product=apache_commons_daemon}
vendor search: commons-io http://commons.apache.org/proper/commons-io/ The Apache Software Foundation Apache Commons IO
product search: commons-io Apache Commons IO
Trying to find CPE for commons-io http://commons.apache.org/proper/commons-io/ The Apache Software Foundation Apache Commons IO commons-io Apache Commons IO
Search string product:(commons\-io Apache Commons^5 IO^5) AND vendor:(commons\-io http\:\/\/commons.apache.org\/proper\/commons\-io\/ The Apache^5 Software Foundation Apache^5 Commons^5 IO^5) search returns 23 23 3.0286756
Verifying entry: IndexEntry{vendor=vcard4j, product=vcard4j}
Verifying entry: IndexEntry{vendor=neo4j, product=neo4j}
Verifying entry: IndexEntry{vendor=libpam4j_project, product=libpam4j}
Verifying entry: IndexEntry{vendor=j2store, product=j2store}
Verifying entry: IndexEntry{vendor=entity_api_project, product=entity_api}
Verifying entry: IndexEntry{vendor=shipwire_api_project, product=shipwire_api}
Verifying entry: IndexEntry{vendor=storage_api_project, product=storage_api}
Verifying entry: IndexEntry{vendor=mangoswebv4_project, product=mangoswebv4}
Verifying entry: IndexEntry{vendor=4homepages, product=4images}
Verifying entry: IndexEntry{vendor=phonearabs4_project, product=phonearabs4}
Verifying entry: IndexEntry{vendor=scramdisk_4_linux, product=scramdisk_4_linux}
Verifying entry: IndexEntry{vendor=po4a, product=po4a}
Verifying entry: IndexEntry{vendor=bento4, product=bento4}
Verifying entry: IndexEntry{vendor=all4www, product=all4www-homepagecreator}
Verifying entry: IndexEntry{vendor=myweb4net, product=myweb4net_browser}
Verifying entry: IndexEntry{vendor=falt4, product=falt4_extreme}
Verifying entry: IndexEntry{vendor=4site, product=4site_cms}
Verifying entry: IndexEntry{vendor=iphone4, product=iphone4.tw}
Verifying entry: IndexEntry{vendor=search_api_autocomplete_project, product=search_api_autocomplete}
Verifying entry: IndexEntry{vendor=a4desk, product=a4desk_flash_event_calendar}
Verifying entry: IndexEntry{vendor=j00lean-cms, product=j00lean-cms}
Verifying entry: IndexEntry{vendor=j2k-codec, product=j2k-codec}
Verifying entry: IndexEntry{vendor=provider4u, product=vsftpd_webmin_module}
Verifying entry: IndexEntry{vendor=isdn4linux, product=isdn4linux}
Verifying entry: IndexEntry{vendor=mad4media, product=com_mad4joomla}

Am I missing data?    If so, any ideas as to why, and how to workaround or fix it?

Thanks,
Mike

[Corum, Michael]

I’ve been getting this error since March 30^th.

 

[DependencyCheck] Exception Caught: org.owasp.dependencycheck.data.update.exception.UpdateException

[DependencyCheck] Cause: Error making HTTP GET request.

[DependencyCheck] Message: Unable to download the NVD CVE data.

 

It looks like there are Jenkins plugin updates that are required but those depend on components that also require a very current version of Jenkins v2.  It sounds like we can’t use Jenkins v1 anymore.  I don’t know if this is related to your issue or not.  May not be.

 

Michael Corum 

VP, Technical Architecture Solutions

 

RGA Reinsurance Company

16600 Swingley Ridge Road

Chesterfield, Missouri 6301701706

T 636.736.7066 

www.rgare.com

 

From: "dependen...@googlegroups.com" <dependen...@googlegroups.com> on behalf of Mike Yawn <mi...@theyawns.com>
Date: Monday, April 2, 2018 at 2:54 PM
To: "dependen...@googlegroups.com" <dependen...@googlegroups.com>
Subject: CPEs not being found (e.g., commons-io, commons-lang, velocity, lucene, h2, more)

 

External e-mail. Use caution! / Courriel externe. Faites attention!

---

--
You received this message because you are subscribed to the Google Groups "Dependency Check" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Yawn]

I should mention that I've updated the NIST URLs due to the recent changes; also this issue first appeared before that change was made.  So I'm reasonable certain this is unrelated to the URL changes.

Mike

[Mike Yawn]

I think that's a different issue -- there's a separate thread about the NIST URL updates.  I had that issue but was able to change the configuration of my project POM to override the URLs with the new ones.    In my case I'm just doing a Maven build, I haven't tried to get this running in our continuous integration environment yet but that will be next up.

Thanks
Mike

To unsubscribe from this group and stop receiving emails from it, send an email to dependency-check+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hans Aikema]

Mike,

Did you check the size of your database after the purge / update-only? As indicated in the thread you mentioned the index that’s used for resolving cpe’s is constructed in memory based on the contents of that database.

(when using the defaults the database is found as an h2 database in <mvnrepository>/org/owasp/dependency-check-data/3.0 which, at the time of writing, for my system is approximately 309 megabyte (323821568 bytes) )

kind regards,
Hans Aikema

> On 2 Apr 2018, at 22:13, Mike Yawn <mi...@theyawns.com> wrote:
>
> I think that's a different issue -- there's a separate thread about the NIST URL updates. I had that issue but was able to change the configuration of my project POM to override the URLs with the new ones. In my case I'm just doing a Maven build, I haven't tried to get this running in our continuous integration environment yet but that will be next up.
>
> Thanks
> Mike
>
>
> On Monday, April 2, 2018 at 4:00:14 PM UTC-4, bluespower27 wrote:
> I’ve been getting this error since March 30th.
>
>
> [DependencyCheck] Exception Caught: org.owasp.dependencycheck.data.update.exception.UpdateException
>
> [DependencyCheck] Cause: Error making HTTP GET request.
>
> [DependencyCheck] Message: Unable to download the NVD CVE data.
>
>
> It looks like there are Jenkins plugin updates that are required but those depend on components that also require a very current version of Jenkins v2. It sounds like we can’t use Jenkins v1 anymore. I don’t know if this is related to your issue or not. May not be.
>
>
> Michael Corum
>
> VP, Technical Architecture Solutions
>
>
> RGA Reinsurance Company
>
> 16600 Swingley Ridge Road
>
> Chesterfield, Missouri 6301701706
>
> T 636.736.7066
>
> www.rgare.com
>
>
>
> From: "dependen...@googlegroups.com" <dependen...@googlegroups.com> on behalf of Mike Yawn <mi...@theyawns.com>
> Date: Monday, April 2, 2018 at 2:54 PM
> To: "dependen...@googlegroups.com" <dependen...@googlegroups.com>
> Subject: CPEs not being found (e.g., commons-io, commons-lang, velocity, lucene, h2, more)
>
>
> External e-mail. Use caution! / Courriel externe. Faites attention!
>

> To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "Dependency Check" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.

[Mike Yawn]

I've checked it and its nearly identical to yours (326037504 bytes).  

I've added a couple of dependencies that have known vulnerabilities and the CPEs of those were found OK.   So I'm wondering if the software works differently than I assumed ... I assumed we'd build a CPE first, and use that to look up vulnerabilities.   So if you can't build a valid CPE, you'll never find associated vulnerabilities.

Now I'm wondering if I have it backwards ... it looks like maybe we're doing a Lucene search against a set of CPEs, but maybe that list of CPEs only includes those with known vulnerabilities.   If that's the case, you'll never find a CPE unless the software has (or in the past had) a vulnerability. 

Mike

[Jeremy Long]

dependency-check does not build CPEs - rather it collects textual data from the dependencies and uses the data to perform lookups in a Lucene Index of the CPE from the NVD. More information can be found on the documentation site: https://jeremylong.github.io/DependencyCheck/general/internals.html

--Jeremy

[Jeremy Long]

Mike,

Can you open up a ticket on github for this issue? 

https://github.com/jeremylong/DependencyCheck/issues/new

--Jeremy