A study comparing current vulnerable dependency detection tools

[Nasif Imtiaz]

Hi,

We recently completed a study where we compared ten existing vulnerable dependency detection tools including OWASP DC. We have submitted the paper to an academic conference and currently awaiting peer-review. The draft is attached. 

The study shows differences in the results from different tools and discusses the possible reasons behind differences. We will highly appreciate any feedback or research idea from you on this paper. 

************************************

Nasif Imtiaz

Graduate Research Assistant 

North Carolina State University

https://nasifimtiazohi.github.io/

[Mike Salsbury]

Thanks Nasif, that was an interesting read.

One question might be is it better to report many vulnerabilities through multiple paths.  And thus over report, but provide accurate locations.  Or to only report each vulnerable package once, on the assumption that once updated it will 'fix' all the vulnerabilities.  What do you think of each approach?

Also the commercial tools allow false positives to be 'marked' or 'accepted', so that they are excluded from future runs.  This isn't something that free tools I've seen currently provide.  I'd be interested to hear of any that do.

Are there any plans to do a similar analysis of dynamic analysis tools?

[Jeremy Long]

Nasif,

Thanks for sharing the analysis. I didn't see what version of ODC was used in the analysis - and I'm betting a large number of the false positives came from the Node JS analysis. With 6.0.3 we've made a drastic improvement on the FP for Node projects (see the release notes which recommends disabling one of the analyzers). However, FP and FN are always an issue.

Mike - regarding the ability to suppress false positives with ODC; please see https://jeremylong.github.io/DependencyCheck/general/suppression.html

--Jeremy

--
You received this message because you are subscribed to the Google Groups "Dependency Check" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dependency-che...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dependency-check/1645beba-7ce3-4302-8f5e-b0022b0798a6n%40googlegroups.com.

[Steve Springett]

Nasif,

First of all, thanks for sharing. There’s some great research in here. Kudos.

One thing in the paper I’m not seeing is component identity. SCA tools are designed to identify risk, not produce an accurate inventory. In many cases, ‘good enough’ is generally fine. Every SCA tool I’ve ever used (commercial and open source) has produced inaccurate inventories at times. This is especially true for open source components that are commonly used in other things, or which have been modified in some way. 

Most analysis that I’ve seen or have been involved with start with how accurate is the identification of component inventory. This can be measured and compared with competing tools.

Once an accurate inventory has been baselined among the tools, then various analysis methods and results can be evaluated. By separating out component identity from vulnerability analysis, you’ll also be able to account for newer, non-SCA tools in the mix, which don’t have to make educated guesses on component identity, but which are provided an accurate inventory to start with (via SBOM).

Also, I would be cautious with regard to static analysis and how it can determine if components are used or reachable within a call graph. The amount of false negatives in this area is quite high IME. Use of non-standard design patterns, unknown libraries and frameworks, polyglot environments, etc, introduce an enormous amount of complexity that static analysis typically does not support, at least out of the box. Also with regard to reachability, reachability can only be determined when the application is operating as expected. Unexpected behavior, such as when an adversary pivots from one vulnerability to another, may alter whether or not a component is reachable or not.

The paper incorporates a lot of different types of tools, with varying methodologies, and analysis methods. As a potential follow-up idea, it may be really interesting to see how these differences benefit certain use cases or industries.

Thanks again.

— Steve

To view this discussion on the web visit https://groups.google.com/d/msgid/dependency-check/CAKAumBFJpfw%3DqLaeU%2BhLaUNpYP-esGN1Ov2_v3t7%2BkenpoCxDQ%40mail.gmail.com.

[Nasif Imtiaz]

Hi,

Thanks a lot for your responses. For some reason this thread was not subscribed, and I didn't get any email notifications. So I thought this post went unnoticed. Thanks a lot for your feedback and sorry for me not responding back immediately. 

@Mike, 

One question might be is it better to report many vulnerabilities through multiple paths.  And thus over report, but provide accurate locations.  Or to only report each vulnerable package once, on the assumption that once updated it will 'fix' all the vulnerabilities.  What do you think of each approach? -  The usability aspect of presentation of the alerts was not a goal of this study, therefore, I won't have much to comment on that. Anyways, no matter what way you group the results, I think there should be some way to know about each detail. As different vulnerabilities in the same package can get fixed in different version, developers may individually assess the risk if for some reason they don't just want to adopt the latest release. 

Also the commercial tools allow false positives to be 'marked' or 'accepted', so that they are excluded from future runs.  This isn't something that free tools I've seen currently provide.  I'd be interested to hear of any that do. -  GitHub has an option to dismiss alerts under various reasons. Also, Jeremy pointed out a way to suppress with OWASP DC. 

Are there any plans to do a similar analysis of dynamic analysis tools? - Are you talking about DAST tools like a fuzzer or SCA  tools that incorporate dynamic analysis? For the former, evaluating DAST tools is not in my research plan. But I think there is already quite a lot of work been done on dynamic analysis tools. If you want, I can try to find some references. 

@Jeremy,

We used version 5.3.2. In the related issue, https://github.com/jeremylong/DependencyCheck/issues/2796, I see that 5.3.2 was probably free of the Node analyzer bug. 

However, FP and FN are always an issue. - One thing we pointed out in this paper is how OWASP DC inflates alerts by reporting the same CVE over multiple related packages. This is probably due to relying on the CPE configuration presented in the CVE data. This discussion is presented in Section 5.1. As one example, CVE-2014-3625 was only reported for spring-webmvc by MSV, Snyk, Steady, and Commercial A. However, OWASP DC reported this CVE for five separate spring packages. I would appreciate your opinion on this matter as we're currently revising the paper. 

Regarding False Negative, what do you think are possible reasons besides an incomplete database? Could there be dependencies used in a project that tools are not able to pick up? What should a good tool do in that regard do you think?

Also, I can tell that is OWASP DC does not maintain its own vulnerability database like Snyk or GitHub, rather only incorporates data from third-party sources. Is it right?

@Steve,

There’s some great research in here. Kudos. - Thanks a lot. As a student, this is really encouraging for me.

I am not sure if I have fully understood your comment. Can you elaborate more on the term "component identity"? Does it simply mean the list of components/dependencies (inventory) ? In the study, we assume the tools would do both the things - 1) identifying the components, 2) and reporting if they contain any known vulnerabilities based on some vulnerability database. Our initial assumption was that most tools would output the same results (before a further advanced vulnerability analysis), however, we found that there is difference even in the basic cases. 

you’ll also be able to account for newer, non-SCA tools in the mix, which don’t have to make educated guesses on component identity, - Can you elaborate how we can incorporate non-SCA tools? Are you referring to the part where some tools perform reachability analysis which can be done by non-SCA tools as well if the vulnerability report were already known?

I agree with the limitations on evaluating reachability through both static and dynamic analysis. Code analysis has many pitfalls, and we probably need further research to understand how acceptable is such analysis to the developers when assessing risk of vulnerabilities in dependencies. 

The paper incorporates a lot of different types of tools, with varying methodologies, and analysis methods. As a potential follow-up idea, it may be really interesting to see how these differences benefit certain use cases or industries. - Thank you for the suggestion. Do you think interviewing developers on what analysis methods fit well in what use cases can be a starting point to do such research?

Thanks,

Nasif

[Steve Springett]

Nasif,

For clarification, when I’m referring to "component identify”, I’m referring to the attributes in which the component can be identified. This includes (but is not limited to), coordinates (group, name, version), Package URL (purl), file hash, AST fingerprint, and SWID tag ID.

Expect SCA products to achieve >80% accuracy for real-world projects. I would love to see a study done specific to this topic actually. This range is from my own observations across multiple commercial/internal projects and companies and using all notable commercial SCA tools. I brought this up because the paper assumed that components were accurately identified prior to performing the vulnerability analysis. Were they? If so, was the projects representative of the types of projects companies actually create? I don’t know. But what I do know is that open source is the ultimate supply chain, and components are constantly forked, modified, repurposed, repackaged, renamed, and redistributed. I also know that naming is a very hard problem to solve.

There are two competing ways that organizations create an inventory for the components they use: rely on an SCA product to tell them, or integrate build-time SBOM creation and enhancement into pipelines themselves. The later is a bit more work but does have the potential to dramatically increase accuracy. 

Some non-SCA, SBOM-first tools to look at are SCANOSS and Dependency-Track.
 
As far a recommendations for future study (methodologies, analysis methods, etc), including developers would be good, but ideally, a study would include software supply chain experts and vendors.


Best,

— Steve

To view this discussion on the web visit https://groups.google.com/d/msgid/dependency-check/fb87a755-5054-412a-bc88-3a703a510f72n%40googlegroups.com.

[Nasif Imtiaz]

Thanks a lot, Steve. This was very educational for me and I will keep that in mind during future studies.

Nasif

You received this message because you are subscribed to a topic in the Google Groups "Dependency Check" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dependency-check/JrhSBcAkzkw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dependency-che...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dependency-check/6bdf17c5-3026-4aa4-a4a6-35ac58d4ff0b%40Spark.

--