How Long Do Vulnerabilities Live in the Code? A Large-Scale Empirical Measurement Study on FOSS Vulnerability Lifetimes

[danakj]

Hi all,

An interesting paper on statistical analysis of software security that features Chromium prominently.

https://www.usenix.org/system/files/sec22summer_alexopoulos.pdf

Some quotes to wet the appetite:

How long do vulnerabilities live in the repositories of large, evolving projects? Although the question has been identified as an interesting problem by the software community in online forums, it has not been investigated yet in adequate depth and scale, since the process of identifying the exact point in time when a vulnerability was introduced is particularly cumbersome.

With our approach, we perform the first large-scale measurement of Free and Open Source Software vulnerability lifetimes, going beyond approaches estimating lower bounds prevalent in previous research. We find that the average lifetime of a vulnerability is around 4 years, varying significantly between projects (~2 years for Chromium, ~7 years for OpenSSL). The distribution of lifetimes can be approximately described by an exponential distribution.

Figure 7 shows how vulnerability lifetimes progressed over the years for the dataset as a whole, as well as for Firefox, Chromium and Linux. These were the projects that had enough CVEs (>20) for each year to confidently assess their lifetime over an extended period. Specifically, the grey area in the plots covers the years before the first year when at least 20 CVEs with fixing commits were available for the project. Overall (Figure 7a), vulnerability lifetimes show a sign of increase over the years with some fluctuation.

Considering the other 3 selected projects in particular, for Chromium (Figure 7c) and Linux (Figure 7d) we can observe clear increasing trends, whereas for Firefox (Figure 7b), vulnerability lifetimes are stable, even with a slight decreasing trend. It is interesting to note that although the overall increasing trends for Chromium and Linux are similar, lifetimes for Chromium can fluctuate significantly over the years, while the values for Linux fluctuate less.

We established that there is no evidence to support that we are introducing (and consequently fixing) significantly fewer new vulnerabilities over time. We also made the following observations: (a) vulnerability lifetimes are on average lower than regular code age, and (b) for some projects this spread increases over time.

=====

I wonder if it would be interesting to reproduce the study but with all Chrome security bugs instead of just CVEs (~in-the-wild bugs). In particular I think it may say something more useful about fuzzers.

Enjoy,

Dana

[Mitch Phillips]

I think @Samuel Groß has done a study of the lifetime of Chrome (maybe just v8) CVEs very recently.

--
You received this message because you are subscribed to the Google Groups "memory-safety-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memory-safety-...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/memory-safety-dev/CAHtyhaQHBppwzd3OEB8mrpEA6e9v3hjX%3DN1ntwb-qfuajVf0nw%40mail.gmail.com.

[David Adrian]

For Chrome, would the equivalent age be the difference between FoundIn and the patch date?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/memory-safety-dev/CAFxCCyp9TPa%3DsNuH8GghSaTJed64s2rvz_psjnSbLBfut6fpbA%40mail.gmail.com.

[danakj]

On Mon, Jun 3, 2024 at 7:08 AM David Adrian <dad...@google.com> wrote:

For Chrome, would the equivalent age be the difference between FoundIn and the patch date?

FoundIn caps out at whatever the current stable is, so it does not tell you when a bug was introduced. We'd need a methodology like used in the paper to find a rough introduction date.

The patch date does not represent the fix reaching users. The call that t_{fix} in the paper, but want to measure until the time all hosts are patched, which is called t_{p}.

[Titouan Rigoudy]

If the bug reproduced on ClusterFuzz, we should get a regression range, and thus a much better estimate of when the bug was introduced.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/memory-safety-dev/CAHtyhaQMS%2Bq4%2B3y7fzpvpbLZanw%3DHG9-Dcr31aQBJgO2-FOBag%40mail.gmail.com.