Hello,
My name is Nuthan Munaiah and I am a Senior Researcher at Secure Decisions. I and my colleague, Chris Horn, were on the Securing Critical Projects WG meeting on January 14, 2021 when the
Open Source Project Criticality Score Program was mentioned. The program seems similar to a project I worked on as a Graduate Student at RIT called
reaper (
paper and
dataset). In an email conversation with Abhishek Arya, we found that the Criticality Score Program implements some of the parameters that
reaper uses.
When looking at the
results from the Criticality Score Program, we wondered if the criticality score is merely a proxy for popularity. As it turned out, there are a few comments [1,2,3] alluding to this question on the
Hacker News Discussion.
We evaluated this hypothesis by assessing the correlation between criticality score of a repository and its popularity (quantified using GitHub Stargazers). The outcome from the analysis (shown in the table below) was interesting and we thought the Group could benefit from the insights as well.
| Language | ρ | Effect | p | Significant |
|------------|----------|----------|-------------|-------------|
| rust | 0.417577 | Moderate | 7.66115e-10 | Yes |
| ruby | 0.404109 | Moderate | 2.9531e-09 | Yes |
| c# | 0.382657 | Moderate | 2.24522e-08 | Yes |
| javascript | 0.368158 | Moderate | 8.16308e-08 | Yes |
| java | 0.337799 | Moderate | 9.99069e-07 | Yes |
| c++ | 0.321293 | Moderate | 3.50299e-06 | Yes |
| php | 0.287965 | Weak | 3.55208e-05 | Yes |
| go | 0.284187 | Weak | 4.53817e-05 | Yes |
| c | 0.255176 | Weak | 0.000265666 | Yes |
| shell | 0.222957 | Weak | 0.00150682 | Yes |
| python | 0.169501 | Weak | 0.0164191 | Yes |
Interpretation: Yes, criticality score of a repository is positively correlated with its popularity but the effect is not as strong as some of the comments [1,2,3] from the Hacker News Discussion seems to suggest.
Thank you,
Nuthan Munaiah
References