Potential Security Audit for libjpeg-turbo

[Derek Zimmer]

Hello everyone in the libjpeg-turbo community!

I'm Derek from the Open Source Tech Improvement Fund (ostif.org). We've been working with open source communities to improve their security and libjpeg-turbo is a crucial piece of infrastructure that we've identified as a potential project for us to work on together. We've worked with dozens of other projects in the past and I'd be happy to give you references on our working relationships with past open source communities.

I'm *hoping* that this is the appropriate place to have this discussion. If there's a better channel to communicate, please reach out to me.

Would you be interested in working with our security experts on libjpeg-turbo? This would likely come in the form of code review, work on integrating some security tools that give you valuable results without wasting your time, and some steps toward securing your supply chain (with help along the way with anything that you need). I see that you're on ossfuzz, but there's also some recent overflow issues so this review would likely entail manual review to look for logic/design level issues in addition to seeking better code coverage with fuzzing or optimizing existing fuzzers to find memory safety problems. With so many builds for various platforms, some differential fuzzing might also be interesting for seeking out potential issues. Any custom tools or code that are created by our teams for these projects are given to you, for free at the end of the engagement.

Additionally, OSTIF is running a pilot program where we compensate projects directly for the additional work that is done by you. This usually comes in the form of a direct payment at the end of the engagement provided that we hit all of the agreed-upon milestones. We know that the most valuable resource for open source developers is time, and that while our engagements are designed to be as lightweight as possible, there are still some time commitments that are needed for us to succeed. I saw that you're interested in additional funding here and thought I should mention this pilot. We can discuss the details of how that works as well if you're interested.

We can talk some specifics if this interests you!

All the best,

Derek Zimmer

Executive Director

Open Source Tech Improvement Fund

[DRC]

I would be a lot more open to your overture if it didn't come across
like a buzzword-heavy form letter addressed to a middle manager.  (Hint:
Open source projects don't have a "supply chain.")  I am an independent
open source developer and have (barely) managed to eek out a living as
the sole maintainer of libjpeg-turbo, VirtualGL, and TurboVNC for the
past 14+ years. Thus, e-mailing me directly with technical details would
have been much better received than blasting business speak to a mailing
list that is intended for low-level discussions among libjpeg-turbo
contributors.

libjpeg-turbo was mostly a loss leader (subsidized by my work on
VirtualGL and TurboVNC) until 2018, when I had to put so much pro bono
labor into finishing the 2.0 release that I went into debt because of my
inability to take on enough paying contracts that year.  I had to put my
foot down and say that I could no longer afford to contribute any pro
bono labor to libjpeg-turbo and that, without general funding, the
feature set would essentially have to be frozen.  Fortunately a couple
of companies, including Google, have stepped up since then to provide
general funding, so I no longer have to lose money maintaining the
project.  However, libjpeg-turbo still runs on a shoestring budget.  The
General Fund only pays for a couple of days of my labor per month, and I
had to borrow against that fund until the end of 2023 in order to finish
the 3.0 beta release.  All of the OSS-Fuzz integration work you mention
was funded by Google, and that is generally what this project needs more
than anything else-- funding.  When I say "funding", I mean general
funding to maintain the project in the manner that has proven to be the
most beneficial.  I don't mean funding to pursue someone else's agenda,
the benefits of which are unproven.  OSS-Fuzz is beneficial because it
has an extremely high signal-to-noise ratio, so it has spared me more
labor than was required to integrate it.  I can't say that about very
many other tools, except for the ones that I wrote myself based on my
intimate knowledge of the code base.

The overflows you cite were in the dev branch, the result of new
features that I was still debugging.  They were not actually released
into the wild.  libjpeg-turbo has not had an issue filed against it in
the CVE system since 2021, nor have any such issues yet been discovered
in 3.0 beta since it was released a month ago.  That's mostly because of
OSS-Fuzz.  Since integrating OSS-Fuzz in 2020, it has flagged 16 issues
with libjpeg-turbo, but only two of those issues were in released code. 
One of those two issues was innocuous, and the other was an issue in the
rarely-used 12-bit-precision code path (which wasn't even included in
99.999% of public libjpeg-turbo binary builds/implementations at the
time, including ours.)  I have worked with some security experts
(Cure53, specifically) in the past whose audits were very helpful, and
Mozilla paid for me to co-author a report
(https://libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf)
based on that work.  However, most of the security issues filed against
libjpeg-turbo (particularly in the CVE system) were unexploitable, and
at least one of those issues was blatantly invalid (a bug in a
downstream library resulting from its incorrect usage of libjpeg-turbo,
not a bug in libjpeg-turbo itself.)  Thus, I have a lot of skepticism
regarding security audits and reporting processes in general.

I definitely cannot introduce any dependence on custom tools that are
licensed more strictly than the licenses under which libjpeg-turbo is
made available.  That would include closed-source/proprietary tools as
well as, for instance, GPL-licensed tools.  Anything that goes into the
libjpeg-turbo source tree must be under a BSD-style license.  Assuming
that your custom tools are licensed compatibly with our project, then my
openness to introducing them would depend largely on how automated and
useful they are.  If they save me more time in the long term than is
required to integrate them, then OK.  If, however, they create more work
for me in the long term, then nope.  Because of the aforementioned 2023
budget shortfall, my receptiveness to anything other than critical bug
fixes is extremely low at the moment.  Of course you're going to claim
that your tools give valuable results without wasting my time, but my
experience is that about 9 out of 10 people who have made that claim to
me since 2009 were full of it. If you want to provide more technical
details off-list, then feel free to e-mail me directly:
https://libjpeg-turbo.org/About/Contact.  Just understand that I don't
have the time (== money) to be anything other than a hard skeptic when
it comes to this project, so the burden of proof is on you.

In the past, a model that has worked for security audits (including the
aforementioned Cure53 audit) is to go ahead and do the audit and send me
the report, and that will demonstrate the general utility of your
process.  libjpeg-turbo derives from a code base that originated in
1991, before many professional software developers were even born.  Tom
Lane, the founder of libjpeg, worked on that code base for seven or
eight years.  At this point, I have worked on libjpeg-turbo for nearly
double that amount of time.  There isn't much that someone can tell me
about its logic and design that I don't already know, and as an
ISO/ITU-T reference implementation that implements a 32-year-old
pseudo-standard API with exposed structures, there isn't much of that
design that I can even change at this point.  I have a lot of interest
in fixing real bugs but very little interest in fixing hypothetical
ones.  If you think you can optimize our fuzzers or improve coverage,
then great.  Do it.  Submit a PR.  That is certainly something that I am
willing to integrate using general fund money.  If you think you can
find legitimate bugs or security exploits, then great.  Find them.  But
don't expect me to agree to help you audit my code base until you have
proven that you can provide something more than I can already provide
myself.

Thanks,
DRC

[Derek Zimmer]

Hello DRC,

I will reach out directly via the contact you've provided. I hear all of your concerns and I'd urge you to take a look at our past work, particularly with solo maintained projects like curl and jackson. We are not here to dictate anything about your project, and we want to respect your time and your work. We can work in whatever capacity is best for you.

I'll be in touch shortly!

Best,

Derek Zimmer