iCal fuzzing
Gary Kwong
exploits in calendar code:
http://www.digitaldwarf.be/products.html
Although not necessarily a priority at this stage, security in
applications is always a factor, especially since Calendar is nearing
0.9 / 1.0
Some things to note:
- I'm not sure if the original python iCal fuzzer is licensed in a way
suitable for Mozilla code.
- Could we somehow incorporate this into some of the automated tests
that we are currently proposing to set up?
-- Ideally, the resulting testcase should be reproducible, reducible (to
the smallest malicious area) and as automated / scalable as possible,
i.e. simply being able to parse in an iCal file via command-line and
output the error message and quit.
-- We have to tweak the fuzzer to output standard iCal files that are
accepted by Calendar code / conform to some RFC standard (Clint told me
some of the output, like a 3,000 character category description or
something like that, wasn't supported in Calendar)
-- In the majority of cases, the correct error code / message should
spit out, but should the ultimate result be different, that could be a
software defect, and possibly even an exploit.
-- Such tests can be continuously run on spare CPU cycles on any machine
to harness the power of distributed computing (not exactly similar, but
the idea is there)
Thoughts?
-Gary
Michiel van Leeuwen
> I managed to find an iCal fuzzer that I thought might be useful to find
> exploits in calendar code:
>
> http://www.digitaldwarf.be/products.html
> - Could we somehow incorporate this into some of the automated tests
> that we are currently proposing to set up?
I don't think that's a good idea. The reason is that the fuzzer always
creates different output. If you would run it automatically, you don't
know if any error is caused by a code change or just by this one ics file.
> -- Ideally, the resulting testcase should be reproducible, reducible (to
> the smallest malicious area) and as automated / scalable as possible,
> i.e. simply being able to parse in an iCal file via command-line and
> output the error message and quit.
That's why xpcshell exists :)
> -- We have to tweak the fuzzer to output standard iCal files that are
> accepted by Calendar code / conform to some RFC standard (Clint told me
> some of the output, like a 3,000 character category description or
> something like that, wasn't supported in Calendar)
Ehm, having to tweak the fuzzer is not the goal of it, I guess. I mean,
you can't tweak other calendar clients either. So sunbird/lightning
should just work with whatever you throw at it. No artificial limits.
Michiel
Gary Kwong
> I don't think that's a good idea. The reason is that the fuzzer always
> creates different output. If you would run it automatically, you don't
> know if any error is caused by a code change or just by this one ics file.
The way jsfunfuzz.js does it is that it also ignores known errors from
certain outputs, though I'm not entirely sure. A log of potential errors
could be generated separately at the same time for a dev to look through
and filter out invalid cases.
>
>> -- We have to tweak the fuzzer to output standard iCal files that are
>> accepted by Calendar code / conform to some RFC standard (Clint told
>> me some of the output, like a 3,000 character category description or
>> something like that, wasn't supported in Calendar)
Pardon my mis-quote; it was something else that wasn't generated
properly by the fuzzer, not Calendar's fault. :|
-Gary
Ray Kiddy
> Gary Kwong wrote:
>> I managed to find an iCal fuzzer that I thought might be useful to
>> find exploits in calendar code:
>>
>> http://www.digitaldwarf.be/products.html
>
>> - Could we somehow incorporate this into some of the automated tests
>> that we are currently proposing to set up?
>
> I don't think that's a good idea. The reason is that the fuzzer always
> creates different output. If you would run it automatically, you don't
> know if any error is caused by a code change or just by this one ics file.
>
What is not a good idea? Using a fuzzer to create tests, or running a
fuzzer as part of a test? Of course, using a fuzzer to find errors,
which can then be characterized and turned into tests is useful, as
Jesse Ruderman's experience with jsfunfuzz illustrates.
Theoretically, a fuzzer can be run as part of an automated test, but in
practice it is too hard to figure out what a failure would mean.
>> -- Ideally, the resulting testcase should be reproducible, reducible
>> (to the smallest malicious area) and as automated / scalable as
>> possible, i.e. simply being able to parse in an iCal file via
>> command-line and output the error message and quit.
>
> That's why xpcshell exists :)
Yes, and an error found by a fuzzer would lead to the creation of a test
case that would be run by xpcshell. Again, I think that this is how
Jesse used his fuzzer.
>
>> -- We have to tweak the fuzzer to output standard iCal files that are
>> accepted by Calendar code / conform to some RFC standard (Clint told
>> me some of the output, like a 3,000 character category description or
>> something like that, wasn't supported in Calendar)
>
> Ehm, having to tweak the fuzzer is not the goal of it, I guess. I mean,
> you can't tweak other calendar clients either. So sunbird/lightning
> should just work with whatever you throw at it. No artificial limits.
>
One always hears the criticism that fuzzers do not find the same bugs
that people would find. But that may not be relevant. A fuzzer finds
some bugs. Perhaps these bugs have obvious visibility to a user and
perhaps not.
Without getting into a philosophical discussion of what a bug "means", a
bug is a bug. If a fuzzer finds a bug, the decision about what to do
with it is for the organization to make.
- ray
> Michiel
Michiel van Leeuwen
> What is not a good idea? Using a fuzzer to create tests, or running a
> fuzzer as part of a test? Of course, using a fuzzer to find errors,
> which can then be characterized and turned into tests is useful, as
> Jesse Ruderman's experience with jsfunfuzz illustrates.
Using a fuzzer as part of regression tests is not a good idea. If you
would do that, two things would change between runs: the code and the
testcase. You can't tell which change is to blame for a regression.
But I agree that some sort of automated run of fuzzer test is a good
idea. Any automation of testing is a good thing. But see below.
>>> -- Ideally, the resulting testcase should be reproducible, reducible
>>> (to the smallest malicious area) and as automated / scalable as
>>> possible, i.e. simply being able to parse in an iCal file via
>>> command-line and output the error message and quit.
>>
>> That's why xpcshell exists :)
>
> Yes, and an error found by a fuzzer would lead to the creation of a test
> case that would be run by xpcshell. Again, I think that this is how
> Jesse used his fuzzer.
>
With xpcshell I was referring to running the fuzzer from the
commandline. xpcshell scripts are a nice way to do that.
Once a bug is found, a standalone testcase should be made that does not
use the fuzzer directly, but only the output that caused the problem.
> Without getting into a philosophical discussion of what a bug "means", a
> bug is a bug. If a fuzzer finds a bug, the decision about what to do
> with it is for the organization to make.
I don't really know what the ics fuzzer is trying to find. It would be
good at finding crash bugs, but from looking at the current bug reports,
I don't think crashing is happening a lot. (I think crashes is what
jsfunfuzz tried to find)
Something else is checking if the parses works correctly. But with a
fuzzer, you don't know what the expected result is, so you can't
(automatically) check if the parser got the same result.
The only thing left is checking if the parser doesn't spit out any
errors. That does requires knowing that the fuzzer always gives ics
files that are completely correct. I don't know if it does that.
Michiel