Setting up Clusterfuzz for LayoutQuickCheck

[Pavel Panchekha]

Hello Jonathan and the Clusterfuzz team,

I'm a professor at the University of Utah, and my student Nathan and I have been working with Chris Harrelson (CC'd) and the Chrome Rendering Team to find under-invalidation bugs in Blink layout. We've built a fuzzer that finds this kind of correctness bug, called LQC, and have used it to find about a dozen confirmed Blink bugs.

We'd like to run LQC in Clusterfuzz so the Chrome team can access it easily. Unfortunately, I've been unable to get Clusterfuzz set up following the official instructions, and so I'm kind of at a loss for how to move forward on this. I've also read through the Clusterfuzz docs, but there seems to be a bit of a mismatch between the kinds of bugs LQC finds (correctness bugs, no crashes) and what Clusterfuzz is set up to do.

Jonathan, would you or anyone from the Clusterfuzz dev team be willing to meet with us to talk about how we can get LQC running on Clusterfuzz?

—Pavel Panchekha

[Oliver Chang]

Hi Pavel,

Thanks for reaching out! Would your fuzzer be what we consider to be a black box fuzzer? 

As you mention, ClusterFuzz typically reports crashes. For the correctness fuzzing we've done so far we've typically surfaced them as assertion crashes. 

How does your fuzzer find/report these correctness issues? i.e. How are you running Chrome and interpreting the results to make a judgement about whether something is a correctness bug or not? 

--

Oliver

[Pavel Panchekha]

Hi Oliver, (Chris -> BCC)

The fuzzer is black-box, yes. Basically, it randomly generates a web page, executes JavaScript on it, and the result of the JS execution tells us if there's a correctness bug (think of the JS as returning a boolean). Right now, we run Chromium through Selenium, and we also have a second frontend for Grizzly, Mozilla's fuzzing system. Unlike the fuzzers you are using, it doesn't do any kind of mutation or seeds, since it's already generating valid HTML and CSS and so on. It also doesn't need a build with santizers—we've been using official builds for our tests—because we don't expect to crash Chrome. (Though, I mean, it could use a sanitizer-on version of Chrome, and in principle might find a crash.)

Here's some more details about what LQC is doing and the bugs it is finding. Basically, when some JS code make a change to the page's CSS (by modifying a node's style field) the browser can reach a different layout than if that style was there when the page was loaded. Basically whenever that happens, it's a bug. So LQC generates: a web page; an initial CSS file for it; and a set of new CSS properties to set. It loads the initial page, with its initial CSS, and then runs JavaScript that sets the new properties. It then saves the layout data, asks Chrome to recompute the layout from scratch, and checks if any layout data changes. If it does, that means the web page + changes form a bug.

In terms of surfacing assertions, what we do for Mozilla's frontend is we trigger a custom JS function that they expose in test mode, which causes a crash. We could do the same for Chrome if a function like that is available. (Specifically, the function crashes Firefox and also accepts a string to print as the "stack trace" for their bug bucketing to work.) Or we could output special text with console.log, if that's accessible to ClusterFuzz, or make a web request to a service. Can the fuzzers communicate directly with Chrome (to run JS and get outputs back)? If yes, we could also have the fuzzer itself write out a file or even manually kill Chrome (though that would slow it down). Happy to meet to hammer this out.

—Pavel Panchekha

[Oliver Chang]

Hi Pavel,

Thanks for the explanation! 

A custom JS function only exposed in test mode that crashes sounds like the best/most reliable way to go for Chrome as well. Perhaps Chris or someone from his team would be able to help with this? I'm not sure if there is a function like this that exists already.

--

Oliver

[Pavel Panchekha]

Eh, I shouldn't have moved Chris to BCC!

Chris—do you know if Chrome exposes any function in testing builds that lets you purposely crash Chrome from JavaScript? That would make it easy to integrate LQC with Clusterfuzz. Who should I ask about this?

—Pavel Panchekha

[Chris Harrelson]

+Philip Rogers told me that there is a crash() on internals.idl. Should be available if you pass --expose-internals-for-testing on the commandline.

[Pavel Panchekha]

Wow, thanks for the quick answer, Chris!

Oliver, sounds like all the infrastructure we need is there. It should be possible to get something running pretty quickly! Can you help guide us a bit in setting up and testing Clusterfuzz?

—Pavel Panchekha

[Chris Harrelson]

One problem: internals.idl is only exposed to content_shell, not chrome. But looking at the implementation may give hints at a solution.

[Oliver Chang]

Would content_shell be sufficient to test what this fuzzer is testing, or do we actually need the full chrome? 

--

Oliver

[Pavel Panchekha]

I think content_shell will be enough. We just need to open pages, do layout, and run javascript. It should work headless too.

—Pavel Panchekha

[Philip Rogers]

Does internals.crash() show up in clusterfuzz? I didn't find any crashes when searching testcases on clusterfuzz but I don't see any filters to exclude it either. In any case, we could add a new internals assertion codepath if necessary.

[Pavel Panchekha]

Hi all, I think this thread got dropped. I'd like to get LQC running on Clusterfuzz, since it sounds like we have all the building blocks we need. I'm blocking on installing Clusterfuzz. I'm running macOS and getting the following error message in when calling local/install_deps.bash: An error occurred while installing cryptography==3.4.7 and then a long list of hashes. Separately installing the cryptography package works fine—but I guess the wrong hashes are passed? I checked the Pipfile and cryptography isn't listed there, so it must be some dependency...

We've also got access to a Windows and a Linux machine if either of those is more likely to work; I think we tried Windows but got a similar error there. I just want to get a test environment up, so if there's a VM or a Docker that is known-good, I'm happy to use that as well. I'm also happy to get on Meet or similar to debug things, that might be faster than emailing back and forth.

—Pavel Panchekha

[Chris Harrelson]

Hi all,

I think Pavel is just asking for help getting it to run on his computer...can someone help who has gotten it to work on Mac? Or is that not supported and Pavel should switch to Linux?

Thanks,

Chris

[Pavel Panchekha]

Yes, that's right—if I could get Clusterfuzz set up I think Nathan and I are happy to get LQC running...

—Pavel Panchekha

[Oliver Chang]

Hi Pavel,

Linux is the only supported development environment. Would you be able to use that instead for development? 

Let me know if you run into any issues -- happy to help get you set up :) 

Thanks,

--

Oliver

[Pavel Panchekha]

Yes, I'll try to set it up on my Linux box.

—Pavel Panchekha

[Pavel Panchekha]

Hi Oliver and others, I did get Clusterfuzz 2.5.6 working on Linux. I'll be trying to write a fuzz file for LQC next.

—Pavel Panchekha

[Oliver Chang]

Hey Pavel,

Thanks for the update! That's awesome to hear. Let me know if you run into any issues.

Cheers,

--

Oliver