Presenting an ASAN log aggregation solution

[Samy Bahra]

Hi all,

Wanted to share something with you that we’ve built over the last few weeks and have been using with great success in our environments along with some of our customers. We use a combination of memory sanitizers here at Backtrace, including ASAN.

The issues we faced were:
1. It’s quite difficult to aggregate the errors and feed them into our engineering workflows
2. It’s quite easy to miss the fact that you might have introduced a new non-benign leak or memory error.
3. No automation around creation of tickets.
4. No deduplication of call sites across reports.

Details for all of this are up in https://engineering.backtrace.io/posts/sanitizers/ - essentially, all you have to do is HTTPS POST / curl your ASAN or TSAN logs into a server and it will aggregate the data for you in a more actionable way. You can also drill down into the raw logs of an individual error. It does require that you use log_path along with your relevant tool’s *SAN_OPTIONS.

I set up a seeded environment with an example from our test corpus (mix of real world applications and synthetic) up at https://sanitizers.sp.backtrace.io/p/sanitizers/triage (username: sanitizers / password: appleorangewat1). If you click on the “Saved Views” button (left-most icon on the top right) then the “All” tab, you can see some example views we found particularly useful here.

Currently, I’ve only added support for address sanitizer, leak sanitizer and thread sanitizer, but if folks find this useful for other tools, please let me know and I’m happy to add support! Feedback is also greatly appreciated.

One thing that made this a bit more challenging is unfortunately the output grammar is ambiguous and so a lot of tricks had to be played for heuristics to detect some components of the output.

Hope others find this useful as well.