Re-processing a crash dump

[Neil Williams]

Hello,

We are playing with a Socorro + ElasticSearch + Kibana setup. One thing I would like to do is to rerun the processor on a crash dump so that the dump is re-indexed in ES.

This is coming up when we are uploading missing symbol files after we have received a crash, and want to update the document to get the full(er) traces.

I could just re-post the dump, but I would prefer not to have the duplicate in ES.

Thanks a lot,
Neil

PS, I don't suppose Mozilla shares their generated .sym files for the mac/windows system libraries do they? Would save me a chunk of time :)

[Peter Bengtsson]

Hi Neil,
If you're using the Socorro webapp, that has a mechanism for reprocessing.
It can be reached without the webapp too. Since it's just a Python class
[0].

Regarding generated .sym files, all debug symbols that we use are freely
available just like the Firefox source code. But it's only for Firefox so
I'm not sure it helps you.
E.g.
https://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym

[0]
https://github.com/mozilla/socorro/blob/master/socorro/external/rabbitmq/crashstorage.py#L304

> _______________________________________________
> tools-socorro mailing list
> tools-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-socorro
>



--
Peter Bengtsson
Mozilla Engineering Operations

[Ted Mielczarek]

On Fri, Apr 7, 2017, at 08:23 AM, Peter Bengtsson wrote:
> Regarding generated .sym files, all debug symbols that we use are freely
> available just like the Firefox source code. But it's only for Firefox so
> I'm not sure it helps you.
> E.g.
> https://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym

FWIW, those symbols are exactly what you'd get if you fetched the PDB
files from Microsoft's symbol server and ran dump_syms on them yourself.

-Ted

[Neil Williams]

Hello,

Thanks you both for your replies (I was sure I replied last week, must have forgotten to actually send it).

> If you're using the Socorro webapp, that has a mechanism for reprocessing.
> It can be reached without the webapp too. Since it's just a Python class

We have stopped short of using the webapp as the socorro doc does warn away from it, I will have a look at the class you provided, thanks!.


Regarding the Symbols; that is exactly what I was looking for. For Windows I was going to pull down the symbol packs from Mircosoft and create some scripts to run through the pdbs and create the sym files, then upload them to our store. However if Moz have already generated a .sym file for a certain dll version, then it saves us a lot of trouble in hunting down the exact pdb file and converting it ourselves, so this is really appreciated.

Unfortunately Apple don't provide the same sort of thing so it is going to be a little more difficult on there. The system frameworks and the objective-c, c, c++ runtime libs are probably going to provide the most annoyance.

I predict that for most of the crashes symbols from our own binaries and libraries we ship will give us enough info to fix. For other I may need to find extra symbols and re-run the analysis.

[Ted Mielczarek]

On Thu, Apr 20, 2017, at 07:22 AM, Neil Williams wrote:
> Regarding the Symbols; that is exactly what I was looking for. For
> Windows I was going to pull down the symbol packs from Mircosoft and
> create some scripts to run through the pdbs and create the sym files,
> then upload them to our store. However if Moz have already generated a
> .sym file for a certain dll version, then it saves us a lot of trouble in
> hunting down the exact pdb file and converting it ourselves, so this is
> really appreciated.

It is indeed a bit of a hassle. As long as you cache things on your side
it should be no problem to fetch symbols from our symbol server.

> Unfortunately Apple don't provide the same sort of thing so it is going
> to be a little more difficult on there. The system frameworks and the
> objective-c, c, c++ runtime libs are probably going to provide the most
> annoyance.

We have some scripts to scrape symbols out of Apple's system update
packages:
https://github.com/luser/breakpad-mac-update-symbols

I don't have them set up to run on a regular basis currently, but we
have run them several times in the past and acquired usable symbols.

-Ted

[neil.w...@livedrive.com]

Hello Peter,

So I tried having a go at using ReprocessingOneRabbitMQCrashStore to reprocess a crash dump, although I am failing as it appears that I need to pass a config to the class instance I create and I am not sure how I create a config for my current running socorro setup. (This is probably not helped by me not knowing python).

I had a look at some of the *_app.py files, would creating a reprocessor app
in the same way as fetch_app.py et al be the way forward (I would like to avoid editing existing socorro source if possible).

Any help would be appreciated.

Thanks,
Neil