Question: how to track cause of build failure

[trasko]

TL;DR:

My goal is for each failed build (for builds I care about), I would like to know the cause of the build failure.  The possibilities I care about right now are "infrastructure", "build system" or "developer".   How would you recommend doing this?

More:

I am fine with this process being manual (at least to start) as long as it can be modified programmatically somehow going forward.  This would open the door to a "regular expression" style auto-classification (like the "Build Failure Analyzer Plugin" (1)) or a learning algorithm.

I am struggling to find a way to assign a simple variable to a build and to be able to assign it AFTER the build has completed

What I've tried:

1. Metadata plugin and manually set a String variable after the build is done.  This seems to not be allowed -- it can only be set at build time?
2. "Build Failure Analyzer Plugin" (https://wiki.jenkins-ci.org/display/JENKINS/Build+Failure+Analyzer)  This works well for setting the build failure based upon a regular expression but I'm finding it difficult to use for my use-case for a few reasons:
1. Still no solution to report on the results
2. Sometimes the same error (eg Permission Denied: file XYZ.txt) will be caused by a developer and sometimes by the build system misbehaving.  I would prefer manual control in these cases.
3. When we add a new "Build Failure Cause" regex, you need to rescan builds in order to properly assign these.  There is a concern (voiced on the plugin page) of duplicate entries and also I'm not sure how to trigger a re-scan on all builders (we have 100+). 
3. BFA Plugin + mongodb + graphite  This seems like it could work to solve the reporting issues(#2-1), but not the other ones (#2-2, #2-3).  

What I'm considering:

Scrape the logs from Jenkins into a custom web application and providing a simple interface to manually set the build failure cause.  I could -- as a separate component -- implement the same "regex-based" classification that the BFA plugin does.  I really don't want to maintain more infrastructure and code just to get a rather simple data point.

Does anyone have any pointers?

Is there a way to keep metadata about a build and update/query it after a build has completed?

Thanks!