Rethinking elmo, buildbot, elasticsearch

[Axel Hecht]

Hi,

sounds scary, kinda is, but here it goes:

We've been making OK progress getting buildbot logs onto ES for elmo,
still a good deal of work to do though.

News that happened in the meantime:

- buildbot 0.7.12 doesn't run on new macs anymore
- we may switch to git

Which makes me think the following:

It might prove useful to decouple the data storage of compare-locales
from buildbot logs, and loose the coupling to hg.

So here's the alternative proposal:

Store the buildbot logs only for a day or so, in case some failures
require investigation.
Store the json inside of those logs itself into ES, and loosen the
coupling between the elmo data model and buildbot (notably, add a ES_ID
column to l10nstats.models.Run)

Less data, more real data, less bounds to hg and buildbot. Sounds like a
clear win.

The big loss is of course that we need to restart the thinking about
what the data looks like we put into ES.

The native json is a hierarchical tree structure. The nodes are tuples
of ['path', object], and the objects have a either a 'children' or a
'value' entry, children being a list of nodes again. Values are listing
missingEntities, errors, obsolete, etc.

That's pretty minimal, but ES doesn't like it naively.

I can reformat that to one list of objects, with full path and values,
which seems to be absorbed by ES just fine.

Both would have similar metadata, a summary of missing, obsolete, error
and warning counts, and a revisions list, [{"name": reponame,
"revision": hash},..]

Any comments? I'll paste example data below, the full original data set
has an example on http://www.pastebin.mozilla.org/2453920

I'm interested in efficiency, overall idea, caveats, etc.

Axel

PS: Orig data sample

{
"details": {
"children": [
[
"ku",
{
"children": [
[
"browser",
{
"children": [
[
"README.txt",
{
"value": {
"obsoleteFile": "error"
}
}
],
[
"chrome",
{
"children": [
[
"branding",
{
"children": [
[
"brand.dtd",
{
"value": {
"obsoleteFile": "error"
}
}
],
[
"brand.properties",
{
"value": {
"obsoleteFile": "error"
}
}
]
]
}
],
[
"browser/aboutDialog.dtd",
{
"value": {
"missingEntity": [
"channel.description.end",
"channel.description.start",
"community.end3",
"community.exp.creditsLink",
"community.exp.end",
"community.exp.middle",
"community.exp.mozillaLink",
"community.exp.start",
"update.applying",
"warningDesc.telemetryDesc",
"warningDesc.version"
],
"obsoleteEntity": [
"community.end2"
]
}
}
]
....

vs
{
entries: [

{
path: ku/browser/README.txt
value: {
obsoleteFile: error
}
}
{
path: ku/browser/chrome/branding/brand.dtd
value: {
obsoleteFile: error
}
}
{
path: ku/browser/chrome/branding/brand.properties
value: {
obsoleteFile: error
}
}
{
path: ku/browser/chrome/browser/aboutDialog.dtd
value: {
missingEntity: [
channel.description.end
channel.description.start
community.end3
community.exp.creditsLink
community.exp.end
community.exp.middle
community.exp.mozillaLink
community.exp.start
update.applying
warningDesc.telemetryDesc
warningDesc.version
]
obsoleteEntity: [
community.end2
]
}
}