Just an FYI:
I made some good progress the last two weeks adding a new MySQL table
("stats") to store aggregate stats. There's a separate row for each
combination of label (eg, "Oct 1 2011"), slice ("All", "Top 100" , "Top
1000", "intersection") and device (right now just "IE8" and "iphone").
This makes it very fast to generate charts, and also adds more
flexibility as we change the charting code. (Previously the data and the
chart GUI were cached together.)
I just wrote a high-level blog post about the changes:
http://www.stevesouders.com/blog/2011/10/20/http-archive-new-code-new...
I've incorporated ~10 patches from contributors over the last month -
thanks! I'm excited about the UI improvements to the waterfall charts.
As I mention in the blog post, there are a few more big tasks I have to
tackle before we start increasing the # of URLs we crawl. I'm shooting
to get those done before the Nov 1 run, but it might slip until Nov 15.
Feel free to ask questions. Thanks.
-Steve