Authoritative Data Store for Chromium (hackability-cy@chromium.org)

[Emil Eklund (via Google Docs)]

I've shared an item with you.

Authoritative Data Store for Chromium

Google Docs: Create and edit documents online.

[Ojan Vafai]

I'm not sure from the doc what sorts of query use-cases you're trying to support. Are you saying that the flakiness dashboard and sheriff-o-matic would talk to this datastore?

I think the flakiness dashboard will likely be the most complicated consumer of this in terms of the demands it places on the datastore (it's both a lot of data and needs to be near real time). We should flush out an end to end design for the flakiness dashboard to make sure it would work. szager and I have been talking the past few weeks about reworking test-results.appspot to work better for the flakiness dashboard use-cases. Just coming up with a dedicated solution for flakiness dashboard that works well has proven difficult. We have a plan in mind now, but it's not general.

Of course, if we could do something general with your datastore that meets the requirements, that'd be much better. It might be that the conclusion we come to is that test-results/flakiness dashboard is a special case and that this datastore should be for all our other storage needs.

--
You received this message because you are subscribed to the Google Groups "Chromium Hackability Code Yellow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hackability-c...@chromium.org.
To post to this group, send email to hackabi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/hackability-cy/047d7b15aeb75b81830502bbb8b5%40google.com.

[Emil A Eklund]

On Wed, Sep 10, 2014 at 10:54 PM, Ojan Vafai <oj...@chromium.org> wrote:

I'm not sure from the doc what sorts of query use-cases you're trying to support. Are you saying that the flakiness dashboard and sheriff-o-matic would talk to this datastore?

Yes, the idea is to replace all data acquisition and storage to ensure that we get a consistent world-view and so that we only have to maintain one parser for each data source.

 

I think the flakiness dashboard will likely be the most complicated consumer of this in terms of the demands it places on the datastore (it's both a lot of data and needs to be near real time). We should flush out an end to end design for the flakiness dashboard to make sure it would work.

The fact that the data is static makes it a lot easier. It only really needs to be real time for the last few hours and then it can fall back on pre-aggregated/pre-parsed data as historic data never changes. Even that might not be required as it is a well known set of queries that can be optimized for.

 

szager and I have been talking the past few weeks about reworking test-results.appspot to work better for the flakiness dashboard use-cases. Just coming up with a dedicated solution for flakiness dashboard that works well has proven difficult. We have a plan in mind now, but it's not general.

Of course, if we could do something general with your datastore that meets the requirements, that'd be much better. It might be that the conclusion we come to is that test-results/flakiness dashboard is a special case and that this datastore should be for all our other storage needs.

I picked the test results data source and flakiness dashboard data consumer as v1 customers for precisely this reason. If we can get it to work for some of the hardest use cases that'll validate the design. If it doesn't work on the other hand we might have to rethink the entire effort.

[Julie Parent]

A few people have been asking about timeline for this project.  I'll let Emil flesh out the details, but my understanding is that he has data from the past 30 days in the record form he wants to work with, and will be running performance tests on a few storage solutions (BigQuery, CloudSQL, AppEngine ndb?) early next week, to determine feasibility so we can proceed.

--

You received this message because you are subscribed to the Google Groups "Chromium Hackability Code Yellow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hackability-c...@chromium.org.
To post to this group, send email to hackabi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/hackability-cy/CADu_oUDmeHDZhRHF%3DKtfBPMypoanfnycVww0bnqKfkjLmCpRXQ%40mail.gmail.com.