How to store and update huge collection of simple XYZ point structures?

[Matas Girdauskas]

Hi,

We need to regularly log some experiment measurement points somewhere. We are trying to use RavenDB for this, however, we are not sure about such application design. 

In relational database we would have two tables:

About 10 points are generated every second. One experiment can easily end up having 100,000 points. Raven should be updated every minute or faster (instant updates would be ideal).

We tried saving everything as a single document per experiment:

class Experiment {

    string Id;

    string Name;

    string TimeStarted;

    string TimeFInished;

    List<XyzPoint> Points; 

}

However, document with 100,000 points takes about 10MB and every time we add or remove points - raven has to send entire object. This seems not right for me.

My question are: 

1. Is it normal to have huge documents like in our example or is it better to save every point as a separate document? Millions of documents?! 
   
2. Should we consider using RavenFS to store experiment points?
3. Any other recommendations for this use case?

Kind regards,

Matas

[Anders Strömberg]

Is there a requirement that you need to do this update? Can't you wait until you gathered all your measurement points and then make a single commit?

BR

Anders

[Michael Yarichuk]

1) There are some use-cases where large documents make sense, and we support large documents. However, relatively large documents has performance overhead that needs to be taken into account. (large documents > 500kb)

2) RavenFS is basically a store for large blobs of binary data.

3) Hard to give concrete recommendations without knowing more about your problem domain and requirements

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,

 

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

 

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Chris Marisic]

On Saturday, May 30, 2015 at 4:10:24 AM UTC-5, Matas Girdauskas wrote:

About 10 points are generated every second. One experiment can easily end up having 100,000 points. Raven should be updated every minute or faster (instant updates would be ideal).

However, document with 100,000 points takes about 10MB and every time we add or remove points - raven has to send entire object. This seems not right for me.

This is just not a candidate for any database. Sure you CAN stuff that in nearly any database but it doesn't make it a good choice to do so.

You want to store aggregated data in a database.

If you need every single point, you should really look at storage as binary or into cloud storage such as azure data lake  unfortunately that doesn't help this second.

100,000 points in a single experiment just really has no relevance to a database. Why would you ever update a point? Why would you ever want to get a single point that is not a max/min or other significant value? Why would you want to query 7 points?

[Matas Girdauskas]

Well, these points are latter used to show various graphs. Experiment can take few days or months and latter user might want to see data of specific minute / hour / day.

[Matas Girdauskas]

Some experiments are continuous and never ending. So yes, we need to regularly add new points.

[Oren Eini (Ayende Rahien)]

See the auction model for this.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--