Single large document vs broken out smaller documents

[amacd...@technophobia.com]

Hi,  So this might be an interesting question/observation.

I'm in the process of understanding how best to use RavenDB and part of that is deciding on how where to break my object model down into aggregates.

I'm using a production grid filled with employee schedules, where each grid represents a single five day week.

I initially decided to store the entire production grid as a single document, my thought behind this was that only a single windows service will add/update this document (this is done by parsing an xls file).. I had a few issues with this - now resolved thanks to Oren Eini, however during the answering of the initial issue it was suggested that he might break the embedded employee schedules out into their own documents.

With the above in mind, I decided to get above working, make a few notes about folder sizes and time taken to query the more complex index used.  I then modified my code to store the employee schedules as individual files, adjusting indexes in the process.

For clarity of this topic, my index's were as follows:

For the single document:

public class EmployeeScheduleIndex_ByDateAndName: AbstractIndexCreationTask<ProductionGrid>

{

public EmployeeScheduleIndex_ByDateAndName()

{

Map = productionGrids => from productionGrid in productionGrids

from employee in productionGrid.EmployeeSchedules

                        select new

                               {

                               ScheduleDate = employee.Schedule.Min(y=>y.Date),

employee.Name,

employee

                               };

Store("employee", FieldStorage.Yes);

Index("employee", FieldIndexing.No);

}

}

        public class EmployeeIndex : AbstractIndexCreationTask<ProductionGrid, EmployeeIndex.EmployeeNames>

{

public EmployeeIndex()

{

Map = productionGrids => from productionGrid in productionGrids

from employee in productionGrid.EmployeeSchedules

select new EmployeeNames() { Name = employee.Name };

Reduce = results => from result in results

group result by result.Name

into g

select new EmployeeNames() { Name = g.Key };

}

public class EmployeeNames

{

public sring Name { get; set; }

}

}

and for the second multi doc approach:

public class EmployeeScheduleIndex_ByDateAndName: AbstractIndexCreationTask<Employee>

{

public EmployeeScheduleIndex_ByDateAndName()

{

Map = employees => from employee in employees

                        select new

                               {

                               ScheduleDate = employee.Schedule.Min(y=>y.Date),

employee.Name

                               };

}

}

public class EmployeeIndex : AbstractIndexCreationTask< Employee, EmployeeIndex.EmployeeNames >

{

public EmployeeIndex()

{

Map = employees => from employee in employees

select new EmployeeNames(){Name = employee.Name};

Reduce = results => from result in results

                   group result by result.Name

                   into g

                   select new EmployeeNames() {Name = g.Key};

}

public class EmployeeNames

{

public string Name { get; set; }

}

}

The results when comparing the database folder structure was as follows:

Single doc: logs -> 64MB, temp stuff -> 1.1MB, Data -> 1.1MB & index stuff -> 385KB

Multi doc: logs -> 64MB, temp stuff -> 1.1MB, Data -> 17MB & index stuff -> 68KB

As you can see storing the production grid in multiple documents required much, much more data space than a single document while the storage of employee schedules within the search index left a larger index size, but still used far less disk space than the multi doc version.  btw both solutions answered queries virtually within the same time frames ~140ms (using the studio query features)

So now I'm wondering why I might still want to go for the second option that requires much more data space, over the first that has a smaller data footprint, but at the cost of the search index when searching seems to be as quick for both - I'm personally swaying towards the first options at this point.

I should mention that while the production grid might be updated between 10 - 15 times within the week, it is only updated by one user (the windows service) while the rest of the time it will be heavily read for a week before being left to collect dust (I'm likely to stick logic in to remove historic production grids - those older than a month).

Thoughts? arguments?

Thanks in advance too :)

[Kijana Woodard]

If you only have one "user" updating the documents in a single threaded manner and the load/query times are acceptable, I'd probably lean toward one doc as well.

I do a lot of multi-doc designs in situations where I expect either multiple users to contribute data, potentially at the same time, or I expect some aspect of the document to grow at an unknown rate and that data isn't needed all the time. 

So for a FB style app, I would have the users profile separate from the timeline because the profile is relatively fixed in size, but the timeline can grow to different sizes based on the user's activities.

In your situation, it sounds like when you need the doc, you need the whole thing anyway. It sounds similar to the  "RPG With Me" app. It's a large document, but you need the whole thing on load and it's controlled by a single user, so one doc is a "good thing".

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[amacd...@technophobia.com]

Yup...  It's likely that (not built the user interface yet - just seeing how far I can push RavenDB) when user x logs into the web site that shows the production schedule, they will see their own schedule for the week initially, though they will be able to zoom out on team, project etc.

But the first index in my topic covers those scenarios.

[Chris Marisic]

In terms of a social network, you can't reasonably have a timeline-per-user document, what happens when a user posts their 100,000th tweet and your document is 500mb. Social network applications are very special problems, I believe Oren has talked about them occasionally in the forums, also i think facebook has a published slideshow and/or video presentation that discusses modeling social networking data.

[Kijana Woodard]

Oh, I meant the timeline being auctions and bids style not being one big second document. But being able to model the timeline differently from the main doc is one reason to break it out. You could also keep "current timeline" (say last 100 in the main doc and keep history in an auctions and bids. Again, doc db gives you lots of options.