Map Reduce Question

[Valeriob]

Given the following model and indexes, the store contains 6M Parents with about 2 Children for each one.

Map_Recuce_Parent is working quite fast for the amount of data stored, while the Map_Reduce_Children is taking days and i do not know when it will finish :D

The SelectMany inside the map reduce is the real killer i guess, my question is : how mapping works ? what is raven actually doing that way?

Thanks

Valerio

 
    public class Parent
    {
        public List<Child> Children { get; set; }
        public string Attribute_P { get; set; }
    }
 
    public class Child
    {
        public string Attribute_C { get; set; }
        public double Amount { get; set; }
    }

    
    public class Result
    {
        public string Attribute { get; set; }
        public double Amount { get; set; }
        public int Count { get; set; }
    }

    public class Map_Reduce_Children : AbstractIndexCreationTask<Parent, Result>
    {
        public Map_Reduce_Children()
        {
            Map = docs => from parent in docs.SelectMany(a => a.Children)
                           select new { Attribute = parent.Attribute_C, Count = 1 };
 
            Reduce = results => from result in results
                                group result by new { result.Attribute }
                                into g
                                select new { g.Key.Attribute, Amount = g.Sum(r => r.Amount), Count = g.Sum(r => r.Count) };
        }
    }
 
    public class Map_Reduce_Parent : AbstractIndexCreationTask<Parent, Result>
    {
        public Map_Reduce_Parent()
        {
            Map = docs => from parent in docs
                          select new { Attribute = parent.Attribute_P, Count = 1 };
 
            Reduce = results => from result in results
                                group result by new { result.Attribute }
                                into g
                                select new { g.Key.Attribute, Amount = g.Sum(r => r.Amount), Count = g.Sum(r => r.Count) };
        }
    }

[Matt Warren]

What build number are you using, are you using an older build? 

In later builds there was a lot of speed improvements for indexing.

Also, you are now forced to have the output of your Map statement match the output of your Reduce, so it should now look like this:

    Map = docs => from parent in

 docs
                 select new { Attribute = parent.Attribute_P, Count = 1, Amount = 0 };
 
    Reduce = results => from result in results
                  group result by new { result.Attribute }
                  into g
                  select new { g.Key.Attribute, Amount = g.Sum(r => r.Amount), Count = g.Sum(r => r.Count) };

[Matt Warren]

Given the following model and indexes, the store contains 6M Parents with about 2 Children for each one.

Map_Recuce_Parent is working quite fast for the amount of data stored, while the Map_Reduce_Children is taking days and i do not know when it will finish :D

The SelectMany inside the map reduce is the real killer i guess, my question is : how mapping works ? what is raven actually doing that way?

RavenDB runs the Map statement over all the docs in the data store and then write the intermediate results back into the data store.Then it takes batches of results and runs the Reduce statement over them.

So the SelectMany will make a difference, but it shouldn't be taking days compared to hours without it. If you look at the "\stats" http endpoint what does it tell you, are there any errors, how many docs has it indexed?

[Valeriob]

I'm on build 888, and yes the mapping got lost translating from my domain to the generic model, 

but that is not the problem :D 

The error log is empty, this is a piece of log that i considered significant, does it ring any bells ?

http://img19.imageshack.us/img19/3496/reduce.png

Consider Articolo = Attribute in the prev example.

It takes 4 minutes between two "Found 512 mapped results for keys", 

It uses up to 1.5g memory (maybe that's good) 

On every shutdown it has to fix lucene indexes "Unclean shutdown detected on Peso/Per/Articolo, checking the index for errors. This may take a while."

I hope this info helps !

Thanks

Valerio

[Matt Warren]

How are you shutting down your app, is seems strange that is doesn't shut down cleanly every time and so has to be fixed.

You might be running into the problem in this thread, but from what I remember that was the opposite scenario, i.e. the indexes not being checked when they should be.

In the Lucene index directory, is there a file called "write.lock" after a shutdown?

[Valeriob]

Nope there is not, but i get this on restart: http://img404.imageshack.us/img404/7848/startupi.png

i dont think this is related to the poor performances.

Thanks

Valerio

[Itamar Syn-Hershko]

So how _do_ you shut it down?

[Valeriob]

i just press q.

Valerio

[Oren Eini (Ayende Rahien)]

Hm,

That should do an orderly shutdown.

Is this reproducable to you?

[Valeriob]

Hi Oren, 

i'll try to build a test solution to replicate the behavior and let you know ! 

Valerio

[Oren Eini (Ayende Rahien)]

Thanks