[mongodb-user] Another m/r performance email

[Flaper87]

Hi guys,

I just read this[0] blog post and I was kind of =0 about the time needed to analyze those millions of docs.

I just wanted to know if any of you have an idea about why did it take that much? I mean, is it something you already know and that your fixing/improving or it's the way mongos m/r work and will never change?

how can that be improved??

/me has many questions in his head....

Thanks guys for the hard work...

--
Flavio Percoco Premoli, A.K.A. [Flaper87]
http://www.flaper87.org
Usuario Linux registrado #436538
Geek by nature, Linux by choice, Archer of course.
Key Fingerprint: CFC0 C67D FF73 463B 7E55  CF43 25D1 E75B E2DB 15C7
The Solution to everything:
python -c "from struct import pack; print  pack('5b', (41*len('99')), pow(8,2)+20, 4900**0.5, range(78)[-1], 10)"

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Eliot Horowitz]

1.5.0 should be faster than 1.4 

The big problem is JavaScript though but were looking for alternatives that might be better. Probably for 1.8 not 1.6

[Flaper87]

hehehe, I forgot the link....

http://steveeichert.com/2010/03/31/data-analysis-using-mongodb-map-reduce.html 

2010/4/24 Flaper87 <flap...@gmail.com>

[Flaper87]

Interesting....

is it possible to know what alternatives you already considered (if any)? 

It would be nice to know that and maybe give some opinions if possible.

In any case, thanks a lot to you and all the mongodb team/community.

2010/4/24 Eliot Horowitz <elioth...@gmail.com>

[Eliot Horowitz]

Some areas we're going to be investigating/implementing

1) external executable - any program in any language, would stream
bson in and out for speed
2) hadoop plugin
3) declarative map/reduce
4) faster embedded language (lua for example)

[Ryan Angilly]

I like you guys.  :-D

[reelman]

mongodb/10gen rocks!

[Nathan White]

On Apr 24, 2010, at 8:32 PM, Eliot Horowitz <elioth...@gmail.com>
wrote:

> Some areas we're going to be investigating/implementing
>
> 1) external executable - any program in any language, would stream
> bson in and out for speed
> 2) hadoop plugin
> 3) declarative map/reduce
> 4) faster embedded language (lua for example)
>

Good stuff. I know that mongo utilizes spidermonkey but I saw a note
on the wiki that a v8 is in the works. Do you know when we might see
this? Do you expect any significant performance gains from v8?

Thanks

[Adam Greene]

a hadoop plugin seems very appealing. Hadoop is the m/r standard; a
fast plugin to pull/push info to hadoop and then let us use that
sounds appealing. That way mongo focuses on what it does well and for
those of us who want/need m/r functionality, we add that functionality
on top.

Adam

On Apr 25, 10:39 am, Nathan White <changereal...@gmail.com> wrote:
> On Apr 24, 2010, at 8:32 PM, Eliot Horowitz <eliothorow...@gmail.com>  

> wrote:
>
> > Some areas we're going to be investigating/implementing
>
> > 1) external executable - any program in any language, would stream
> > bson in and out for speed
> > 2) hadoop plugin
> > 3) declarative map/reduce
> > 4) faster embedded language (lua for example)
>
> Good stuff. I know that mongo utilizes spidermonkey but I saw a note  
> on the wiki that a v8 is in the works. Do you know when we might see  
> this? Do you expect any significant performance gains from v8?
>
> Thanks
>
>
>
>
>
> > On Sat, Apr 24, 2010 at 5:53 PM, Flaper87 <flape...@gmail.com> wrote:
> >> Interesting....
> >> is it possible to know what alternatives you already considered (if  
> >> any)?
> >> It would be nice to know that and maybe give some opinions if  
> >> possible.
> >> In any case, thanks a lot to you and all the mongodb team/community.
>

> >> 2010/4/24 Eliot Horowitz <eliothorow...@gmail.com>

>
> >>> 1.5.0 should be faster than 1.4
> >>> The big problem is JavaScript though but were looking for  
> >>> alternatives
> >>> that might be better. Probably for 1.8 not 1.6
>

> For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.

[Eliot Horowitz]

we have v8 mostly working (passes unit tests but leaks and probably
other issues)
its not a very large speed increase, which is why we're not pushing on it.

it turns out most of the time is not in js processing per se, but in
transformations in/out, setting up calls, etc..

If you're curious, you can try out v8 here:
http://downloads.mongodb.org/linux/mongodb-linux-x86_64-v8-latest.tgz

[Flaper87]

2010/4/25 Eliot Horowitz <elioth...@gmail.com>

Some areas we're going to be investigating/implementing

1) external executable - any program in any language, would stream
bson in and out for speed
2) hadoop plugin
3) declarative map/reduce
4) faster embedded language (lua for example)

Awesome, really awesome.

I like them all, but I think that number 2 is really interesting and would be really useful too.

Thanks Eliot and thanks to the team.

[Asger Ottar Alstrup]

Hi,

Here is a paper, which also might be interesting to look into:

http://www.cs.yale.edu/homes/hl293/download/damp05g-mou.pdf

This concept generalizes pure map/reduce, scan and nested scan methods. (Scan methods apply a binary operation to all elements, such as summing. Nested scan does the same for many lists, one by one. Think sum group by.) This approach is efficient, because it will divide the elements into N lists, if there are N cores. Therefore, each core will be able to operate on a good, long list of elements by itself without communication except at the start and end.

Regards,

Asger Ottar Alstrup

--
Asger Ottar Alstrup, SVP - Research & Development
Area9, Nyhavn 63d, 3, DK-1052 København K
Phone: +45 33 11 00 90, cell (DK): +45 26 16 79 03, cell (US): 630 995 9277

[Eliot Horowitz]

Already exists:

http://github.com/mongodb/mongo/blob/master/scripting/engine.h

All you would have to do is implement the ScriptEngine and Scope interface.

On Fri, Jun 11, 2010 at 10:26 AM, Sascha Matzke <sascha...@gmail.com> wrote:
> Hi,

>
> On Sun, Apr 25, 2010 at 02:32, Eliot Horowitz <elioth...@gmail.com>
> wrote:
>>
>> 4) faster embedded language (lua for example)
>

> While you're at it... What about a generic C/C++ Plugin-API to let
> contributors implement a binding to any language they might find useful (if
> it doesn't affect the overall design too much)? This might work for $where
> clauses too.
> Sascha
> --
> Through the darkness of future past
> the magician longs to see
> One chants out between two worlds
> Fire walk with me.
>

[Sascha Matzke]

Hi,

On Sun, Apr 25, 2010 at 02:32, Eliot Horowitz <elioth...@gmail.com> wrote:

4) faster embedded language (lua for example)

While you're at it... What about a generic C/C++ Plugin-API to let contributors implement a binding to any language they might find useful (if it doesn't affect the overall design too much)? This might work for $where clauses too.

Sascha

-- 

[duca]

+1 for lua, specially luajit

On Apr 25, 2:32 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Some areas we're going to be investigating/implementing
>
> 1) external executable - any program in any language, would stream
> bson in and out for speed
> 2) hadoop plugin
> 3) declarative map/reduce
> 4) faster embedded language (lua for example)
>
>
>
>
>
> On Sat, Apr 24, 2010 at 5:53 PM, Flaper87 <flape...@gmail.com> wrote:
> > Interesting....
> > is it possible to know what alternatives you already considered (if any)?
> > It would be nice to know that and maybe give some opinions if possible.
> > In any case, thanks a lot to you and all the mongodb team/community.
>

> > 2010/4/24 Eliot Horowitz <eliothorow...@gmail.com>

>
> >> 1.5.0 should be faster than 1.4
> >> The big problem is JavaScript though but were looking for alternatives
> >> that might be better. Probably for 1.8 not 1.6
>