Slow mgo performance

Showing 1-6 of 6 messages
Slow mgo performance Matt Billenstein 3/28/13 2:55 PM
Hi, new to go, been working with Python/mongo for a long time...

So, just testing the go/mgo out with a simple program and comparing to python performance, this runs in ~8 seconds in Python, but takes almost a minute in go:

http://pastebin.com/bbX1uLTJ

http://pastebin.com/KQSBez19

There's a pprof trace at the end of the go version - nothing sticks out to me really.  Am I doing the unmarshalling bit the right way?

Perhaps I'm doing something wrong in here?  mongod doesn't seem to be sweating running either program.

m
Re: Slow mgo performance Gustavo Niemeyer 3/28/13 3:55 PM
Try to run it after:

    export GOMAXPROCS=4

This should drop that timing a lot. Some of the design decisions on
mgo were made with scalability and concurrency in mind. It performs
well if you allow it to use more threads, and it should continue to
perform well under quite a bit of abuse when you do more stuff
concurrently with whatever else is being done. [1]

On Thu, Mar 28, 2013 at 6:55 PM,  <mbil...@gmail.com> wrote:
> Perhaps I'm doing something wrong in here?  mongod doesn't seem to be
> sweating running either program.

If you can send an example that is self-contained, including the data
creation bits, I can run it here as well to compare. Otherwise, I have
no idea about how you're exercising that code, which makes it harder
to help.

[1] Out of coincidence, this was posted today:

    https://github.com/rossan/kuenea

Look at the timings there.


gustavo @ http://niemeyer.net
Re: Slow mgo performance Matt Billenstein 3/28/13 4:25 PM
Thanks, that helps:

mattb@capdev2:~/src/golang $ time GOMAXPROCS=4 ./mongo
821226
426964

real    0m21.812s
user    0m26.662s
sys     0m1.284s

And a testcase if you want to look:

http://vazor.com/drop/mgo_testcase.tar.gz (35MB)

A mongodump of that collection and the python/go src.

thx

m
Re: Slow mgo performance Gustavo Niemeyer 3/28/13 4:59 PM

Thanks.  I'll find some time over the next week to have a look.

Meanwhile, here is a fun idea to experiment with: put a second iteration to run in parallel with a Copy of the first session, and time how long they both take together.

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Re: Slow mgo performance Gustavo Niemeyer 3/28/13 5:52 PM
On Thu, Mar 28, 2013 at 8:59 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Thanks.  I'll find some time over the next week to have a look.

Okay.. I couldn't resist looking. :-)

Some significant amount of time is being spent on the map logic. Use
this result type instead:

    var res struct{ EventType string `bson:"eventType"` }

This plus the GOMAXPROCS already beats the Python timing here.


gustavo @ http://niemeyer.net
Re: Slow mgo performance Matt Billenstein 3/28/13 6:07 PM
Cool - I suspected I was doing that bit in a naive way -- thanks for the pointer.

m