Slow mgo performance

1,318 views
Skip to first unread message

mbil...@gmail.com

unread,
Mar 28, 2013, 5:55:09 PM3/28/13
to mgo-...@googlegroups.com
Hi, new to go, been working with Python/mongo for a long time...

So, just testing the go/mgo out with a simple program and comparing to python performance, this runs in ~8 seconds in Python, but takes almost a minute in go:

http://pastebin.com/bbX1uLTJ

http://pastebin.com/KQSBez19

There's a pprof trace at the end of the go version - nothing sticks out to me really.  Am I doing the unmarshalling bit the right way?

Perhaps I'm doing something wrong in here?  mongod doesn't seem to be sweating running either program.

m

Gustavo Niemeyer

unread,
Mar 28, 2013, 6:55:32 PM3/28/13
to mgo-...@googlegroups.com
Try to run it after:

export GOMAXPROCS=4

This should drop that timing a lot. Some of the design decisions on
mgo were made with scalability and concurrency in mind. It performs
well if you allow it to use more threads, and it should continue to
perform well under quite a bit of abuse when you do more stuff
concurrently with whatever else is being done. [1]

On Thu, Mar 28, 2013 at 6:55 PM, <mbil...@gmail.com> wrote:
> Perhaps I'm doing something wrong in here? mongod doesn't seem to be
> sweating running either program.

If you can send an example that is self-contained, including the data
creation bits, I can run it here as well to compare. Otherwise, I have
no idea about how you're exercising that code, which makes it harder
to help.

[1] Out of coincidence, this was posted today:

https://github.com/rossan/kuenea

Look at the timings there.


gustavo @ http://niemeyer.net

mbil...@gmail.com

unread,
Mar 28, 2013, 7:25:10 PM3/28/13
to mgo-...@googlegroups.com
Thanks, that helps:

mattb@capdev2:~/src/golang $ time GOMAXPROCS=4 ./mongo
821226
426964

real    0m21.812s
user    0m26.662s
sys     0m1.284s

And a testcase if you want to look:

http://vazor.com/drop/mgo_testcase.tar.gz (35MB)

A mongodump of that collection and the python/go src.

thx

m

Gustavo Niemeyer

unread,
Mar 28, 2013, 7:59:40 PM3/28/13
to mgo-...@googlegroups.com

Thanks.  I'll find some time over the next week to have a look.

Meanwhile, here is a fun idea to experiment with: put a second iteration to run in parallel with a Copy of the first session, and time how long they both take together.

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Gustavo Niemeyer

unread,
Mar 28, 2013, 8:52:00 PM3/28/13
to mgo-...@googlegroups.com, mbil...@gmail.com
On Thu, Mar 28, 2013 at 8:59 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Thanks. I'll find some time over the next week to have a look.

Okay.. I couldn't resist looking. :-)

Some significant amount of time is being spent on the map logic. Use
this result type instead:

var res struct{ EventType string `bson:"eventType"` }

This plus the GOMAXPROCS already beats the Python timing here.


gustavo @ http://niemeyer.net

mbil...@gmail.com

unread,
Mar 28, 2013, 9:07:24 PM3/28/13
to mgo-...@googlegroups.com, mbil...@gmail.com
Cool - I suspected I was doing that bit in a naive way -- thanks for the pointer.

m
Reply all
Reply to author
Forward
0 new messages