"Not using" MongoKit is fast!
454 views
Skip to first unread message
Peter Bengtsson
Oct 21, 2010, 6:38:03 AM10/21/10
to MongoKit
I've got an app that has an HTTP API. My app is taking some parameters
from the request, fetches the objects from MongoDB, massages some
values and pushes the data out as JSON.
Did some benchmarks and found that I was only getting about 40
requests per second. Did some light-weight profiling and found that
the time was spent on getting data out of the MongoDB on lines like
this:
for thing in self.db.events.Event.find(search):
do_something(thing.title)
...
So I rewrote it with some slight changes and skipped the object
mapping so it's only like this:
for thing in self.db.events.find(search):
do_something(thing['title'])
...
In other words "skip" MongoKit and go straight to the source. Then I
re-ran the benchmark and going from 40 requests/second I now do 300+
requests/second
This is one of the best things about MongoKit; the ability to go
straight to the source (pymongo) when and if you need to.
from the request, fetches the objects from MongoDB, massages some
values and pushes the data out as JSON.
Did some benchmarks and found that I was only getting about 40
requests per second. Did some light-weight profiling and found that
the time was spent on getting data out of the MongoDB on lines like
this:
for thing in self.db.events.Event.find(search):
do_something(thing.title)
...
So I rewrote it with some slight changes and skipped the object
mapping so it's only like this:
for thing in self.db.events.find(search):
do_something(thing['title'])
...
In other words "skip" MongoKit and go straight to the source. Then I
re-ran the benchmark and going from 40 requests/second I now do 300+
requests/second
This is one of the best things about MongoKit; the ability to go
straight to the source (pymongo) when and if you need to.
Matt Billenstein
Oct 21, 2010, 7:14:25 AM10/21/10
to mong...@googlegroups.com
Did you go any further to see what the root slow-down was? Are your
documents big or deeply nested or perhaps you're mapping a large number
of them per request?
documents big or deeply nested or perhaps you're mapping a large number
of them per request?
Might even be fruitful to compare a profile dump of before and after
this change to see where the time is being spent. This is nearly an
order of magnitude difference in performance.
Matt
--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/
Peter Bengtsson
Oct 21, 2010, 7:30:30 AM10/21/10
to MongoKit
NOTE! The important thing in the benchmark is the difference. Not the
number.
For the benchmark I ran after the optimization, if I instead use 10
concurrent request the number goes from 300+ to 1300+.
number.
For the benchmark I ran after the optimization, if I instead use 10
concurrent request the number goes from 300+ to 1300+.
Nicolas Clairon
Oct 22, 2010, 7:54:39 AM10/22/10
to mong...@googlegroups.com
I'm glad you enjoying the way how MongoKit is designed. I'm still
working hard to make it faster thought.
working hard to make it faster thought.
sebest
Nov 6, 2010, 6:00:15 PM11/6/10
to MongoKit
Hello,
It would also be interesting to see the numbers of q/s of mongokit
with skip_validation=True
It would also be interesting to see the numbers of q/s of mongokit
with skip_validation=True
Nicolas Clairon
Nov 6, 2010, 8:55:05 PM11/6/10
to mong...@googlegroups.com
skip_validation set to True is faster than False but it will be slower
than a vanilia pymongo query/insert as the skeleton of the document
should be built.
than a vanilia pymongo query/insert as the skeleton of the document
should be built.
Latest versions of MongoKit are twice faster in average. My goal
before reaching v0.6 is to continue the optimization and build
benchmarks (in addition with unittests).
Finally I'll try to not break compatibility in stored data -- unlike I
said in a previous message "Changes planed in v0.6".
http://groups.google.com/group/mongokit/browse_thread/thread/193dd8c95e2042fc
N.
Reply all
Reply to author
Forward
0 new messages