driver benchmarks?

[Liam Staskawicz]

Hiya - I recall there used to be a page in the wiki with some benchmarks for the various drivers available, but I can't find it anymore. I was hoping to check there to see if my driver is more or less where it should be in terms of performance. Is this info available elsewhere?

Thanks,
Liam

[Kyle Banker]

We removed those benchmarks for various reasons. Have you implemented our standard benchmark suite? The suite can be found in the codebase of any of our supported drivers and can be run on your own machine, which will give you the most accurate comparison.

--

You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To post to this group, send email to mongo...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-dev?hl=en.

[Mathias Stearn]

The most performance sensitive portion of the drivers is BSON serialization and deserialization. You might want to focus on that when writing benchmarks.

As a tip, if you are doing a lot of array work, stringifying ints can be an expensive operation in many languages, especially if they use the sprintf C function. You may want to consider using an optimized version or caching the lowest and most common numbers as we do in the C and C++ drivers.

[Geir Magnusson Jr.]

why remove them?

geir

[Liam Staskawicz]

I'd also be curious to know - if there's a standard benchmark suite, why not post the numbers?

It's also not super clear how to run the benchmarks - in the ruby driver, for example, there are errors in examples/benchmarks.rb (coll.clear instead of coll.drop). Does this mean I'm looking at the wrong benchmark? Or just hasn't been touched in a little while?

Also, in the java driver, there are quite a lot of things commented out in PerformanceTest.java, and the suite is not the same as benchmarks.rb. Maybe the standard suite is elsewhere in the package?

Thanks,
Liam

[Eliot Horowitz]

There wasn't really a good "standard suite" was one of the big problems.
Its also unclear what a good suite is, so the old one was not updated
and misleading, so we took it down rather than keeping the mess.
At some point we should come up with a good set up thing to test (bson
serialization, message serialization, etc...) anyone? :)

[Kyle Banker]

Liam,

Can you make sure you're using the latest Ruby driver code?  Collection#clear is part of a deprecated api, so I'm guessing you're using an old version.

Kyle

[Liam Staskawicz]

Kyle - it seemed like something that must be a weird version issue, but the gems build as 0.18.2 and I'm pointing at github master.  At the time of this writing, http://github.com/mongodb/mongo-ruby-driver/blob/master/examples/benchmarks.rb contains coll.clear.

Eliot - that sounds understandable :)  One of the documents I found in my searching was a benchmark done for the Factor driver - results at http://gist.github.com/75320.  Maybe that could serve as the basis of an updated suite?

Liam

[Kyle Banker]

Liam,

The latest benchmarks are at bin/standard_benchmark.rb. You're right -- the code from /examples is in need of some updates.

Kyle