Regarding OpenCV SURF versions and benchmarks
Juan Navarro
I'm a computer engineer, in the need of finding the fastest implementation of the SURF algorithm for a personal project. Starting with the one in OpenCV, later I found about about OpenSURF and the one in BoofCV.
As I need speed for real time processing, I find the benchmarks in your wiki page really interesting:
http://boofcv.org/index.php?title=Performance:SURF
and also the comparison between OpenCV and BoofCV
http://boofcv.org/index.php?title=Performance:OpenCV:BoofCV
But I'd like to warn about the OpenCV's SURF implementation, and how it could be very important for benchmarking appropriately.
As noted in some last year's benchmarks done by the Computer Vision Talks blog:
http://computer-vision-talks.com/2011/07/comparison-of-the-opencvs-feature-detection-algorithms-ii/
it seems that the single-thread implementation of SURF might get 'deprecated' soon, in the sense of not receiving more optimizations, as the algorithm internals were restructured in the last OpenCV 2.3.1 version, in order to integrate the Intel Threading Building Blocks. These TBB libraries are meant for parallel computing, and it seems (yet to confirm) that the integration came with the cost of downgrading the SURF's algorithm performance when OpenCV is NOT compiled with TBB support. It is a fact that the (single-thread) SURF implementation in v2.2 was faster than the current (single-thread) one in v2.3.1.
In the previous link I gave, you can read comments about this performance change, and comments from "emilie" and "EKhvedchenya" on the March 6th, 2012, seem to clarify a bit about this.
Still I have to look in the OpenCV forums to check if there is any conversation about this topic, to confirm all this information. But, if it happens to be true, then OpenCV's SURF algorithm may not be properly benchmarked if not using a TBB-enabled (multiple threads) compilation.
Peter Abeles
Thanks for the comments. I'll update the comment about TBB on the
benchmark webpage to highlight this issue. I remember seeing some
message that OpenCV was going to have another major release soon. After
that release I'll check to see what has been updated.
This could present a bit of a problem as far as the benchmark goes if
they deprecate the non TBB version, but don't provide a way to force it
to use a single thread. There are good reasons to not want a single
application to suddenly hog all the available resources. A
multi-threaded benchmark has been under consideration.
The OpenCV code is based off example code on their website and uses the
C++ interface. The benchmark is designed replicate the performance that
a non-expert, who is not intimately familiar with a library's internal
working, would get. So I don't think it would be appropriate to jump
through hoops beyond what their own instructions provide.
When going through OpenCV's code to try to understand what they were
doing, I was under the impression that the TBB/non TBB code shared a
fair amount in common. "emilie" might have been invoking the
multi-threaded version on a multicore/processor computer using the C++
interface and single threaded version using the c interface. So I'm not
entirely sure that a single threaded TBB code would be any faster than
single threaded non TBB. The SURF code in OpenCV is a bit of a mess and
I could be mistaken.
- Peter