optical flow very slow on samsung galaxy s

[yostane]

Hi,
I wrote a function for the processos class (in cvcamera example).
This function runs very slow on samsung galaxy s. Is there a way to
optimize it?

Mat currentGrey = pool->getGrey(input_idx);
Mat img = pool->getImage(input_idx);
static Mat prevGrey = pool->getGrey(input_idx);

if (/*prevGrey.empty() ||*/ currentGrey.empty() || img.empty())
return; //no image at input_idx!

keypoints.clear();

FeatureDetector* fd = &fastd;
fd->detect(prevGrey, keypoints);

vector<Point2f> prevPts, nextPts;
vector<uchar> status;
vector<float> err;

for (int i=0; i<keypoints.size(); i++){
prevPts.push_back(keypoints[i].pt);
}

Size winSize=Size(10, 10);
cv::calcOpticalFlowPyrLK(prevGrey, currentGrey, prevPts, nextPts,
status, err
, winSize, 2);

for (int i=0; i<nextPts.size(); i++)
{
if (status[i] == 1){
double m = norm(prevPts[i]-nextPts[i]);
if (m > 1){
line(img, prevPts[i], nextPts[i], CV_RGB(255, 0, 0));
}
}
}
currentGrey.copyTo(prevGrey);

Thnaks

[Ethan Rublee]

Depends on the number of points mostly. I get near frame rate if i down
sample the image and limit the number of points to say 50.

[yostane]

Thanks for the answer :)
It now runs more quickly but 50 points is very low...

[yostane]

Here is the code

void Processor::detectAndDrawOpticalFlow(int input_idx, image_pool*
pool)
{
const double factor = 2;

Mat img = pool->getImage(input_idx);

Mat currentGrey;
resize(pool->getGrey(input_idx), currentGrey, Size(), 1./factor, 1./
factor);
static Mat prevGrey(currentGrey);

if (prevGrey.empty() || currentGrey.empty() || img.empty())

return; //no image at input_idx!

keypoints.clear();

FeatureDetector* fd = &fastd;

//fd->detect(currentGrey, keypoints);

fd->detect(prevGrey, keypoints);

vector<Point2f> prevPts, nextPts;
vector<uchar> status;
vector<float> err;

for (int i=0; i<50; i++){
prevPts.push_back(keypoints[i].pt);

}

cv::calcOpticalFlowPyrLK(prevGrey, currentGrey, prevPts, nextPts,

status, err);

for (int i=0; i<50; i++)

{
if (status[i] == 1){
double m = norm(prevPts[i]-nextPts[i]);

if (m>0.1 && m<20){
line(img, prevPts[i]*factor, nextPts[i]*factor, CV_RGB(255,
255, 255), 3);
}
}
}
currentGrey.copyTo(prevGrey);

[Ran]

did you try horn-schunk?
I believe it's faster.

[baldzar]

Performance mostly depends on number of points, next thing is window
size, pyramid levels, termination criteria etc.
But the biggest problem is that opencv uses float and doubles for most
algorithms and most devices does not have hardware fpu.
As i was experimenting with performance of soft floating numbers vs
fixed point numbers i noticed that fixed point numbers 20x faster than
soft floating numbers.
So I would suggest you rewriting the algorithms you need using fixed
point numbers.

[yostane]

You mean that we should rewrite opencv functions?

[Zoran Angelov]

Yes, actualy it is copy paste from opencv source code and replacing floats and doubles with fixed point number class. You will need tto identify range of possible values in the algorithm and configure the fixed point number class to fith the required range. Afcourse you will sucrifice precision but mobile devices have smaller screens so you do not high precision. Somethimes low precision will result in fewer feature points so you will have to track fewer points.
For example i have rewritten surf feature detector using fixed point numbers, and it aproximately needed 5 microseconds per pixel compared to 47 microseconds of opencv's floating point version.

On 11 Mar 2011 17:28, "yostane" <yassine....@gmail.com> wrote:

You mean that we should rewrite opencv functions?

On 11 mar, 17:12, baldzar <bald...@gmail.com> wrote:

> Performance mostly depends on number of poin...

[bustih]

Hi,

interesting discussion because I'm also trying to evaluate computer
vision algorithms (using OpenCV) on Android. Using fixed point
arithmetics may be one possible idea for increasing performance.
But as far as I know current Android devices do have a FPU (in
hardware) don't they? I'm not sure how these devices perform comparing
fixed vs floating point numbers.
@Zoran: On which devices have you tested your fixed vs. floating point
experiments?

On 11 Mrz., 18:31, Zoran Angelov <bald...@gmail.com> wrote:
> Yes, actualy it is copy paste from opencv source code and replacing floats
> and doubles with fixed point number class. You will need tto identify range
> of possible values in the algorithm and configure the fixed point number
> class to fith the required range. Afcourse you will sucrifice precision but
> mobile devices have smaller screens so you do not high precision. Somethimes
> low precision will result in fewer feature points so you will have to track
> fewer points.
> For example i have rewritten surf feature detector using fixed point
> numbers, and it aproximately needed 5 microseconds per pixel compared to 47
> microseconds of opencv's floating point version.
>

[Andreas]

Dear Zoran,

I am very interested in what you just explained.

Is it possible to show one line of code of any opencv function in
floating point and how you converted it to fixed point just to give us
an idea?

Thanks in advance,

Andreas

[Zoran Angelov]

Hi,
at first i apologize for delayed reply.

@bustih: I'm testing on zte racer. It is qualcomm msm7227 based device
OS: android 2.1 update1, and it appears that it has hardware fpu.
But my tests and several android benchmark tools shows that integer
unit is much faster than fpu unit (NBench reported: integer
index=5.070 and fpu index=0.384).
The same tests that i performed on desktop PC showed that fpu is much
faster than integer unit. Maybe the reason is that android uses
soft_fpu calling convention (passing values through integer registers)
for floating numbers,
or chip vendors are limited when optimizing fpu unit or must take care
of power consumption.

@Andreas: I've uploaded my code at http://code.google.com/p/lib-ucv/ .
There is surf class that is actually rewritten opensurf source code
(not from OpenCV but the principle is the same).
There you can find configurable fixed_point class template that is
based on fpmath from Peter Schregle : http://code.google.com/p/fpmath/
with some modifications.
There is also a gpu ( opengl es 2.0 ) version of some parts of SURF
algorithm ( SURF hessian detector ) that i was experimenting with, it
performed nice ( again faster than OpenCV's ) but requires OpenGL ES
2.0 hardware ( vertex and fragment shaders, and FBO feature).

[Andreas]

Dear Zoran,

Thank you for your reply!

One last thing, did you use any profiler in order to test the
performance of the application?
I am trying to use ARM RealView Profiler.. Do you know any other tools
that I can use to track performance on native code?


Thanks in advance,

Andreas

[bustih]

Hi,

FYI: NBench results on HTC Desire:
Integer: 4.015
Float: 0.912

So I suppose you can expect speedups of "only" 2-3 times (on newer
devices maybe less).


Best regards,
bustih