Grabbing speed using OpenCV2.3.1

[Philip]

Since many members have talked/asked about OpenCV and speed, I was
wandering what would be the FPS if no processing is done.
In order to achieve that, I modified "tutorial-2-opencvcamera" and
commented out this line
bmp = processFrame(mCamera);
so basically, what's left is just the time of mCamera.grab()

Surprisingly, I got ~10FPS (in average even less) even for the
smallest image size (width = 128, height = 96) on my Samsung Galaxy
Nexus !
Do u guys have any idea why?

I'm afraid that if that's the case, no matter how optimized our code
is written - will never achieve better than 10FPS.

[Rui Marques]

That is weird i have about 30fps, even without removing all
processing, on my Motorola Atrix.

[Philip]

Thanks Rui,
Your OS is gingerbread...
Could it be that the "new" camera libraries for ICS (my OS is ICS)
cause the slow-down?

[Ricardo Aires]

Well, actually I am facing the same issue on the GalaxyS running
Gingerbread.

I created an application and the only thing it does is show a 640x480
rgba camera image, and by measuring fps on each execution of the while
cycle, I can only obtain ~15fps. This is not an acceptable performance
for my application wherein I also need to use SURF and OF. The problem
is resolved by decreasing the image res, but I need an image that fits
the entire screen!

This is my app file: http://pastebin.com/WMQkQ1eW

The weirdest thing is that when I run the same code in the GalaxyTab
with the same image resolution it runs slower (8FPS)! This is very
strange, since the tab is a Tegra2 dual-code 1Ghz. Although I suspect,
in this case, there are some memory issues, since it seems the gc gets
the bitmap reference and does a memory dealloc/alloc every frame, this
is a good example on how trickly different android devices work.

I also noticed that there is a Log tag like this:
"I/OpenCV_NativeCamera(19783): ### Camera FPS ### [8580] Frames, 30.06
FPS"

However, even when my application is slowing down, it ALWAYS shows the
same value... around 30FPS. This might actually be the "grabing" speed
and not the retrieving!

I have to agree with Philip, that no matter we optimize code, we will
never achieve better speeds?

Is there anyway to solve these problems?
Thanks!


On Jan 26, 6:57 pm, Philip <sa...@matconsult.com> wrote:
> Tnx Rui for your input.
> Your OS is Gingerbread, while Galaxy Nexus is ICS. Could it be that
> the "new" native libraries cause the slow down:http://groups.google.com/group/android-opencv/browse_thread/thread/e5...

>
> On Jan 26, 5:45 pm, Rui Marques <rui....@gmail.com> wrote:
>

[Ricardo Aires]

[Rui Marques]

If you run the sample tutorial, without your changes, is it also
slow?

[dschaudel]

Same issue here. I can only obtain 5-15 fps (android-opencv samples
with fixed camera libs).
I'm going to use android-opencv to implement a traffic sign
recognition and detection algorithm.
I'm currently thinking about offloading the processing to the cloud
(e.g. Amazon EC2).

[Philip]

Rui - are you using the opencv native camera interface VideoCapture?
I'm afraid that native camera calls are much slower than using the
Java camera implementation.

Does anyone have any insight on this issue (VideoCapture Vs. Camera)?

[Ricardo Aires]

Hello again!

@Rui
Yes, it is slow with all the samples, with the 640,480 rgba image.

@Philip
I've tested both implementations and for that type of image, the
Android camera works slowly. The problem here is, I think, the
conversion between the Yuv frame that comes from the Android camera
and the Rgba Mat format used by most OpenCV routines.

You can check the tests I tried here:

Android camera: http://pastebin.com/dwQ3yP9C
Opencv camera: http://pastebin.com/6irxf7T5

Feel free to scrutinize!

The results with the SGS are as follows
*Results:*
W/Android Camera(24643): Average frame time: 95.829ms ##Frame Rate:
10.435fps
W/Opencv Camera(25051): Average frame time: 64.445ms ##Frame Rate:
15.517fps



On Jan 29, 2:23 pm, Philip <sa...@matconsult.com> wrote:
> T

[Rui Marques]

I've used the native camera, from what i read it is faster.
Here some info about the cameras (i found it its not mine):

native camera:
+ higher framerate
+ capture RGBA, no need to convert from android yuv format
- compiled only for armv7 architecture
- not work on all devices (for now)
- not support autofocus, setting gain.. (for now)

[Andrey Kamaev]

Hi Rui,

The info about camera is not fully correct:

+ higher framerate

It is expected that this will be the case. At least native camera avoids two full-frame data copies: native to java and java to native.
But it takes accurate benchmarking to be objective.
 

+ capture RGBA, no need to convert from android yuv format

It just means that conversion is performed implicitly. You can get gray-scale frame on the cost of copying and RGB or RGBA on the cost of color conversion.
 

- compiled only for armv7 architecture

Camera libraries for Android 2.2.x, 2.3.x, 3.x.x are compiled both for armv5te (armeabi) and arm-v7a (armeabi-v7a). Prebuilt libraries for Android 4.0.x are available only for arm-v7a.
Most probably x86 version will be available in upcoming OpenCV release.
 

- not work on all devices (for now)

It is true for devices running Android 2.x.x (for example, Motorola devices). The problem should be fixed for Android 3.0.1 and newer.

- not support autofocus, setting gain.. (for now)

Correct. Might be fixed in upcoming OpenCV release.

/Andrey

[Philip]

Hi Andrey,

Though I didn't take a benchmark on it, I found that I achieve much
higher frame rate for Java camera implementation over VideoCapture if
no processing is taking at all (meaning, just obtaining images and
display).
Yet, in both cases if I do take any kind of precessing the frame rate
drops drastically.

From your experience, have u achieved acceptable (>20) frame rate for
the OpenCV samples?

Sagiv

[Philip]

I found this long and interesting thread:
http://code.google.com/p/android/issues/detail?id=2794
which discuss in detail the huge overhead of allocating and freeing
memory in onPreviewFrame(byte[] data, Camera camera)
The thread also offers an alternative which is to use:
public final void setPreviewCallbackWithBuffer(PreviewCallback cb)
public native final void addCallbackBuffer(byte[] callbackBuffer);

I guess that OpenCV VideoCapture doesn't implement such buffer, and
therefore suffers from a bottleneck in the performance (low fps)

As I wrote before, I tried to run the OpenCV samples without doing any
image processing (I commented out the processFrame function) and got
10-15fps for full image size on my Galaxy Nexus. I did the same test
with the example from the above thread and got 30fps!

Why hasn't OpenCV implemented such buffer for their VideoCapture
object?

Tnx

[Kirill Kornyakov]

Sounds interesting! It this approach is better, you can contribute a patch to OpenCV4Android.

Why hasn't OpenCV implemented such buffer for their VideoCapture
object?

Don't you know that library still has a beta-status and the development team has limited resources? The library is developed professionally, but we have tons on TODOs on every level: C++, Java API, accuracy, performance, samples, feature requests and yes, interaction with Android camera. So, if you have a piece of useful code, please share it. We will integrate it with pleasure!

Thanks,

Kirill

[Philipp]

Dear Kirill,
I didn't mean to be rude. Your work is highly appreciated and great.
I'm sure that you guys have a lot on your desk...

It's just that this issue (capturing performance) has been discussed
widely on the web for almost 2 year, so I assumed that you experts
have decided to implement VideoCapture in the way you did for a good
reason.


Best,
P.



On Feb 2, 9:57 am, Kirill Kornyakov <kirill.kornya...@itseez.com>
wrote:

[Kirill Kornyakov]

OK, we'll look into it. Actually ICS provide native OpenMAX API for the camera access and Video IO, and we're going to switch to this officially supported native API. Older native camera implementation is a bit hacky because Google supported only Java API for camera and changed C++ API - this is why older implementation of native camera was broken on Android 3.2 and 4.0. Currently we have a workaround for new Android, yes it is slow, and we'll work on native camera before the next public release. But our goal here is OpenMAX, because it is officially supported and may give us video files reading/writing as a bonus :)

Regards,

Kirill

[Bo WU]

Hi Kirill,

Are you sure the OpenCV native camera can be change to use OpenMAX API? According to the Android NDK document: android-ndk-r7/docs/openmaxal/index.html, CreateCameraDevice is not supported yet. 

[Linda Li]

This is an old post. But I happen to meet a similar problem.
I need to process the video frame, and then present on the screen, and save to a video file.

From what I researched, it seems the previewFrame can not fit the requirement.
http://stackoverflow.com/questions/8676046/why-fps-is-not-same-as-original-camera-app
One ID said: "The preview callback is merely giving you a sample of the actual video capture, not the entire video capture -- since most 15fps videos will still look smooth, the preview callback is only giving you a lower resolution, lower fps preview version of the actual video being captured. Even the built-in samsung capture application will only show the preview version, because that's all the preview interface is being given."

So preview is only for presentation purpose. The real frame needs to be from video recording.