Re: Android NDK Image Processing

[Jim Graham]

On Wed, Jul 11, 2012 at 07:02:52AM -0700, mkerim wrote:

> I currently have an algorithm which basically processes every preview
> frames. I have tried to capture the video using MediaRecorder of SDK,
> however, apparently I can't access the preview frames while MediaRecorder
> is capturing the video.

I assume you're capturing this video at around a few frames/second and/or
at a very, VERY tiny resolution? Or are you using a Cray? :-)

Seriously, you don't generally process a video stream during its capture.
The time to process video is AFTER you've capbured it. That is, unless
you don't care about reaching either the PAL or NTSC frame rates of
25 fps (PAL) or 29.97 fps (4:3 NTSC) or 23.97 fps (various NTSC widescreen
formats).

> First question is; is it possible to do this using NDK? I mean capturing
> the video and processing the frames in the mean time.

I don't know if it's possible, but I can tell you it's not generally a
good idea. Do your editing/processing in post, not during capture.
Then you don't have to worry about how many frames/second you can manage
with your processing algorithm....

Later,
--jim

--
THE SCORE: ME: 2 CANCER: 0
73 DE N5IAL (/4) | "Now what *you* need is a proper pint of
spook...@gmail.com | porter poured in a proper pewter porter
< Running Mac OS X Lion > | pot.."
ICBM / Hurricane: | --Peter Dalgaard in alt.sysadmin.recovery
30.44406N 86.59909W |

Android Apps Listing at http://www.jstrack.org/barcodes.html

[mkerim]

Thank you for your answer, actually that is exactly what I had in mind. As of now, adopting this post-processing approach seems to be the best way. Actually, with post processing in my mind, let me ask another question.

Scenario will be the following in my project.
Video will be captured, encoded and saved. Right after that, maybe using a button , video will be decoded and every frame will be processed by the algorithm (native code). After processing, certain metadata will be generated by the algorithm. After that, I have to encode the video again and save it (metadata will be expressed as some pixel alteration on the respective frames, indication of an "event"). 

The question is, as far as i know, I can decode the video using SDK. However, after processing the frames, saving the video again with altered frames, is it possible with tools SDK provide? Should I deal with NDK again? I am trying to avoid NDK because project has a strict time limitation.

If the above approach requires NDK, I think I will just decode the video and generate the metadata, and express this in another way, without the need of saving the video again.

Thanks.

[Jim Graham]

On Thu, Jul 12, 2012 at 06:26:26AM -0700, mkerim wrote:

> The question is, as far as i know, I can decode the video using SDK.

That depends on the size of your video (frame size, bit rate for both
audio and video, audio sampling rate, and length). And if your video
is small enough to fit into the Java side's memory restrictions, can
it do the frame-by-frame editing? I don't know.

Good luck.

Later,
--jim

--
THE SCORE: ME: 2 CANCER: 0

73 DE N5IAL (/4) MiSTie #49997 < Running Mac OS X Lion >
spook...@gmail.com ICBM/Hurricane: 30.44406N 86.59909W

Do not look into laser with remaining eye.

[Steven Winston]

You don't need a super computer to process images in real time; that's ridiculous.  We do it all the time; it's a whole subject area known as augmented reality.  There's several techniques for managing the load on a modern cell phone while keeping up 20-30 fps.  The normal techniques that are well known to the industry work well on Android (process only some frames; in a separate thread; use dead reckoning; and display on top of the stream rather than writing to it etc.)
As for the original question, you can use 3rd party APIs quite well in post.

[Jim Graham]

On Thu, Jul 12, 2012 at 09:40:12AM -0700, Steven Winston wrote:
> You don't need a super computer to process images in real time; that's
> ridiculous.

It's only ridiculous if you actually believe I meant that literally. And
if you do believe that...I've like to sell you some GREAT land about 20
miles south of where I live. An advanced open water certification from
PADI is recommended. Interested?

> We do it all the time; it's a whole subject area known as augmented
> reality.

It was my understanding that the reason was to capture video...not AR.
Video generally means one of three frame rates: 25 fps for PAL, or 23.97
or 29.97 fps for NTSC. And depending on the frame size, compression (if
any), etc., that can be a problem. There are a lot of variables, and
how bad (or if) can vary a LOT. But based on my experience, I'd still
go for post-processing, where you don't have to worry about it at all.

Later,
--jim

--
THE SCORE: ME: 2 CANCER: 0

73 DE N5IAL (/4) | Tux (E Cat): DS B+Wd Y 6 Y L+++ W+ C++/C++ I+++
spook...@gmail.com | T++ A E H+ S V- F++ Q+++ P/P+ B++ PA+ PL SC---

< Running Mac OS X Lion > |

ICBM / Hurricane: | Tiggerbelle: DS W+S+Bts % 1.5 X L W C+++/C+
30.44406N 86.59909W | I+++ T A E++ H S++ V+++ Q+++ P B++ PA++ PL+ SC

[Steven Winston]

I was taking issue with the idea that this is impossible, or even not recommended, not to give the validity of which way is the best for an application; which I'll note didn't mention the design requirements.
Capturing video and displaying it are two separate processes.  Someone saying realtime in video processing, or even working with the preview images says to me that they want to worry about immediate display on the device.  Why else would anyone not do post processing?
If the idea is to do post processing only, then getting the right codec is a must.  FFMpeg is quite popular for doing such things; probably unnecessary if you know what the codec is and always write back to the same codec (i.e. just do processing on each image).  In which case, just grab the correct codec, run over the finished recording once it finishes the capture, and that should work.
Another way to do this entire process is to use OpenMAX.  Then join the fun of kicking Google in the shins until they fully implement that wonderful API in the NDK as parts of it aren't ready for prime time yet.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[mkerim]

Thanks a lot for the answer guys. Let me clarify the scenario a bit more.

There is a main activity screen which has certain buttons to indicate and initiate some events, such as start/stop video recording. Indicator buttons change color with respect to metadata produced by native algorithm. Algorithm work pretty good, using upto %65 of CPU in the worst case I have seen so far.

Whenever I initate videorecording, live feedback on the UI dies. By that, I mean buttons on the main interface do not change colors. After brief debugging, I have seen that onPreviewFrame method is not called after video recording starts (I pass every frame to native code in this method and obtain the metadata as well).  

I have checked some forums and seen that onPreviewFrame can't be called when MediaRecorder takes over camera. Therefore, I had no choice but to use NDK. At this point, I have to produce a demo that actually works, therefore real time or post processing does not matter. After this project is accepted, I will have loads of time to improve the performance. 

Since I want the demo to be running nice and sound, I think I will go with post processing since I doubt I can write a nice real time processing code piece in short notice, which being few weeks.

I think I have to join the ranks of ffmpeg, decode the video, process it frame by frame, get the metadata and list them somewhere else. I can kiss my live feedback dreams goodbye due to time limitations.

Thanks a lot again.

To unsubscribe from this group, send email to android-ndk+unsubscribe@googlegroups.com.

[Jim Graham]

> Capturing video and displaying it are two separate processes.

Obviously....

> If the idea is to do post processing only, then getting the right codec is
> a must. FFMpeg is quite popular for doing such things;

Back when I ws doing this, I used mjpeg. I can't remember the setting,
but I was able to fit about 2.5 hours into a 220 GB drive. Then I'd do
any editing (since this was usually from TV, remove commercials, make
sure each cut/join is clean, and so on), add a noise filter if needed,
resize from 720x480 to the correct aspect ratio (in this case, since I
was capturing from a TV, 4:3), at 640x480, and before going to bed, or
going to work, start a conversion to Divx or Xvid, which would usually
take about 2--3 hours per hour of video. This was ca. 2000--2004, and
my computer (the one I used for video...the one I used for everything
else was a FreeBSD system) back then was only a 1.6 GHz P2 running
mickeysoft windoze xp.

Later,
--jim

--
THE SCORE: ME: 2 CANCER: 0

73 DE N5IAL (/4) | DMR: So fsck was originally called
spook...@gmail.com | something else.
< Running Mac OS X Lion > | Q: What was it called?
ICBM / Hurricane: | DMR: Well, the second letter was different.
30.44406N 86.59909W | -- Dennis M. Ritchie, Usenix, June 1998.

[Shervin Emami]

If you mean that you want to analyze & modify preview camera frames as they come streaming in and to display your processed frames on the screen in real-time using NDK, then yes you should try OpenCV for Android "http://opencv.org/platforms/android.html", it is a free library with several examples for doing exactly this, and includes about 1000 image processing & computer vision algorithms, and will soon include hardware accelerations for mobile devices. But if you need to store the resulting image to a video file (eg: an AVI) then OpenCV is not yet able to do that, since Android makes it difficult to do so, but hopefully it will be possible soon.

-Shervin.

On Wednesday, July 11, 2012 7:02:52 AM UTC-7, mkerim wrote:

I currently have an algorithm which basically processes every preview frames. I have tried to capture the video using MediaRecorder of SDK, however, apparently I can't access the preview frames while MediaRecorder is capturing the video.

First question is; is it possible to do this using NDK? I mean capturing the video and processing the frames in the mean time.

Second question is; if it is possible, what 3rd party API's do you prefer ( apparently NDK doesn't have an official API for this)? I am developing the app for HTC Desire S and using SDK 2.2.

Thanks a lot.

[mkerim]

Hi again,

I have already developed an algorithm for computer vision, it is tested and working just fine. The problem is, as I have stated before in previous posts, video recording. SDK doesn't allow this real time processing and capturing, and sadly NDK doesn't have any API to support video capturing, like SDK does. I have already gone deep with native coding and planning to use ffmpeg for encoding and saving purposes. I hope it will work just fine. Thanks for the answers all, much appreciated.