Access for Camera in NDK

[Karan Parikh]

Hey All,
I am developing a camera based application. I have implemented the
application in Java. The Java version provides low fps due the
presence of Garbage Collector and decrease in the speed of conversion.
I am planning of accessing the stream directly in the NDK part of it.
So is there any way I could access the camera stream in NDK ?
Regards,
Karan.

[Tim Hutt]

2009/7/10 Karan Parikh <karan...@gmail.com>:

You really want to access this class:
http://android.git.kernel.org/?p=platform/frameworks/base.git;a=blob;f=include/ui/Camera.h;h=e593feab7899aa29c1c0267fb369b8d3ec2644e9;hb=8ecb36eec61f119f500a805b82438aadb3396a19

It's not supported officially but you might be able to get it to work
by copying that header (and dependent headers) to your app and somehow
linking with the necessary library. It's going to be complicated.

[karan parikh]

Thanks Tim. Will try it out and let you know.
Regards,
Karan.

--
Regards,
-Karan.

[Dianne Hackborn]

This is NOT SUPPORTED.  Especially when it comes to native code, when you go outside of the official APIs you are -guaranteeing- that your app will break in the future.

Btw, heads up: I have seen an app now that breaks in the next release because it put some code in the lib directory that is not "armeabi".  This is not following the well-defined rules, so I am not going to put any effort into making it work.  We go to a lot of work to help people understand what is and isn't supported; if people don't follow those guidelines, we can't spend time trying to keep them running.

On Sat, Jul 11, 2009 at 6:11 AM, Tim Hutt <tdh...@gmail.com> wrote:

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[D. Kevin McGrath]

This is a little extreme.  You are not -guaranteeing- anything.  You are simply running the risk of breakage.  An unstable API isn't something that will DEFINITELY change.  Only something that can't be guaranteed to remain the same.  Look up the meaning of the unstable someday.

Unless, of course, you know exactly what's going to change and when.  In which case, publicizing that would be of enormous benefit to the community.

And I don't recall seeing anyone asking for you to "spend time trying to keep them running".  As developers, we generally have a good understanding that when we are doing something unofficial or NOT SUPPORTED, there is a risk of breakage.  But, for what we need, it works right now, and we'll fix it later as needed.

I don't mean to be rude or offensive here, but sometimes it comes across that android is YOUR playground, and you only let us, the developers, play so long as you like us.  And as soon as you don't, you take your toys and go home.  But without us, your playground would be a weed choked lot somewhere.

Now, back on topic to Karan's question.  I'm assuming you are capturing the image in some form of binary format.  Odds are some sort of 2D binary array (or 3D, depending on format).  Assuming that is the case, passing a pointer to this structure to C/C++ via a JNI call should be a low overhead operation, and you can do whatever transforms you desire using native code.  Then pass back the structure to the Java code to display it.

Obviously, not ideal, but as long as you want to stay in the supported playground, probably the only real way to do it.  Maybe someday we'll get a real NDK, rather than this crippled version they are currently "supporting".

And if that seems a little bitter, well, it probably is.

Kevin

[Dianne Hackborn]

On Sat, Jul 11, 2009 at 12:20 PM, D. Kevin McGrath <d.kevin...@gmail.com> wrote:

This is a little extreme.  You are not -guaranteeing- anything.  You are simply running the risk of breakage.  An unstable API isn't something that will DEFINITELY change.  Only something that can't be guaranteed to remain the same.  Look up the meaning of the unstable someday.

I am pretty sure the C++ APIs have changed for every release, and have done so again in the next one.  There is a reason these things aren't public.
 

Unless, of course, you know exactly what's going to change and when.  In which case, publicizing that would be of enormous benefit to the community.

We know what we can't be maintaining in its present form, which is the reason they aren't part of the public API.  There is no need to know exactly when and how they will change, the important thing is that we can't make a guarantee that they -won't- change.

And more than that, there are lots of manufacturers who will be releasing Android phones, who may have changed any private API in ways nobody working in the platform may have imagined.  Given a large number of people customizing the platform, and rapid evolution of the platform itself, you just need to assume that if you are using a private API, you will break at some point.
 

And I don't recall seeing anyone asking for you to "spend time trying to keep them running".  As developers, we generally have a good understanding that when we are doing something unofficial or NOT SUPPORTED, there is a risk of breakage.  But, for what we need, it works right now, and we'll fix it later as needed.

I am certainly not going to stop you.  But I also want to make it clear to anyone reading the post that what was being so cavalierly suggesting was going to result in an app that most likely breaks in the future.

As far as somebody asking me to spend my time to keep them running -- I guess you aren't aware of all the complaining that has happened when private APIs have changed in the past and broken people's apps.  People certainly do ask for this.
 

I don't mean to be rude or offensive here, but sometimes it comes across that android is YOUR playground, and you only let us, the developers, play so long as you like us.  And as soon as you don't, you take your toys and go home.  But without us, your playground would be a weed choked lot somewhere.

We have spent literally months of engineering time trying to clearly define what can be supported and what can't, and setting things up so that developers will as clearly as possible know when they get outside of that supported world.  In spite of all that effort, if you know the right tricks you can easily subvert those protections -- such by using reflection in Java, or copying in a header like here in C++ -- which others then see and use without realizing what they are doing.

Numerous times I have seen people post on android-developers about some code they got from somewhere to accomplish something, which uses such tricks and will in fact break in the future.  The developer posting it however has no idea.

So what should I do?  Just be silent and let their code break in the future?  I would rather post wherever I see this so that people will know what they are doing.  If after that they still decide to go ahead and bypass the rules at the risk of breaking, that is fine, but at least they know.

I don't know why you see this as a personal vendetta.  No, you can't safely access every last API in the system.  That is no different from every other operating system.  It certainly has nothing to do with liking you.

And you will note that nowhere did I demand that the private API -not- be used.  I can't prevent anyone from using it.  I just stressed that it was indeed private and would lead to a broken app in the future.  Given that knowledge, if you decide to run the risk and use it, that is ultimately your choice.  And later when I get a bug filed against me saying your app isn't working on some later version of the platform, I will see that it is using a private API and can rest easy at night closing that bug as "will not fix."

Well actually I wouldn't be the engineer getting that particular bug, but I will free to speak for them. :)

--

[Tom Gibara]

I'm side-stepping the discussion with Dianne regarding platform stability, I just wanted to explain that your suggestion to Karan for keeping his application on a 'firm' footing doesn't work.

The problem with the Camera API is that for every preview frame captured it allocates a new Java array to contain the preview data. So for every single frame captured it allocates approximately 0.5MB of fresh memory. This causes the GC to continually spin into action and kills any effort to produce an application that performs well.

To make matters worse, the method to which this preview frame is delivered is required to return promptly (or the process crashes out somehow - not sure why) so you generally need to hand-off the supplied array to another thread if you have any non-trivial work to do. Add to this the complication that the preview frames are pushed at a fixed rate that you can't adjust while the camera is in use (if at all) and it's clear that the situation is irredeemable within the framework as it stands.

I'm not a fan of the Camera API (there are a number of inadequacies) but this is by far its greatest transgression. The correct solution isn't entirely obvious either*, but the correct approach is surely to suggest/supply fixes to the AOSP.

Tom

*I'm currently thinking that a byte buffer combined with a read lock would be adequate, I'll post the rest of my comment to the framework group.

2009/7/11 D. Kevin McGrath <d.kevin...@gmail.com>

[Tim Hutt]

2009/7/11 Dianne Hackborn <hac...@android.com>:

> I don't know why you see this as a personal vendetta.  No, you can't safely
> access every last API in the system.  That is no different from every other
> operating system.  It certainly has nothing to do with liking you.

I think people are just frustrated that the stable APIs are so limited
at the moment. They basically don't let you do anything you couldn't
do without them:

* libc (C library) headers
* libm (math library) headers
* JNI interface headers
* libz (Zlib compression) headers
* liblog (Android logging) header
* A Minimal set of headers for C++ support

Not exactly thrilling stuff! Everyone will be much happier when you
can access phone hardware from native code. These would be my
priorities (in order):

OpenGL
Button & touchscreen input (btw is it going to support multi-touch?)
Camera input
Audio output
Accelerometer & compass.

That's pretty much everything.

[D. Kevin McGrath]

Add to this real C++ support. Basically, without the STL or at least
RTTI, what's the point of providing C++ at all?

[Jack Palevich]

There's little need for a native version of some of the APIs you're
listing. For example, events are low-frequency enough that it there is
relatively little overhead caused by passing them through Java. The
same for the accellerometer and the compass.

OpenGL and Audio and Camera access certainly would be useful, and we
hope to offer them in the future when they stabilize.

[Doug Schaefer]

OpenGL stabilize. That still makes shake my head. It's a sign that the NDK is very much an after thought. How hard is it to pick an shared library name and get the hardware providers to stick to it, or at worse, create soft links to hide the real one.

 

As I'm sure you can tell. The official line is growing thin with this community. A real plan might help us with our planning or you're going to see these kind of hacks end up in apps on the market and as they do start breaking, it's not going to look good for anyone.

[Jack Palevich]

If you read the Android sources, you'll find lots of C++ code. C++ is
a very useful language even without RTTI, STL, and exceptions. After
all, C++ was popular for many years before these features were added.

As I think people have mentioned when this was discussed before, RTTI,
STL, and exceptions are not supported because they are not appropriate
for a small-memory, relatively low-performance system like a phone.

That being said, I think several people are working to make ports of
STL available. Very few people care about RTTI (at least, we've never
heard anyone make a strong case for why it's needed or useful.).
Exceptions are the tricky part, because they are very useful for some
developers, but they need runtime support in order to work -- you
can't just bolt them on after the fact.

(For example, what happens if you pass code to qsort and it throws an
exception? Should you expect to receive the exception? If so, it means
that qsort has to be written to support exceptions, even though qsort
is a C function.)

[Jack Palevich]

Most community members understand what we're doing and accept it. I'm
sorry that you personally are frustrated, and I hope you'll be pleased
when we provide more APIs in the future.

[D. Kevin McGrath]

C++ without at least exceptions and some form of STL support is
essentially useless in this situation, given the JNI interface. One
of the fundamental advantages of C++ over C is the ability to shield
yourself from the most common pointer related errors, but the JNI
interface as provided makes extensive use of pointers. And really,
without templating and exceptions, there isn't much advantage of C
over C++, in this situation, due again to the JNI interface.

Well, at least if you want to code your apps in such a way as to avoid
memory leaks and JNI errors.

Also, let's be realistic here. You claim phones represent low memory,
low performance. And in today's computing environment, that is
reasonably true. However, modern phones such as the Ion or the Hero
have as much or more computing power and memory, relative to many
computers in use even 10 years ago. Computers that often made use of
templates, exceptions, and RTTI.

Waving around performance and "low" memory as if it is a panacea to
cover the lack of credible native APIs is not only ineffective, it is
disingenuous. And quite frankly, an insult to the community.

OK, I admit to being slightly irritated with this whole situation,
because I'd really like to make better use of the android platform.
The future is mobile ubiquitous computing. And I think those of you
at Google are well aware of this. Limiting your mobile platform in
this fashion does neither us nor you any service. As previously
stated, these hacks *will* likely end up in shipping android apps.
And when the next OTA update occurs, many applications will cease to
function. Now, as a developer, when an app stops functioning due to a
system level upgrade, I know it is usually the fault of the
application. However, to most users, there is no distinction, and
when half of their apps break, they won't blame the devs. They will
rant and rave about "google/tmobile broke my device!!!" You know this
as well as we do.

So basically, at this point, the NDK seems like a bandaid and a
crutch, just so you can point and say "we have native development
too!" And it needs to be more than that. OpenGL is a fundamentally
stable API. The STL is also. I am willing to concede button/
touchscreen input has no real need for a native interface, so long as
a java based app frontend is required. But the compass, the
accelerometer, audio, camera....these are all fundamentally low level
devices hanging off the bus...best addressed with native code.
However, given the relative newness of the platform, I do entirely
understand the instability in some of these APIs. But that will begin
to wear thin as donut, eclair, and definitely flan come along to the
scene. Basically, if by the time flan comes along, we are still in
the same boat, I imagine Android will go the way of the PalmOS. And I
definitely don't want to see that happen. Competition in the mobile
space is nothing but a win for everyone.

I'm sorry this comes across as me being angry. I'm not angry at all,
just disappointed. I'd really love to love Android. But at this
point, it's too half baked -- not even half baked, really. I'm quite
frankly surprised it isn't one of the eternal betas we see lately.
And claiming API instability in a shipping, non-beta platform is
rather...unprofessional. However, this is of course my own opinion.
I not only grant you the right, but encourage you to disagree with me,
and convince me that I'm wrong. I'd love to be wrong. But without
honest, sincere criticism, the platform cannot advance. And that was
what I intended this to be.

Kevin

[Doug Schaefer]

I'm not personally frustrated at all. I know OpenGL is a stable API so I'm good. Just don't try to mess things up by trying to combine ES 1.1 and ES 2.0 or something ;).

 

I'm just worried that the comments from the inside are fanning the flames more than helping. Just some friendly advice from someone who's been there. A little more transparency would really help the spirit of the community at least.

 

As for the STL/RTTI discussion, we probably should be sticking to the technical reasons why it hasn't been included. Whether someone thinks it's useful or not is immatereal. The early android devices are really tight on space so I can understand this call in the short term. But, long term as devices grow, that'll change.

 

I think we'd all just like to see the Android NDK support standard APIs, or at least the APIs we see in other mobile platforms. Remember that Android is fighting for developer mindshare and we're just trying to make the best apps for Android that we can.

[Tim Hutt]

2009/7/12 Jack Palevich <jac...@google.com>:

> As I think people have mentioned when this was discussed before, RTTI,
> STL, and exceptions are not supported because they are not appropriate
> for a small-memory, relatively low-performance system like a phone.

Really? Even compared to interpreted Java? Sounds dubious.

> That being said, I think several people are working to make ports of
> STL available. Very few people care about RTTI (at least, we've never
> heard anyone make a strong case for why it's needed or useful.).

I've used dynamic_cast<> in the past. Occasionally it is useful or unavoidable.

[Jack Palevich]

On Sat, Jul 11, 2009 at 8:08 PM, D. Kevin
McGrath<d.kevin...@gmail.com> wrote:
>
> C++ without at least exceptions and some form of STL support is
> essentially useless in this situation, given the JNI interface.  One
> of the fundamental advantages of C++ over C is the ability to shield
> yourself from the most common pointer related errors, but the JNI
> interface as provided makes extensive use of pointers.  And really,
> without templating and exceptions, there isn't much advantage of C
> over C++, in this situation, due again to the JNI interface.
>
> Well, at least if you want to code your apps in such a way as to avoid
> memory leaks and JNI errors.

It's possible to write safe efficient JNI code without exceptions and
the STL. People typically to write a few small utility classes to help
with memory management, and typically to use return codes to handle
error checking.

>
> Also, let's be realistic here.  You claim phones represent low memory,
> low performance.  And in today's computing environment, that is
> reasonably true.  However, modern phones such as the Ion or the Hero
> have as much or more computing power and memory, relative to many
> computers in use even 10 years ago.  Computers that often made use of
> templates, exceptions, and RTTI.

I think you may not be entirely correct here. 10 years ago was 1999.
Most of the world was running Windows 98SE. I can't think of any major
application people ran 10 years ago that was written in C++ with
templates, exceptions, and RTTI. Most were written in straight C. Some
(on the Windows side) were still coded partly in assembly.

Maybe I'm just not thinking of the same applications or hardware that
you were -- could you give some examples?

Hi Kevin,

Thanks for taking the time to explain you position. I am sorry you
can't use the NDK it its current state. I hope you'll check back again
in future releases.

[pin xue]

As it is already off-topic, ^_-

C++ standard history:
1990-7 template
1990-11 exception
1993-3 rtti
1993-7 namespace

Borland released OWL 2.0 based on BIDS for windows developers in 1993, BIDS is template based data structure bundled with borland c++ 3.1.

And visual c++ 6.0 is released in 1997, though I were one of borland fans and didn't use it at that moment.

--
Best Regards!

Yang Wu
--------------------------------------------------------
Location: Pudong, Shanghai, China.
EMail    : pin...@gmail.com
Website: http://www.pinxue.net

[Holly Lee]

Even today most of C++ programs are written without templates, exception and rtti :-)

--
Programming is Art

[Tim Hutt]

2009/7/12 Holly Lee <holl...@gmail.com>:

> Even today most of C++ programs are written without templates, exception and
> rtti :-)

Rubbish.

[Holly Lee]

Why?

--
Programming is Art

[Tim Hutt]

2009/7/12 Holly Lee <holl...@gmail.com>:
> Why?

Well exceptions aren't that commonly used and RTTI is pretty rare, but
nearly all C++ programs use some kind of templated container or string
class (std::string, QList, CString, etc.)

This is now really off-topic.

[D. Kevin McGrath]

Correct. It is rubbish. Those aren't C++. They are C with OO bolted
on. Further, it's even more crap. What do you think strings are?
Pairs? dynamic_cast? min, max, sort, copy?

If you aren't making use of templates, at least, you aren't using the
features of C++ that differentiate it from C. And if you aren't using
exceptions, you aren't writing fault-tolerant code. And given the
nature of computers, fault tolerance is rather...important.

[Elvis Dowson]

Hi,

On Sun, Jul 12, 2009 at 9:36 PM, D. Kevin McGrath <d.kevin...@gmail.com> wrote:

If you aren't making use of templates, at least, you aren't using the
features of C++ that differentiate it from C.  And if you aren't using
exceptions, you aren't writing fault-tolerant code. And given the
nature of computers, fault tolerance is rather...important.

Templates are not the the only reason one would choose to program in C++. You do it mainly for object orientation, data encapsulation, polymorphism, inheritance, composition (via containment or aggregation), code reusability, code modularity, loose coupling and better maintainability. Templates just allow you to write more efficient compact generic code, that can be used for different types. 

The android SDK doesn't implement the full libstdc++, you just have 4 or 5 constructs, so you can reuse many or the standard c++ library features. You can use the uSTL library and port it and get it to work with android, if you want to use templates. 

For safety critical systems, if you use C++, you will need to use a subset of C++ called EC++ (embedded C++). In that, use of exception is totally discouraged, because if you raise an exception and is not handled at run-time, it will cause the application to exit, not good if you're controlling subsystems on an aircraft or automobile. 

Also, for performance reason, reducing code size, etc, the use of RTTI is also avoided, in some applications. 

To improve traceability and debugging, use of multiple inheritance is discouraged, restricting it only to use of single inheritance. 

All memory and objects must also be allocated statically, to avoid out of memory situations, due to dynamic memory allocations. 

So, in short, every safety critical or embedded system project, be it either C, C++ or ADA83 / ADA95 based, will all specify a safe subset of a programming language, and use that subset for a particular project. 

So, Holly Lee is quite correct, from a real-time embedded, safety critical applications perspective. If you look at C++ coding standards for any defense and aerospace company for a real-time embedded project that uses C++, which also get reviewed and certified by the FAA or JAA, you will find that they won't use exceptions, RTTI or multiple inheritance, to improve the determinism and verifiability of the implementation.

Elvis Dowson

[niko20]

On Jul 12, 1:55 pm, Elvis Dowson <elvis.dow...@gmail.com> wrote:
> Hi,
>

> On Sun, Jul 12, 2009 at 9:36 PM, D. Kevin McGrath <d.kevin.mcgr...@gmail.com

Way I look at it is this: Android devices are embedded small devices.
Try programming a Nintendo DS or Gameboy Advance, for example. Each
has its limitations and yes the developer has to do more work and
thinking to optimize their program for that particular platform.

And in a way exceptions are handled, they are passed back up to the
Java layer. I can catch a Throwable, etc, in the Java routine that
calls the JNI routine.

And I don't find the NDK useless even with the limited API, in fact it
allows me to do actual real fixed point math directly on the
processor, since if you try to "emulate" fixed point in Java, you will
find no real performance improvement above using floats or doubles in
Java. Try it out and see! So the NDK will give you good speed up if
you code your algorithm correctly to take advantage of the fact that
there is nothing in between the code and processor now. (pass in a JNI
buffer, act on the data, send it back)

Basically no it doesn't allow the developer to be lazy. If you dont
like the APIS and yet you can't come up with a way to still do what
you want, then you are not very motivated to develop that app I guess.
Maybe stop being so scared to try something different.


-niko