OpenSL ES support

[Laurent Desegur]

What is the status of OpenSL ES support for Android in NDK?

[Angus Lees]

On Sun, Jun 27, 2010 at 09:03, Laurent Desegur
<laurent...@gmail.com> wrote:
> What is the status of OpenSL ES support for Android in NDK?

It may be added in the future, but right now there is no support for
OpenSL in the Android NDK.

- Gus

[Laurent Desegur]

Thanks for providing an answer.

Let me clarify here: on one side you offer OpenGL ES thru the NDK with
direct access to bitmaps in order to write games that have decent
performances (kudos for doing this btw) and on the other hand, you
expect developers to make games without native audio support so they
have to go back to Java for sound input/output support? How does that
work? Are we suppose to build games without any decent audio support?
Just fyi, IPhone has native (read c lang) native support since the
start, so what''s stopping Android from doing this?

Thanks for clarifying if you can. This would help me understand which
platform I going to put my efforts into for my next games.

On Jun 27, 4:06 am, Angus Lees <al...@google.com> wrote:
> On Sun, Jun 27, 2010 at 09:03, Laurent Desegur
>

[Laurent Desegur]

I don't think I can develop my games on android at this time because
of the hight latency in audio output and input on Android, courtesy of
Java. I don't even factor the gc. Is there anything I can do to help
implement OpenSL on Android at this time?

On Jun 27, 4:06 am, Angus Lees <al...@google.com> wrote:

> On Sun, Jun 27, 2010 at 09:03, Laurent Desegur
>

[Jack Palevich]

Unfortunately we are not able to accept contributions to the native
audio stack at this time.

I understand your frustration with the current situation. I hope you
will keep an eye on Android, and consider developing for it once
native audio support is available.

[Jack Palevich]

Unfortunately, there is currently no stable native sound API for
Android.

The key is stability. We need to be able to provide both a working API
and keep it working for every release of Android in the future.

We don't currently have such an API. As soon as we do have one we will
expose it in the NDK.

(For what it's worth we said the same thing about OpenGL ES and OpenGL
ES 2.0, both of which were missing from the first NDK.)

In the meantime, the work-around is to call through the Java sound
APIs, which many applications seem to have success with. Perhaps you
can develop your game using the current APIs now, and retrofit to the
lower latency pure native ones in the future.

On Jun 28, 12:42 am, Laurent Desegur <laurent.dese...@gmail.com>
wrote:

[Doug Schaefer]

I'd be happy if we had the same audio capabilities that the Native
Client SDK has. It's pretty minimal but looks easy to set up. Here's a
sample:

http://code.google.com/p/nativeclient-sdk/source/browse/trunk/src/examples/sine_synth/sine_synth.cc

> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To post to this group, send email to andro...@googlegroups.com.
> To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.
>
>

[Olivier Guilyardi]

On 06/28/2010 03:33 AM, Jack Palevich wrote:
> Unfortunately we are not able to accept contributions to the native
> audio stack at this time.

Sad, if I found time for it, that's exactly what I would have liked to work on.

I suppose that audio things are quite critical in a phone after all...

I just wish that instead of this terrific OpenSL API, you could get some
inspiration from the Jack Audio Connection Kit, or PulseAudio (this one's in
Maemo), or PortAudio, ...

Actually, a very simple process callback would really be great, something like:

void process(short *input, short *output, int nframes);

--
Olivier

[Doug Schaefer]

I actually thin OpenSL is overkill for most purposes. And as you
mention, there are actually multiple audio standards. The question I
have for developers out there who are supporting multiple platforms,
what audio APIs are you using, or are you targeting multiple APIs. If
it's the latter then I don't see the rush.

BTW, I still think you can do something simple as you suggest above
with a native library that communicates with a Java thread that pumps
the buffers into the audio system.

[Laurent Desegur]

I really appreciate the efforts that you guys put into the support of
OpenGL ES in NDK. I also appreciate your responsiveness following some
missing APIs in the ES 2.0 implementation. Kudos to Google for being
so responsive with supporting their developers.

WRT native audio: I would definitely be happy with a simple callback
for now until you guys work on implementing OpenSL ES.

Again, at this point, the latency involving JNI is not good enough, GC
is killing performances, and the integration between java and c++ land
is just too cumbersome. I don't believe that I represent an isolated
case. I've been playing with some games on Android, and the audio
integration seems to be less than ideal. I'd like to have audio output
flawlessly without gaps, very snappy audio effects and doing audio
recording as well. If game developers could get an embryo of OpenAL
for native callbacks, then I could go ahead port the cocosDenshion
project with simple audio engine to Android quickly. It would be a
trivial task to play background music and effects in a native game. As
you can read, I didn't give up yet on my hopes to get great audio
support in Android native apps.

Thanks,

LdS

[Doug Schaefer]

Java calling native is very fast. So if you avoid native calling up to
Java, I don't believe the latency would be significant. Is it?

Basically all you need is a way to feed AudioTrack with an array.
Allocate it in Java, pass it down to native and do a
SetShortArrayRegion and you're good to go. And if you reuse the array,
GC should be limited too. Is that what you're doing and you still see
bad latency?

[Olivier Guilyardi]

On 06/28/2010 09:27 PM, Doug Schaefer wrote:
> On Mon, Jun 28, 2010 at 3:05 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

>> void process(short *input, short *output, int nframes);
>

[...]

> BTW, I still think you can do something simple as you suggest above
> with a native library that communicates with a Java thread that pumps
> the buffers into the audio system.

This is what I'm doing, but it's not sufficiently reliable. Briefly, overruns
occur in many circumstances and latency has to be kept high to minimize them.
Duplex is unfeasible, because of overruns and constant noise on certain devices.

--
Olivier

[Olivier Guilyardi]

On 06/29/2010 01:20 AM, Doug Schaefer wrote:

> Java calling native is very fast. So if you avoid native calling up to
> Java, I don't believe the latency would be significant. Is it?

Latency is not related to how fast Java is. It comes from two things. First the
minimum buffer sizes imposed by AudioTrack and AudioRecord. Then, the fact that,
when using these minimums, overruns are frequent, and thus you need bigger buffers.

>
> Basically all you need is a way to feed AudioTrack with an array.
> Allocate it in Java, pass it down to native and do a
> SetShortArrayRegion and you're good to go. And if you reuse the array,
> GC should be limited too. Is that what you're doing and you still see
> bad latency?

GC freezing the whole audio input/output for dozens or hundreds of milliseconds.
This is exactly the werewolf that audio developers see in their worst nightmares.

Okay, you can work it around by minimizing allocations, etc..
But, from an audio developer point of view, it's wrong by design.

--
Olivier

[fadden]

On Jun 29, 9:28 am, Olivier Guilyardi <l...@samalyse.com> wrote:
> GC freezing the whole audio input/output for dozens or hundreds of milliseconds.
> This is exactly the werewolf that audio developers see in their worst nightmares.

Suppose Dalvik had some sort of concurrent collector that reduced the
duration of the freezes. What would you consider to be the maximum
acceptable duration for a pause?

[Olivier Guilyardi]

Well, I could answer that the *maximum* is 30% of latency. With a latency of
12ms (512 frames at 44.1Khz), that would be 360 micro-seconds.

But the concept of a pause or process freeze is wrong to me. What we need is a
native API and standard Linux scheduling. No GC, no lock.

I think that this could be achieved by having clients send and receive
synchronous streams to an audio server process through lock-free ringbuffers,
using a simple native API.

--
Olivier

[Olivier Guilyardi]

On 06/30/2010 12:26 AM, Olivier Guilyardi wrote:
> On 06/29/2010 11:07 PM, fadden wrote:

>> Suppose Dalvik had some sort of concurrent collector that reduced the
>> duration of the freezes. What would you consider to be the maximum
>> acceptable duration for a pause?
>
> Well, I could answer that the *maximum* is 30% of latency. With a latency of
> 12ms (512 frames at 44.1Khz), that would be 360 micro-seconds.

Humm, well, I meant 3.6ms, I'm tired.. ;)

--
Olivier

[Laurent Desegur]

Has anyone experimented with disabling gc when entering a JNI call to
process audio playback and reenable prior coming back to native land?

[Olivier Guilyardi]

To me, there isn't really a moment when GC is less critical than an other. An
audio thread should never lock.

That said, how do you intend to disable GC?

Olivier