video from I/O on NDK audio, and follow-up links & code

[Glenn Kasten]

The video from the I/O 2013 session

"High performance audio" (using Android native audio based on OpenSL ES)

is now available at http://www.youtube.com/watch?v=d3kfEeMZ65c

You may also the links and source code at this site helpful:

https://code.google.com/p/high-performance-audio/

(it is intended to be a companion to the I/O session)

[Eric Wing]

Glenn, I just watched the video and your presentation was awesome! And
an oscilloscope and photo diode...that is so hard core :)

Ironically, I've been working-on/frustrated-by completely
unrelated/non-audio Android performance problems, and I took a break
to watch your video since I plan to *eventually* get back on audio
stuff. Little did I expect to see a narrative on how you isolated and
profiled performance characteristics on Android. I now have a bunch of
new ideas on what to do next for my current performance problems and
am going to force all my (non-audio) colleagues to watch your video.

Great job!

Thanks,
Eric
--
Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

[Michael Moussa]

Hi Glenn, thank you for the great session at IO. You had told me to contact you on how to enable the 'slow path' in opensl es. In particular I need global audio effects to be enabled when playing audio using OpenSL. Also, is there any way to get at the audio session id from within the NDK using OpenSL ES?

I understand, both can be done easily in the Java mediaPlayer APIs, but I have a need for creating a media player in the NDK.

Thank You,
Mike 

[gadget]

excellent presentation, best of all that I have watched on youtube

[Glenn Kasten]

Thank you.  To force the 'slow path', you can either use a sample rate that does not match the native sample rate, or add of the effect interfaces at time of object creation.  Since the fast path does not support sample rate conversion or output effects, the implementation will notice this and disable the fast path.  Note that you don't have to actually _use_ the effect interfaces, just add them.   That said, we still recommend that you the SDK APIs such as android.media.AudioTrack in this case.  The SDK is still recommended for general-purpose (not lower latency) audio output.

Unfortunately there is no way to get the session ID that I'm aware of.

However I'm noting your request.

[Michael Moussa]

Thank You Glenn. For the audiosessionID thing, I did look in AOSP and it is right there in the CPlayer class if I remember correctly (even though I did notice a lot of changes from 4.0 to 4.1), it is just not exposed to the OpenSL ES API.  I would think it should be an Android extension to OpenSL ES accessed using OpenSLES_AndroidConfiguration.h

Also once this parameter is exposed. I would think using the sessionID should trigger the slow path, if that is possible and makes sense.

Thanks again for the great session,
Mike

P.S. I do also have a need for the fast path, and greatly appreciate the work you have done. I have noticed a lot of positive changes for audio in Android starting with Gingerbread and continuing through ICS and Jellybean.

[Glenn Kasten]

As one of my particular areas of interest is performance, of course I do care

about input latency.  But I can't give a commitment or schedule for either that or USB.

On Friday, May 31, 2013 2:46:36 AM UTC-7, Felix Homann wrote:

You've said that issues besides output latency could have been talked about in IO office hours. That's a pretty exclusive thing. So could you please tell something about input latency and USB audio support?

[Picksalot]

I saw the video "Google I/O 2013 "High performance audio"" today referred to in the article "Why Mobile Low-Latency is Hard, Explained by Google; Galaxy Nexus Still Musicians’ Android of Choice "

here: http://createdigitalmusic.com/2013/05/why-mobile-low-latency-is-hard-explained-by-google-galaxy-nexus-still-android-of-choice/

 

I also happened to see a press release "Sonoma Wire Works Announces Android™ Low Latency Audio Solution"

here: http://www.sonomawireworks.com/pr/android-low-latency-audio-solution.php

 

which claims "Typical input to output latency on Android devices is between 100 and 250 milliseconds. Sonoma Wire Works’ LLA solution brings latency down to approximately 20 milliseconds."

Are their claims credible?

[Glenn Kasten]

First, thanks for watching.  I appreciate your time and interest.

As mentioned in the video, we put initial focus on reducing

audio latency for output.  Reducing audio input latency is

also important, and some work has started on it,

but I can't commit to a release schedule here.

In my experience, after the audio framework is made friendly

for low latency (using appropriate scheduling policies and

priorities, avoiding inappropriate blocking, removing

resource contention and priority inversions, etc.) the next

most important task is making the scheduling predictable. 

This tends to be device-specific: making sure no drivers

disable interrupts for too long, or disable scheduling too long,

or run at elevated priority for long periods, etc.

It also requires tuning the power management system

for a good tradeoff between conserving the battery vs.

responsive audio.

The result is that some devices schedule more predictably than others,

even when running the same platform version, and thus

can tolerate smaller buffer sizes and lower buffer counts.

Also one has to look at what happens after the application

processor (AP) ... components in the signal path after AP can

also add latency, and these are of course device-specific.

One of my goals, besides improving the portable audio

platform performance, is to provide better tools and tests

for device OEMs and SoC providers.  I want to make

it easier for us all to find and fix performance glitches early.

It's my hope that this will raise the bar so that all

Android devices can support great audio performance.

Again, I can't commit to a schedule for these tools and tests,

but it's an area that's important to me.

Regarding your specific question about the press release,

I had heard a mention of it but I have not investigated their

approach myself.  In general, I welcome the discussion

of various solutions, and efforts by the

Android community to raise awareness for

and improve audio performance for all devices. 

If you're an audio app developer please continue to

keep the fire to our feet :-). If you contribute to Android AOSP

platform, drivers, etc. then please keep performance in mind

and visit android-contrib group.

Resources:

http://source.android.com/devices/audio.html

   (see not just the top-level article, but

    the sub-level articles in the index at left side of page)

all the external links at https://code.google.com/p/high-performance-audio

For platform contributors:

https://groups.google.com/forum/#!forum/android-contrib

http://source.android.com/source/contributing.html

[Picksalot]

Thanks Glenn for the detailed reply.

Initially, I thought Android's less than stellar audio performance, particularly on interactive music applications was ultimately controlled by hardware. I was pleasantly surprised by how much can be solved by optimizing software, including the OS, regarding latency.

Android is missing out on the positive impact that creative individuals in the music and film industries could have if they only had great working tools/Apps.

Early on, Apple understood the value that artists, musicians, etc. brought to their brand. There are a number of very useful music Apps Apple has that just aren't available on Android. I suspect in just about every instance, the main obstacles are latency, and OS fragmentation.

Android has done a lot of things right. But, they have been lagging in professionally useful Apps for musicians, etc. I'm glad Glenn, and others are working hard to help Android remedy this very important area of needed development.