How to get a low latency audio in Android4.4 ?

[evan...@gmail.com]

I saw Glenn Kasten's video in YouTube about android5.0 audio latency
improvement, that’s an amazing short latency!

Recently, I met an issue like this: Audio latency is very serious at
our device. Use openSLES, there’s about 200+ms delay.

So I wonder how to gain a low latency audio in Android4.4 ?

----------------------------------------------------------
My idea as below:

There're three buffers: 1. share buffer used by client(AudioTrack) and
server(AudioFlinger); 2. mainBuffer for audio mix; 3. buffer in alsa
driver

So I think low latency can be get by reducing size of these three buffers. XRUN would happen if buffer size is too small, so there must be a balance value.


-----------------------------------------------------------
Now my method as below:

1.  Set period size and count in HAL;

Output: 256 * 8

Input：256 * 6

2.  Set kMinNormalMixBufferSizeMs and kMaxNormalMixBufferSizeMs to 5
and 6 respectively in Threads.cpp;

3. Set minBufCount = 1 in both PlaybackThread::createTrack_l and
RecordThread::createRecordTrack_l;

4. Don't double input buffer in AudioRecord::getMinFrameCount;


-----------------------------------------------------------
My local test result:

Local loopback test: direct HAL read and write.

stream read/write, I open a direct API in AudioFlinger to invoke
PlaybackThread->mOutputSink->write; and
RecordThread->getInputStream()->stream->read.

In theory, HAL read/write should not be longer than pcm read write, I
don’t know why it cost 30ms more. Maybe by Vendor’s resample in HAL
and IPC, need more investigation.

57 ~ 60ms delay.


Local loopback test: direct pcm device read and write. DO NOT use
android API, operate on alsa pcm device directly.

20 ~ 30ms delay.


Test method:

Use laptop computer to record sound before enter Mic and out from
speaker, then analysis pcm wave by CoolEditor.



Thank you so much for your help!


Evan.Xu

[Glenn Kasten]

Thanks for watching the videos.

To get lower latency, it takes a partnership of the app developer, Android framework, OEM and SoC vendor (including audio HAL),

kernel, and testing. Here are a few of the key things for each part.

app:

Use Android native audio based on OpenSL ES.

Check for feature android.hardware.audio.low_latency

  see http://developer.android.com/guide/topics/manifest/uses-feature-element.html

Use the recommended native buffer size and sample rate;

 see http://developer.android.com/reference/android/media/AudioManager.html#getProperty(java.lang.String)

 [the same buffer size and sample rate should be used for input also]

Usually an OpenSL ES buffer count of 1 is sufficient.

Keep your callback handlers short, without bursty CPU usage,

  and no unbounded blocking.  Avoid priority inversion.

  See https://source.android.com/devices/audio/avoiding_pi.html

You may want to consider using a non-blocking library

  to communicate between input and output callback handlers.  See https://source.android.com/devices/audio/avoiding_pi.html#nonBlockingAlgorithms

framework:

We've been gradually improving the framework over successive releases.

This article describes the initial design:

  https://source.android.com/devices/audio/latency_design.html

The article needs to be updated to describe more recent improvements

  (especially on input side). I'm making a note to self to update that.

Generally, newer releases

can offer lower latency than older releases on the same device.

We expect to continue to make more improvements in future,

but I can't give a schedule.

  

OEM and SoC vendor, including audio HAL:

You mentioned 6 or 8 buffers in HAL.  I find that usually 2 buffers is enough.

Avoid priority inversion in the HAL. Pay attention especially to mutexes.

Avoid or reduce extra audio processing in HAL such as signal "enhancements"

  for lower latency use cases.

If there is a DSP after the app processor, try to keep any processing

  by the DSP minimal.

I noticed you made some changes to the portable audio framework.

  Those changes should not be necessary if you do the other suggestions here.

Kernel:

Reduce worst-case interrupt disable and preemption disable times.

Power management can sometimes interfere with scheduling.

Check if your DRM security kernel has any long dark times 

  during which the regular kernel is unable to run.

Testing:

Here are links to the test techniques and tools that I've used:

https://source.android.com/devices/audio/latency_measure.html

https://source.android.com/devices/audio/loopback.html

https://source.android.com/devices/audio/testing_circuit.html

For an overview of latency issues, see

https://source.android.com/devices/audio/latency.html

[evan...@gmail.com]

Hi Glenn，

Thank your so much for your reply!

在 2015年3月2日星期一 UTC+8上午11:23:34，Glenn Kasten写道：

[Glenn Kasten]

In my March 1 post, I neglected to provide a link to one important resource, the NDK documentation:

after installing the NDK, see <NDKroot>/docs/Additional_library_docs/opensles/index.html 

especiall section "Performance".  This section has some useful information about latency reduction techniques.

[Glenn Kasten]

More resources on audio latency:

- video "audio latency: buffer sizes" 

  at https://youtu.be/PnDK17zP9BI

- article "Audio Latency for App Developers"
  at https://source.android.com/devices/audio/latency_app.html

[Anton Persson]

Hi Glenn,

I know this isn't probably on your table, so to speak, but wouldn't it be good if Google put a bit more pressure on companies like - hmm - Samsung, to support this better? I just implemented support for Samsung's "Professional Audio SDK" in my app (vuKNOB on play store) and noticed that the latency using that is much better on a Galaxy S6 than what I get with the OpenSL ES APIs on the same device.  So the platform on that device clearly can achieve fair latency but it's not available using standard Android APIs.

  Best Regards

     Anton

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at http://groups.google.com/group/android-ndk.
For more options, visit https://groups.google.com/d/optout.

[Felix Homann]

Hi Anton,

Am Freitag, 10. Juli 2015 15:34:38 UTC+2 schrieb pltxtra:

but wouldn't it be good if Google put a bit more pressure on companies like - hmm - Samsung, to support this better? I [...] noticed that the latency using that [i.e. Samsung's "Professional Audio SDK"] is much better on a Galaxy S6 than what I get with the OpenSL ES APIs on the same device.

How should they put pressure on Samsung? Samsung's "Professional Audio SDK"

• *does* provide low latency audio (at least better latency, haven't tested; no Samsung devices here except for Galaxy Nexus and Nexus 10),
• it *does* provide familiar and easy(!) to use APIs (JACK)
• it *does* provide routing between different apps through JACK APIs (See the sticky announcement of Peter Brinkmann's Patchfield on top of this group? It never gained any momentum. I'm quite sure this is partially because it doesn't make any sense without low latency and multi channel audio.)

In all of these areas stock Android is miles behind!

Google itself has to change audio on Android. Not by putting pressure on others but by just delivering! And I don't think a little tweaking can do. Android needs a totally different, maybe additional, set of APIs for pro audio. JACK would be the first choice as Samsung have correctly realized. (I guess Google decline to use it because it's LGPL and not BSD or Apache). 

OpenSL ES on the other hand is a major PITA. So much boiler plate code for even the simplest things! So limited in what it can actually do! See e.g. the upcoming multichannel audio support in Android M - it is only available in the JAVA APIs while it is most urgently needed for *professional* audio that which have a need for low latency audio. That's insane!

Than there's there's still this limited and unpredictable support fur USB audio interfaces. 

1. Since every Android device uses a different kernel version, support for USB audio is different on all of them: Some audio interfaces work on some Android devices but not on others.
2. If your device is supported by the kernel but has some USB mixer controls that device is generally unusable since those controls are not exposed in any Android API.

IMO, Android versions should specify the kernel version they use, too, so that device support - not only for audio devices - is unified! 

I personally have given up on low latency audio on Android. I still do audio on Android a lot but nothing that requires multi channel or low latency audio anymore. Those applications that need low latency have been moved to a CubieTruck running Debian that I can control via OSC from my Android devices. That's much simpler and much better *for me*.

BTW, have you tried to use a USB audio interface on an Android device? In most cases you have to choose between charging the Android device or using the audio interface. That's totally unsuitable for *professional* audio! (*Some* Android devices can be charged via  USB-OTG cables with power supply while using *some* audio interfaces but again it's totally unpredictable which combinations work and which do not.) This might be solved with USB 3.1 in future devices though.

Glenn is a very gentle, friendly and helpful person. I'm pretty sure he feels my (our?) pain and would surely want things to change rather sooner than later. I am very thankful for all he has already done! Unfortunately though, Google still does not seem to see the need for professional audio apps on Android. That's a pity!

[Anton Persson]

Hi Felix,

I fully agree with you. I just don't see why Samsung, who offers both APIs, can't deliver small buffers both through JACK and OpenSL.

They are both there already...

    /Anton

--

[Brad Justice]

I would not expect audio latency suitable for pro audio to come directly from Google. Their goal is to provide a generic solution suitable for a wide variety of OEMs. For this market the definition "low latency" is an order of magnitude greater than that desired for the pro audio market.

Even if you look at successful pro audio platforms such as the Mac, they do not support this sort of latency out of the box. They work because they are highly configurable in the field in both hardware and software.

In order to support very low audio latency you need to look beyond the audio stack. Any piece of hardware, any driver, any OS feature integrated into the device can cause delays that result in dropouts. In the absence of field configuration, this needs to be addressed by an OEM who cares about a niche market, as it is very device specific.

Unfortunately it does not seem to me that Samsung is that OEM. Per statements by Samsung at their San Francisco developers conferences, they are running Jack with two 512 sample buffers. At their 48K sample rate this pushes buffer latency above the 20 ms line. Even at that, my testing on a Note 3 indicated a significant underrun issue.

Brad Justice
http://proaudioandroid.blogspot.com/

On Sunday, March 1, 2015 at 6:53:11 PM UTC-8, evan...@gmail.com wrote: