Custom global Audio Effect in Android JB (AudioFlinger)

[Omri]

Hi,

I have a requirement to inject a custom global audio effect in Android (at the core system-level) to the audio pipeline. The requirement is that every buffer of audio that will output from the speaker, ear-piece or a2dp routes will be processed with a custom effect.

I'll be building a custom ROM which includes this feature, implemented as a custom *.so library (software only, no hardware). The library expects 16bit signed stereo PCM audio buffers and output the same.

The library is software based so there will be no need for drivers & porting. Also, I started looking into the AudioFlinger code in android-4.1.2_r1 (maguro), but I'm open to check out newer branches too.

I'm familiar with Android Core sources and been looking into the media server implementation, and AudioFlinger specifically.

I found the lines of code that the AudioFlinger threads writes the PCM buffers to the HAL (to the output sink). Please correct me if I'm wrong here:

1) The Mixer (inherits from PlaybackThread) & FastMixer (Thread, for low latency) threads are the ones responsible for mixing the buffers from the currently played tracks and passing them to the HAL, for the actual sound to come out of the speaker or ear piece.

2) The DuplicatingThread & DirectoutputThread (both inherit from PlaybackThread) are somehow related to A2DP (BT). I'll be happy for some info about them.

Similar to this guide.

With the lines of code above (in 1), I was able to add my changes so that the audio effect was global to speaker output (did not try ear piece yet) by calling my libraries methods on the mixed audio buffer prior to writing it into the sink. Let's assume speaker route is enough for now.

In order to use a better design approach I've been looking into the native API of audio effects in: hardware/​libhardware/​include/​hardware/​audio_effect.h and the source code of the built-in audio effects provided with AOSP (visualizer, virtualizer etc.), including the audio_effects.conf file that maps the libraries of the native effects to uuid(s).

Wrapping my effect library this way would be perfect (it actually maps 1:1 with the API in the above header), but some things are not clear to me here. It seems like this API was deigned to eventually expose these effects to the Java Layer (Apps), to be used by apps (and per-track - not sure about this).

I did not find (yet?) a way to configure, in code or in the conf file, an audio effect library to load with the initialization of AudioFlinger or the playback threads.

I'll be happy to hear if anyone had experience with this low level audio effects API (for custom needs) and if this can be used for my needs.

Thanks.

Omri

[Glenn Kasten]

Audio effect libraries are configured in target device's file

/system/etc/audio_effects.conf

This file specifies the .so pathnames .so to load,

the mapping from effect name to .so and UUID, etc.

See http://source.android.com/devices/audio_implement.html

section "More information" at very end.

There has been some prior discussion of this topic on

android-porting, see for example http://groups.google.com/group/android-porting/browse_thread/thread/1a7c05e624bdf3c6

and I believe there were others that are more recent.

If you are seeking a simpler implementation strategy, you could also modify

the audio HAL's write() function, but then the rest of the system

would not be aware of the effect and could not control the effect

using the standard effect APIs.

Note that if your device is a traditional "phone" then telephony audio often

bypasses the app processor and is inaccessible by framework or HAL.

[Omri]

Hi Glenn & Thanks for the references,

Although I'll be doing my development on the Galaxy Nexus, the main idea is to be able to integrate this effect in custom ROMs on custom devices, by supplying the appropriate libraries and config files.

I'm mainly interested in applying my effect on "regular" playback audio (music, not voice, ringtones etc.), i.e: Audio files being played & Audio coming from Video playback. "Fast path" does not apply audio effects, and I'm good with that (latency is not a problem for my use case).

I've been doing some homework - reading various posts, source files and searching the web about building custom Audio effects.

I still have some questions which I'll be glad if you can clarify:

1) The lowest level of an audio effect in a native shared library implementing the audio_effect.h API and located at "/system/lib/soundfx". AudioFlinger will be aware of this audio effect if it's configured correctly in "/system/etc/audio_effects.conf"
To expose it to the Java layer (apps) there are JNI modules wrapped by the "android.media.audiofx" Java namespace, specific to every built-in effect (Equalizer, Bassboost. I think I'm missing a middle layer here (C++), between the native effect lib and the JNI. Am I?

Can the JNI, Java and other code also be built as an independent module (*.so)? (so it will be easy to integrate it to other ROMs).

2) The main purpose is that my effect will be applied GLOBALLY. From what it seems like there is no way to configure this from the config file or from the effect\library descriptors. For effects to apply to audio, they should be assigned explicitly to an  audio session, usually from Java apps. There is an exception here which is session 0 (the output mix thread, the only thread writing to the hardware in AudioFlinger, correct?), the mixer which can apply effects on the audio about to be written to the hardware - This will affect audio coming from all audio sessions played on the device. So in order to achieve this I'll have to supply a Java app that will assign the effect to session 0. But this is currently deprecated (although it's still enabled).  What is the best/recommended way for achieving effect global-ness?

 

3) I still cannot pin point the difference between an INSERT and an AUXILIARY effect. Although I have a feeling an INSERT effect is what I need, In short - What's the difference?

Omri

[|0xD34D|]

I'm working on something very similar, on a custom ROM as well, where I would like to apply a particular effect on all sound that is sent to the output.  If I use AUDIO_SESSION_OUTPUT_MIX for the sessionId I can see, via some debug log output, that the effect is receiving data from that stream.  However, when playing games or viewing a video on YouTube I get no data coming into the effect.  Now if I change the sessionId for the effect to AUDIO_SESSION_OUTPUT_STAGE I get the opposite results.  I no longer receive data when music is played but I am now able to get the audio from YouTube and the games.  If I could somehow get all audio I would be good to go.

I'm currently in the same boat you are trying to make sense of how the AudioFlinger handles the audio and effects.  Have you managed to make any headway with this since you last posted to this group?

I'm not sure if there is some magical combination of session ID and effect flags that will get the desired results or if there some work to be done inside the AudioFlinger to make this happen.

[Omri]

Are you using the Java API or the native audio effect API in AOSP?