AAudio documentation and headers for O DP1

[Glenn Kasten]

The attached zips contain preliminary documentation and headers for the AAudio API announced on March 21. These correspond to O Developer Preview 1.

I apologize for the delay. Obviously we have lost some review time already, so the sooner we can get your comments and suggestions the better. Your feedback will enable us to update the API before it is frozen. 

Please reply to this post with your comments.

Thank you,
Glenn

[Glenn Kasten]

NDK  r15 Beta 1 with AAudo support is available at:
https://developer.android.com/ndk/downloads/index.html

Online documentation for AAudio is published at:
https://developer.android.com/ndk/guides/audio/aaudio/aaudio.html

Please reply with your suggestions, thank you.

[Felix Homann]

There seems to be a callback interface, see AAudioStreamBuilder_setDataCallback(), which might be closer to how Jack et. al. work. I can't find anything about it in the docs, and neither in the code samples. Could you please elaborate on that?

[Gábor Szántó]

Thank you for listening to the pro audio developer community and offering an easier alternative to the OpenSL ES API. 

About OpenSL ES: The main problem with OpenSL ES was that developers had to write long inflated spaghetti code even for simple tasks. OpenSL ES is not to blame for audio latency or other issues, it's "just a language" to talk with certain features of the audio stack.

I read all the available documentation and went through the example code. Please find my feedback and questions below.

1. API DESIGN

AAudio seems simpler than OpenSL ES and that is very welcome. I really like to call simple C functions. I also like the fact that AAudio doesn't force us to learn new abstract concepts, such as graphs and flows. Awesome!

2. ROUTING

I understand that audio routing is not planned to be a part of AAudio now, however it offers to connect to specific audio devices, so some kind of routing is already part of the API.

Using AudioManager's getDevices() in Java to discover audio devices, then going to native code using JNI just to select that device goes against the simplicity goal of AAudio.

Please consider implementing at least the same audio routing and device discovery features available in Java, so developers don't need to write Java/JNI code for that.

3. EXCLUSIVE SHARING MODE

The exclusive audio device sharing mode is interesting. 

Question: Does it mean some kind of "pass-through" access in the Android media server, connecting the user app with the audio HAL in a direct way? So developers can directly connect audio input and output in the audio processing thread, eliminating the extra buffering required by the separate audio input and output threads of the current audio stack?

4. SAMPLE RATE, BUFFER SIZE, FORMAT

It's great finally having native APIs to get the native sample rate and buffer size of the audio device without Java. I understand that the native sample rate and buffer size is the best to talk with an audio device to achieve low latency audio.

Question: If a developer chooses a different sample rate or buffer size, will the API try to configure/restart the HAL with those settings? Or will it "just" handle sample rate conversion and buffering before the audio data reaches the HAL, which will always run at its native sample rate and buffer size?

5. THREAD PRIORITY

Question: Since high priority threads are paramount to prevent audio dropouts, why is there a possibility to run a non-high priority thread at all? Perhaps some of the APIs could return with errors in this case.

6. STREAM BUFFER FRAME NUMBERS

There are APIs getting frame numbers in internal buffers, such as AAudioStream_getBufferSizeInFrames(). Nice!

Question: In exclusive mode, are they directly connected to the similar properties of the HAL, or are they connected to some kind of internal loop/buffering mechanism (in the media server)?

7. TUNING FRAME SIZE

Question: Since AAudioStream_getFramesPerBurst() provides the optimal, native number of frames of the audio device for the lowest latency, what's the purpose of "tuning" the frame size as shown in the example app?

8. TIMESTAMPS

Question: Since the reported latency of the media server can not be trusted as audio HALs may report false numbers, how does AAudioStream_getTimestamp() know the timestamp?

9. DATA CALLBACK

Question: Is AAudioStreamBuilder_setDataCallback() some kind of convenience feature, for the case when a developer doesn't want to create a custom audio processing loop (as shown in the example app)?

[Phil Burk]

On Wednesday, April 12, 2017 at 11:54:54 AM UTC-7, Felix Homann wrote:

There seems to be a callback interface, see AAudioStreamBuilder_setDataCallback(), which might be closer to how Jack et. al. work. I can't find anything about it in the docs, and neither in the code samples. Could you please elaborate on that?

Thank you for catching this. It was an error in the documentation, and is now fixed in the online version.

Phil Burk

[Phil Burk]

Hello Gábor,

Thanks for the feedback. Responses below...

On Thursday, April 13, 2017 at 7:28:38 AM UTC-7, Gábor Szántó wrote:

1. API DESIGN

AAudio seems simpler than OpenSL ES and that is very welcome. I really like to call simple C functions. I also like the fact that AAudio doesn't force us to learn new abstract concepts, such as graphs and flows. Awesome!

We are glad you like it. If you have specific suggestions for improvements, please let us know ASAP.

2. ROUTING

I understand that audio routing is not planned to be a part of AAudio now, however it offers to connect to specific audio devices, so some kind of routing is already part of the API.

Using AudioManager's getDevices() in Java to discover audio devices, then going to native code using JNI just to select that device goes against the simplicity goal of AAudio.

Please consider implementing at least the same audio routing and device discovery features available in Java, so developers don't need to write Java/JNI code for that.

Thank you. We will consider native device enumeration and selection for a future release.

Note that there is no plan for AAudio to support “auto-routing” . Auto-routing can occur when the “primary” device changes. This could happen, for example, when the user connects a Bluetooth headset. This automatic device switch can cause an irreversible increase in latency that the app is unaware of. The plan for AAudio is to disconnect the stream when auto-routing would normally occur. Then the app has the choice of reconnecting to the primary output device with new settings for latency, etc. What do you think about not supporting auto-routing in AAudio?

3. EXCLUSIVE SHARING MODE

The exclusive audio device sharing mode is interesting. 

Question: Does it mean some kind of "pass-through" access in the Android media server, connecting the user app with the audio HAL in a direct way? So developers can directly connect audio input and output in the audio processing thread, eliminating the extra buffering required by the separate audio input and output threads of the current audio stack?

That is the idea. We are thinking about providing a more direct path between the application and a HAL level buffer, bypassing the shared mixer stage. Note that this is not implemented in DP1. But we are very interested in feedback on whether this would be useful.

 

4. SAMPLE RATE, BUFFER SIZE, FORMAT

It's great finally having native APIs to get the native sample rate and buffer size of the audio device without Java. I understand that the native sample rate and buffer size is the best to talk with an audio device to achieve low latency audio.

Question: If a developer chooses a different sample rate or buffer size, will the API try to configure/restart the HAL with those settings? Or will it "just" handle sample rate conversion and buffering before the audio data reaches the HAL, which will always run at its native sample rate and buffer size?

Ideally the app will use the default AAUDIO_UNSPECFIED value for sample rate, etc. If the app selects a rate then it will try to configure the stream with that rate, which may fail. AAudio, by itself, will not do sample rate conversion because it adds latency and complexity. It is possible, however, that the layers beneath AAudio may do sample rate conversion. Some drivers, for example, do sample rate conversion. It is just math and can be done more easily at the application level where the developer can control it.
 

5. THREAD PRIORITY

Question: Since high priority threads are paramount to prevent audio dropouts, why is there a possibility to run a non-high priority thread at all? Perhaps some of the APIs could return with errors in this case.

 

There are several ways to reduce the probability of dropouts: use high priority threads, use big buffers, reduce worst-case CPU usage per buffer, etc. Some apps do not need low latency and can live with blocking writes from a normal thread.

 

6. STREAM BUFFER FRAME NUMBERS

There are APIs getting frame numbers in internal buffers, such as AAudioStream_getBufferSizeInFrames(). Nice!

Question: In exclusive mode, are they directly connected to the similar properties of the HAL, or are they connected to some kind of internal loop/buffering mechanism (in the media server)?

This refers to the buffer at the top of AAudio that the app is writing into directly. There may be other buffers below this buffer depending on the data path.

7. TUNING FRAME SIZE

Question: Since AAudioStream_getFramesPerBurst() provides the optimal, native number of frames of the audio device for the lowest latency, what's the purpose of "tuning" the frame size as shown in the example app?

“burst” != “buffer”

The tuning process involves setting the size of the larger buffer. For output, the app can write arbitrary number of frames. But the reader will typically read in a fixed size burst, which is returned by AAudioStream_getFramesPerBurst(). It is best for the apps to write one burst at a time. There can be room for multiple bursts in the large buffer.

If you are running with a moderate load then you may not be able to get by running “single buffered”, ie. one burst in the buffer. If you want to be “double buffered” then call AAudioStream_setBufferSizeInFrames(stream, 2 * framesPerBurst).

There is a diagram and an explanation here:
https://developer.android.com/ndk/guides/audio/aaudio/aaudio.html#tuning-buffers

 

8. TIMESTAMPS

Question: Since the reported latency of the media server can not be trusted as audio HALs may report false numbers, how does AAudioStream_getTimestamp() know the timestamp?

It is a serious problem when HALs don’t implement timestamps. We have the Compatibility Test Suite (CTS) to catch this kind of error, but there are still times when bugs slip through. If you see any devices that don’t implement timestamps well, please let us know by filing a bug at b.android.com. The creative audio community has been calling attention to the need for low latency and reliable CPU performance. This helps to improve awareness (and results though it seems slow!). Let’s do the same thing here too, as accurate timestamps are also critical.

9. DATA CALLBACK

Question: Is AAudioStreamBuilder_setDataCallback() some kind of convenience feature, for the case when a developer doesn't want to create a custom audio processing loop (as shown in the example app)?

See earlier answer. Note that the API has not settled. We may have breaking changes from DP1 to the next release.

 

Let us know if you have more feedback or questions.

Phil Burk

[Felix Homann]

Am Freitag, 14. April 2017 01:56:42 UTC+2 schrieb Phil Burk:

Thank you for catching this. It was an error in the documentation, and is now fixed in the online version.

Phil Burk

That's really sad to hear.

Being used to callback driven audio processing (Jack, Core Audio, Portaudio) this is a real bummer. Together with your answers to Gábor, specifically #5, I start wondering what your goal with AAudio really is. I don't understand it. What problems does it solve besides being just a bit easier than OpenSL ES? You still have so many things to take care of yourself that are provided by established "pro" audio APIs. You still have to do essential things in Java.  I'm totally disappointed.

So, let me ask a very bold questoin: If a) latency isn't better than OpenSL ES b) thread management/priorities still is up to the user c) AAudio will only be available for > O devices then why bother using or even invest time trying it? What would be the benefit?

Regards,

Felix

[Gábor Szántó]

2. ROUTING

I understand that audio routing is not planned to be a part of AAudio now, however it offers to connect to specific audio devices, so some kind of routing is already part of the API.

Using AudioManager's getDevices() in Java to discover audio devices, then going to native code using JNI just to select that device goes against the simplicity goal of AAudio.

Please consider implementing at least the same audio routing and device discovery features available in Java, so developers don't need to write Java/JNI code for that.

Thank you. We will consider native device enumeration and selection for a future release.

Note that there is no plan for AAudio to support “auto-routing” . Auto-routing can occur when the “primary” device changes. This could happen, for example, when the user connects a Bluetooth headset. This automatic device switch can cause an irreversible increase in latency that the app is unaware of. The plan for AAudio is to disconnect the stream when auto-routing would normally occur. Then the app has the choice of reconnecting to the primary output device with new settings for latency, etc. What do you think about not supporting auto-routing in AAudio?

There is no need for auto-routing, i fully agree.

Device enumeration and one callback when the available set of devices are changing, that would be very important even for the very first release.

They don't need to be complex. Device enumeration needs to return some description/name string and an identifier per device. The callback needs to be just firing when a device is connected or disconnected.

 

3. EXCLUSIVE SHARING MODE

The exclusive audio device sharing mode is interesting. 

Question: Does it mean some kind of "pass-through" access in the Android media server, connecting the user app with the audio HAL in a direct way? So developers can directly connect audio input and output in the audio processing thread, eliminating the extra buffering required by the separate audio input and output threads of the current audio stack?

That is the idea. We are thinking about providing a more direct path between the application and a HAL level buffer, bypassing the shared mixer stage. Note that this is not implemented in DP1. But we are very interested in feedback on whether this would be useful.

This would be definitely useful!

 

 

4. SAMPLE RATE, BUFFER SIZE, FORMAT

It's great finally having native APIs to get the native sample rate and buffer size of the audio device without Java. I understand that the native sample rate and buffer size is the best to talk with an audio device to achieve low latency audio.

Question: If a developer chooses a different sample rate or buffer size, will the API try to configure/restart the HAL with those settings? Or will it "just" handle sample rate conversion and buffering before the audio data reaches the HAL, which will always run at its native sample rate and buffer size?

Ideally the app will use the default AAUDIO_UNSPECFIED value for sample rate, etc. If the app selects a rate then it will try to configure the stream with that rate, which may fail. AAudio, by itself, will not do sample rate conversion because it adds latency and complexity. It is possible, however, that the layers beneath AAudio may do sample rate conversion. Some drivers, for example, do sample rate conversion. It is just math and can be done more easily at the application level where the developer can control it.

Good, please keep it this way.

  

5. THREAD PRIORITY

Question: Since high priority threads are paramount to prevent audio dropouts, why is there a possibility to run a non-high priority thread at all? Perhaps some of the APIs could return with errors in this case.

 

There are several ways to reduce the probability of dropouts: use high priority threads, use big buffers, reduce worst-case CPU usage per buffer, etc. Some apps do not need low latency and can live with blocking writes from a normal thread.

Apps not needing low latency with blocking writes from a normal thread would not be very interested in AAudio i think.

 

7. TUNING FRAME SIZE

Question: Since AAudioStream_getFramesPerBurst() provides the optimal, native number of frames of the audio device for the lowest latency, what's the purpose of "tuning" the frame size as shown in the example app?

“burst” != “buffer”

The tuning process involves setting the size of the larger buffer. For output, the app can write arbitrary number of frames. But the reader will typically read in a fixed size burst, which is returned by AAudioStream_getFramesPerBurst(). It is best for the apps to write one burst at a time. There can be room for multiple bursts in the large buffer.

If you are running with a moderate load then you may not be able to get by running “single buffered”, ie. one burst in the buffer. If you want to be “double buffered” then call AAudioStream_setBufferSizeInFrames(stream, 2 * framesPerBurst).

There is a diagram and an explanation here:
https://developer.android.com/ndk/guides/audio/aaudio/aaudio.html#tuning-buffers

 

It would be great if AAudio would not do additional buffering, and buffer sizes would be 1-1 to the HAL's buffer size. I don't really get this tuning thing.

In exclusive mode the app would just directly transfer to/from the HAL's buffer. 

In shared mode the media server would ask each AAudio app to do it's thing, then mix them (plus the output of AudioFlinger) directly into the HAL's buffer. Certainly, an AAudio app would introduce dropouts if the cumulated processing time of every AAudio app's audio callback is too long. But this is the case with iOS as well, and nobody complain.

[Phil Burk]

Hello Felix,

Please note that the API is not final. There will be changes between DP1 and the next Beta.

Also note that if you have a high priority thread then you can always just loop and call your own callback function. It is slightly more work but equivalent in the end. 

// pseudo-code callback loop

void callbackLoop() {

    int32_t result = 0;

    while(result >= 0) {

        typicalCallback(this, callbackBuffer, framesPerBurst);

        result = AAudioStream_write(stream, callbackBuffer, framesPerBurst, timestampNanos);

    }

}

> So, let me ask a very bold question: If a) latency isn't better than OpenSL ES 

We have plans for improving latency. Creating a new API gives us a platform where it is easier to make those kinds of improvements.

> b) thread management/priorities still is up to the user

AAudio provides the high priority thread and configures it with an optimal scheduler. The same would be true for a callback.

> c) AAudio will only be available for > O devices then why bother using or even invest time trying it? What would be the benefit?

Because if you give us feedback now then it can affect the API. Once the API is frozen, then we are stuck with it.

So THANK YOU for this feedback. It is very helpful. I  do not make promises about future releases. But please know we that do listen very carefully to feedback and it does make a difference. 

Phil Burk

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk+unsubscribe@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/8650badc-2132-49d8-bfdb-6c2bfe98ab6d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Phil Burk]

Hello Gábor,

Thanks for the comments.

On Fri, Apr 14, 2017 at 5:48 AM, Gábor Szántó <sza...@gmail.com> wrote:

It would be great if AAudio would not do additional buffering, and buffer sizes would be 1-1 to the HAL's buffer size. I don't really get this tuning thing.

 

You can tune the buffer size to be "single buffered", as you describe, by doing this:

AAudioStream_setBufferSizeInFrames(stream, AAudioStream_getFramesPerBurst(stream));

 

That will give you the lowest possible latency. But stuff happens and glitches can occur, as you know. In that case the app can choose to keep the low latency and glitch occasionally, OR bump the latency to "double buffered" and probably not glitch. By tuning, an app can get the lowest latency possible without glitching.

Phil Burk

[Gábor Szántó]

I see two use-cases here:

1. Low latency audio requirements, solvable by AAudioStream_setBufferSizeInFrames(stream, AAudioStream_getFramesPerBurst(stream));

2. Low latency audio is not a requirement. Tuning?

Regarding tuning i see the following problems:

- In order to get a rock-solid buffer size, during the tuning process, the user application must set up it's audio chain to operate at maximum CPU load (enable all effects and features). Plus all synchronization problems need to be existing too (if the developer was not careful about mutexes, locks and stuff). I'm not sure if most developers will do this extra work.

- If the audio processing doesn't happen on a high priority thread, the tuning process may not catch every scheduling problem in that short time when it runs. Such as the scenario when the app goes into background.

- On other platforms developers are simply setting larger buffer sizes for this. 4096 is a popular number on iOS for example. Much less work and code to do!

[Felix Homann]

Hello Phil,

Am Freitag, 14. April 2017 18:55:44 UTC+2 schrieb Phil Burk:

Hello Felix,

Please note that the API is not final. There will be changes between DP1 and the next Beta.

OK, without mentioning it above I was hoping for features in the callback driven API not present in the rest of the API.

Most prominently, I miss bidirectional streams like the ones Portaudio provides and then a callback interface for keeping input and output in sync. Very much the way Portaudio works. Very much the way Jack works.

BTW, I just realized that you're actually the Phil Burk of Portaudio fame so I guess you're much more familiar with Portaudio than I am ;-) On the other hand, I'm even more puzzled now why you have not just settled on using a well established API like Portaudio as Android's audio API (I still would have preferred JACK for its interconnection capabilities.) 

Regards,

Felix

[Phil Burk]

Hello Gábor,

On Sun, Apr 16, 2017 at 12:17 AM, Gábor Szántó <sza...@gmail.com> wrote:

I see two use-cases here:

1. Low latency audio requirements, solvable by AAudioStream_setBufferSizeInFrames(stream, AAudioStream_getFramesPerBurst(stream));

Single buffering may give the lowest latency. But on some systems you might need more double or triple buffering to avoid glitches. The current API allows you to control the amount of latency and the information you need to choose the optimal latency.

2. Low latency audio is not a requirement. Tuning?

I think the tuning is more for low latency apps.

 

Regarding tuning i see the following problems:

- In order to get a rock-solid buffer size, during the tuning process, the user application must set up it's audio chain to operate at maximum CPU load (enable all effects and features). Plus all synchronization problems need to be existing too (if the developer was not careful about mutexes, locks and stuff). I'm not sure if most developers will do this extra work.

Some developers will not do the work. But we want to allow developers who are willing to do the work to achieve optimal performance.

 

- If the audio processing doesn't happen on a high priority thread, the tuning process may not catch every scheduling problem in that short time when it runs. Such as the scenario when the app goes into background.

The tuning can be ongoing. If you ever glitch, even after several hours, you can bump the latency up. We want to give the app developers to do the smarts. Or they can use a wrapper that does it for them.

 

- On other platforms developers are simply setting larger buffer sizes for this. 4096 is a popular number on iOS for example. Much less work and code to do!

That's fine for high latency apps.

Phil Burk

[Phil Burk]

Hello Felix,

> Most prominently, I miss bidirectional streams like the ones Portaudio provides and then a callback interface for keeping input

> and output in sync. Very much the way Portaudio works. Very much the way Jack works.

Bidirectional streams callbacks are a big pain in PortAudio. They are often not implemented correctly. I believe that is a complicated problem that is best solved above the OS level. AAudio supports the use of multiple streams through non-blocking reads and writes.

> I'm even more puzzled now why you have not just settled on using a well established API like Portaudio as Android's audio API

PortAudio is meant to be a wrapper for host APIs.   Given that AAudio is very similar to PortAudio, it should be easy to implement PortAudio on top of AAudio. 

Also note that PortAudio is 20 years old. We had an opportunity to avoid some of the difficult parts of PortAudio. We do not, for example, pass pointers to structures in AAudio.

Also we did not want to be restricted by the PortAudio API. That was a problem with OpenSL ES.

Phil Burk

--

You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk+unsubscribe@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/fd4f7990-0ae8-41e4-ae88-f7ab81c51ff3%40googlegroups.com.

[Jeff McKnight]

On Thursday, April 13, 2017 at 5:07:13 PM UTC-7, Phil Burk wrote:

Hello Gábor,

Thanks for the feedback. Responses below...

On Thursday, April 13, 2017 at 7:28:38 AM UTC-7, Gábor Szántó wrote:

1. API DESIGN

AAudio seems simpler than OpenSL ES and that is very welcome. I really like to call simple C functions. I also like the fact that AAudio doesn't force us to learn new abstract concepts, such as graphs and flows. Awesome!

We are glad you like it. If you have specific suggestions for improvements, please let us know ASAP.

2. ROUTING

I understand that audio routing is not planned to be a part of AAudio now, however it offers to connect to specific audio devices, so some kind of routing is already part of the API.

Using AudioManager's getDevices() in Java to discover audio devices, then going to native code using JNI just to select that device goes against the simplicity goal of AAudio.

Please consider implementing at least the same audio routing and device discovery features available in Java, so developers don't need to write Java/JNI code for that.

Thank you. We will consider native device enumeration and selection for a future release.

Note that there is no plan for AAudio to support “auto-routing” . Auto-routing can occur when the “primary” device changes. This could happen, for example, when the user connects a Bluetooth headset. This automatic device switch can cause an irreversible increase in latency that the app is unaware of. The plan for AAudio is to disconnect the stream when auto-routing would normally occur. Then the app has the choice of reconnecting to the primary output device with new settings for latency, etc. What do you think about not supporting auto-routing in AAudio?

3. EXCLUSIVE SHARING MODE

The exclusive audio device sharing mode is interesting. 

Question: Does it mean some kind of "pass-through" access in the Android media server, connecting the user app with the audio HAL in a direct way? So developers can directly connect audio input and output in the audio processing thread, eliminating the extra buffering required by the separate audio input and output threads of the current audio stack?

That is the idea. We are thinking about providing a more direct path between the application and a HAL level buffer, bypassing the shared mixer stage. Note that this is not implemented in DP1. But we are very interested in feedback on whether this would be useful.

I agree with Gabor:  This would definitely be useful!

[le...@grame.fr]

About "Bidirectional streams" : in OS X this is directly solved at the CoreAudio level by providing the "audio devices aggregation" concept :https://support.apple.com/en-us/HT202000, either at user level by using the Audio Midi Setup" tool or pragmatically. This "device aggregation" thing in particular deals with clock synchronisation issues, by having the master device imposes its rate, and the slaves do some resampling. 

What is situation in Android world: do input/ouput audi devices possibly suffer for clock synchronisation issues? 

Will AAudio SDK offers developers an audio device aggregation service ? (which I think you should ((-; since it would solve the problem "once" at the proper level...)

Stéphane Letz

Le lundi 17 avril 2017 20:27:02 UTC+2, Phil Burk a écrit :

Hello Felix,

> Most prominently, I miss bidirectional streams like the ones Portaudio provides and then a callback interface for keeping input

> and output in sync. Very much the way Portaudio works. Very much the way Jack works.

Bidirectional streams callbacks are a big pain in PortAudio. They are often not implemented correctly. I believe that is a complicated problem that is best solved above the OS level. AAudio supports the use of multiple streams through non-blocking reads and writes.

> I'm even more puzzled now why you have not just settled on using a well established API like Portaudio as Android's audio API

PortAudio is meant to be a wrapper for host APIs.   Given that AAudio is very similar to PortAudio, it should be easy to implement PortAudio on top of AAudio. 

Also note that PortAudio is 20 years old. We had an opportunity to avoid some of the difficult parts of PortAudio. We do not, for example, pass pointers to structures in AAudio.

Also we did not want to be restricted by the PortAudio API. That was a problem with OpenSL ES.

Phil Burk

On Mon, Apr 17, 2017 at 10:58 AM, Felix Homann <showlabor....@gmail.com> wrote:

Hello Phil,

Am Freitag, 14. April 2017 18:55:44 UTC+2 schrieb Phil Burk:

Hello Felix,

Please note that the API is not final. There will be changes between DP1 and the next Beta.

OK, without mentioning it above I was hoping for features in the callback driven API not present in the rest of the API.

Most prominently, I miss bidirectional streams like the ones Portaudio provides and then a callback interface for keeping input and output in sync. Very much the way Portaudio works. Very much the way Jack works.

BTW, I just realized that you're actually the Phil Burk of Portaudio fame so I guess you're much more familiar with Portaudio than I am ;-) On the other hand, I'm even more puzzled now why you have not just settled on using a well established API like Portaudio as Android's audio API (I still would have preferred JACK for its interconnection capabilities.) 

Regards,

Felix

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.

[Phil Burk]

Hello Stéphane,

Good to hear from you.

I want to make synchronization between multiple streams possible. But we need to focus on providing the minimum functionality that is needed to implement this above the OS. There are very big advantages to adding features above the OS.

1. Multiple implementations possible. We don't have to "get it right".
2. Fixes can be delivered in days instead of months or years.
3. Developers do not have to wait for us.
4. The are many more developers outside our Android Audio group than inside.

So the question is, what is the minimum needed from the OS to implement this?

If you had a callback from a "master" stream, and a non-blocking read/write on other streams, then what else would you need to implement a stream synchronization feature?

Phil Burk

[Gábor Szántó]

I fully support Phil. Let's keep it at minimum, then let third parties (like us) implement it for developers!

[Felix Homann]

Am Donnerstag, 4. Mai 2017 02:48:11 UTC+2 schrieb Phil Burk:

 There are very big advantages to adding features above the OS.

1. Multiple implementations possible. We don't have to "get it right".
2. Fixes can be delivered in days instead of months or years.
3. Developers do not have to wait for us.
4. The are many more developers outside our Android Audio group than inside.

Hmm, I guess you just made a case for granting access to ALSA APIs on Android devices ;-) That would be great!

[Achim Leubner]

There has been a recommendation not to "use more than 20% of your callback budget to generate data" with the addition "Higher percentage possible on some devices" (Google I/O 2016/Android high-performance audio). It would be great if there were a way to determine the concrete percentage for a given device. 

Background: I wrote virtual church organ apps. They could make use of thousands of simultaneously active voices. So, it's interesting to be able to determine how much processing power is sustainably available. It would also be nice to use multiple high-prio audio generation threads if that's sustainable.

[Phil Burk]

Hello Felix,

 Nice try! ;-)   The OS should be thin. But not too thin. Some primary goals for an OS are hardware abstraction, resource management, and security.  ALSA is a little too close to the hardware, drivers need to be shared, and there are security issues in some drivers. But my ultimate goal is to make the audio framework seem like a short wire between the app and the HALs.

Phil Burk

[Phil Burk]

Hello,

On Fri, May 5, 2017 at 8:31 AM, 'Achim Leubner' via android-ndk <andro...@googlegroups.com> wrote:

There has been a recommendation not to "use more than 20% of your callback budget to generate data" with the addition "Higher percentage possible on some devices" (Google I/O 2016/Android high-performance audio). It would be great if there were a way to determine the concrete percentage for a given device. 

Using 20% of the CPU time is safe because then you have lots of headroom if there is a scheduling delay. But note that if you are only using 20% of the CPU then the CPU Governor may decide you don't need such a fast CPU. So it will lower the CPU clock frequency.

Your code might then be using 30% of the slower CPU. So you drop the number of voices to get back to 20%.

Then the CPU governor will decide you don't need that much CPU and drops the CPU clock frequency.

This loop can repeat until you end up with the minimum clock speed and not many voices.

Different CPU governors behave differently. But here is an experiment you can do. Pseudocode follows:

int numVoices = 1;

do {

       int cpuLoad = synthesizeVoicesForFiveSeconds(numVoices);

       print cpuLoad;

       numVoices++;

} while (cpuLoad < 0.95);

What you may find is that, at first, the CPU load will be proportional to numVoices up to a certain percentage. This region is at the minimum clock frequency.

As you increase numVoices, you may eventually reach a region where cpuLoad will stay in a limited range, maybe 60-80% as numVoices increases. The range varies by device. This is where the governor is gradually increasing the clock frequency to compensate for your increased load. So they cancel out. You are in a kind of equilibrium.

You will eventually reach a region when load is again proportional to numVoices. This is at maximum clock frequency. You cannot sustain this frequency because thermal protection will kick in and the clock will drop in frequency, driving cpuLoad over 100% and you are now glitching.

In other words, it is complicated.  You also have to consider other tasks and their impact on the CPU load and on the CPU governor. And core migration, and kernel drivers....

So what to do?

You could increase your number of voices until you either have enough voices or you are pushing into the region of equilibrium. This will give you a decent number of voices. But if you are running at 70% and suddenly need twice as many voices then you might suddenly jump over 100%. The CPU governor can take a while to respond. In the meantime you will glitch.

One way to avoid this is to calculate a constant number of voices all the time, regardless of how many are actually sounding. The CPU governor will see a steady load and give you a steady clock frequency. Then when you need more voices you just turn up voices that were already being calculated. So no increase in load.

(Advance DSP note: IIR filters with silent input may eventually experience numeric underflow. So gate the sound off after your low pass filter.)

Phil Burk

[Gábor Szántó]

There is an even simpler way to max the CPU governor. It is a part of a free library you can download, but unfortunately it's not allowed to directly link to that here. Google "solving cpu frequency scaling for android".

[Achim Leubner]

I hope no one else from Google was reading that you recommend burning CPU time ;)

So, no "sustainable audio/background mode" in sight. As you wrote: it is complicated.

I will probably check out SuperpoweredCPU (http://superpowered.com/cpu-frequency-scaling-android) when I get around to it.

[Felix Homann]

Hi Phil,

Am Freitag, 5. Mai 2017 21:55:29 UTC+2 schrieb Phil Burk:

Hmm, I guess you just made a case for granting access to ALSA APIs on Android devices ;-) That would be great!

 Nice try! ;-)

:-) 

 

The OS should be thin. But not too thin. Some primary goals for an OS are hardware abstraction, resource management, and security.  ALSA is a little too close to the hardware, drivers need to be shared, and there are security issues in some drivers. But my ultimate goal is to make the audio framework seem like a short wire between the app and the HALs.

In principle, I do totally agree with you here.

The issue I have is that seemingly your goals for hardware abstraction do - in the short run - not include making lots USB audio devices usable at all ;-) So, as long as class compliant USB audio interfaces won't work with Android devices because either a) they sport too many channels for Android or b) you don't have access to UAC1/2 compliant mixer controls because there isn't any HAL for that I'd prefer having access to low-level ALSA any time, maybe at least for external devices.

Actually, all I want is to be able to use my software with at least one of my audio interfaces with at least one of my $300 to $600 Android devices at least almost as good as I can with a $40 Raspberry Pi. Pretty low expectations for 2017, I guess, but I can't see it coming anytime soon.

Sorry, if I'm already getting on your nerves. I'm going to silence my frustration here now, promised ;-)

Kind regards,

Felix

[Phil Burk]

Hi Felix,

On Mon, May 8, 2017 at 7:51 AM, Felix Homann <showlabor....@gmail.com> wrote:

The issue I have is that seemingly your goals for hardware abstraction do - in the short run - not include making lots USB audio devices usable at all ;-) So, as long as class compliant USB audio interfaces won't work with Android devices because either a) they sport too many channels for Android or b) you don't have access to UAC1/2 compliant mixer controls because there isn't any HAL for that I'd prefer having access to low-level ALSA any time, maybe at least for external devices.

I can understand why you would like direct ALSA driver access. There may be thousands of people who want that. But some ALSA drivers are sketchy. So security would be compromised for the other 1.4 billion users.

I feel your pain. But security trumps audio features.

If we  provide access to those low level ALSA features then it would have to be without allowing direct ALSA driver access.

Phil Burk

[Felix Homann]

Hi Phil,

Am Dienstag, 9. Mai 2017 00:45:59 UTC+2 schrieb Phil Burk:

[...] I feel your pain. But security trumps audio features.

thank you for keeping your patience with me :-)

Kind regards,

Felix

[Gábor Szántó]

Nexus 6P round-trip audio latency measurements:

Android N

built-in speaker/built-in mic: 33 ms

loopback dongle: 17 ms

Android O, DP4

built-in speaker/built-in mic: 54-38 ms, 46 ms avg

loopback dongle: 37-21 ms, 30 ms avg

Questions:

- Is this normal that we get higher audio latency with DP4?

- Is this normal that the latency fluctuates?

Measured with Superpowered Latency Measurement application, which doesn't use this new API, just the previous OpenSL practice.

[Glenn Kasten]

> - Is this normal that we get higher audio latency with DP4?

Thank you for reporting this. I will attempt to repeat your results and then update here.

> - Is this normal that the latency fluctuates?

It is normal for the continuous latency to vary by one or two ms from one measurement test run to the next,

however it should not vary as much as you have found.  I will attempt to repeat your results and then update here.