ADM and audio device selection

[Alexandre GOUAILLARD]

dear all,

we're trying to implement audio device selection in C++ code.

While implementing video capture device / app / window / desktop is relatively easy, we are struggling with audio and the intricacies of the audio device module, Voice Engine, etc.

We've tried to create an ADM outside of the peerconnection factory and pass it in, which allows us to list the input and output devices, but it looks like we would have to fix the input device once and for all for the Perrconnectionfactory, and hence all the peer connections created thereafter. 

ADM => PCF => PC => CreateAudioSource => MSTrack => AddTrack()

We were expecting to be able to actually create an ADM, and an audio source / track separately from the peerconnection object, and only involve the peerconnection when calling addTrack().

capturer => Source => Track => PC->AddTrack( Track )

We have looked into the chrome side of things and got even more confused, even though the agc_audio_stream seems to be the right capturer class. How it ends up in a peerconnection and how to use it is not clear to us.

https://codesearch.chromium.org/chromium/src/media/audio/agc_audio_stream.h

Has anybody done something like that, or could any googler give us hint(s) about the expected flow?

Regards,

Dr. Alex.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Henrik Andreasson]

We've tried to create an ADM outside of the peerconnection factory and pass it in, which allows us to list the input and output devices, but it looks like we would have to fix the input device once and for all for the Perrconnectionfactory, and hence all the peer connections created thereafter. 

ADM => PCF => PC => CreateAudioSource => MSTrack => AddTrack()

It is not clear to me what the actual problem is with this approach. Does it work but you would prefer an alternative (more clean or more inline with Chrome) solution or are there still

issues that you can't resolve? 

[Sergio Garcia Murillo]

Hi Henrik,

IMHO using an external ADM allows little flexibility, from my investigation (please, correct me if I am wrong) if you want to implement a track/device based audio processing you need to provide a dummy external ADM and override the AudioSourceInterface and track sinks to be able to get/fetch the audio data anyway.

Also, (again please correct if I am wrong) it seems there are two different ways of delivering data to the voe channel that work simultaneously, one via the adm RegisterAudioCallback(webrtc::AudioTransport*) and the other via AudioSourceInterfae::AddSink(webrtc::AudioTrackSinkInterface * sink).

Overall it is quite difficult (at least it has been for me) to understand the whole audio processing and IMHO it should be more similar to what chrome currently implements (which would allow device selection by default on the AudioSource), allowing to plug new input/output devices without needing to overwrite quite a lot of code. I also acknowledge that this change would require quite a huge refactor and is far from trivial, so just my 2 cents ;)

Best regards
Sergio

[Henrik Andreasson]

I understand your viewpoint Sergio, but given title of this thread "ADM and device selection"; are there any real issues with the

"external ADM" approach there? The transport mechanism/API is imho another (broader and more difficult) topic.

Also, please note that on most mobile platforms, device handling can be done completely outside the native WebRTC stack.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/1c366089-126f-2d41-e4a8-0497958f2440%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Alexandre GOUAILLARD]

hi henrik,

Thanks for answering.

sergio and I are working together on this issue, so our comments are not independent.

As noted in the first e-mail our goal is to "implement audio device selection in C++ code".

We originally thought ADM was the way to go, but we have come to realise that different webrtc consumer would implement different APIs in different design pattern to achieve it. To answer specifically your question, in the ADM => PCF => PC => CreateAudioSource => MSTrack => AddTrack() case, I dislike the fact that I have to decide what device to use once and for all at peerconnection factory time, and won't be able to change it from one peerconnection to the other (without creating a new adm and a new peerconnection). What I dislike about the chrome approach are the dependencies to other chrome internal packages, and the fact that I would have to reimplement OS dependent classes that, I believe, ADM is dealing with already,

Here is the user story version: as a C++ developper I would like to be able to capture/record the audio frames  from my webcam, my system audio (on windows), and one acquisition device and add them three as different audio tracks to a peerconnection, like i can do in a web application. 

Given the multiple ways to do it out there, Our question is what is the *recommended way* to do it. Of course the recommended way could be different depending on the scope: from libwebrtc stand alone vs chrome, or depending on the OS/platform: on desktop OSes vs mobile OSes. So let's say I'm interested in windows only as describe in the above user story, and that I would like to stay as close to standalone webrtc as possible.

I hope this clarifies.

thanks again.

Dr. Alex.

On Wed, Oct 25, 2017 at 7:01 PM, Henrik Andreasson <hen...@webrtc.org> wrote:

I understand your viewpoint Sergio, but given title of this thread "ADM and device selection"; are there any real issues with the

"external ADM" approach there? The transport mechanism/API is imho another (broader and more difficult) topic.

Also, please note that on most mobile platforms, device handling can be done completely outside the native WebRTC stack.

On Wed, Oct 25, 2017 at 11:05 AM, Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com> wrote:

On 25/10/2017 10:33, Henrik Andreasson wrote:

We've tried to create an ADM outside of the peerconnection factory and pass it in, which allows us to list the input and output devices, but it looks like we would have to fix the input device once and for all for the Perrconnectionfactory, and hence all the peer connections created thereafter. 

ADM => PCF => PC => CreateAudioSource => MSTrack => AddTrack()

It is not clear to me what the actual problem is with this approach. Does it work but you would prefer an alternative (more clean or more inline with Chrome) solution or are there still

issues that you can't resolve? 

Hi Henrik,

IMHO using an external ADM allows little flexibility, from my investigation (please, correct me if I am wrong) if you want to implement a track/device based audio processing you need to provide a dummy external ADM and override the AudioSourceInterface and track sinks to be able to get/fetch the audio data anyway.

Also, (again please correct if I am wrong) it seems there are two different ways of delivering data to the voe channel that work simultaneously, one via the adm RegisterAudioCallback(webrtc::AudioTransport*) and the other via AudioSourceInterfae::AddSink(webrtc::AudioTrackSinkInterface * sink).

Overall it is quite difficult (at least it has been for me) to understand the whole audio processing and IMHO it should be more similar to what chrome currently implements (which would allow device selection by default on the AudioSource), allowing to plug new input/output devices without needing to overwrite quite a lot of code. I also acknowledge that this change would require quite a huge refactor and is far from trivial, so just my 2 cents ;)

Best regards
Sergio

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/1c366089-126f-2d41-e4a8-0497958f2440%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAD2mqc7m9%3DXg_tjC5P4e3ruvawMGpZhRF%3DyN7iJGaWzMVCiVew%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Henrik Andreasson]

My own personal reply is Chrome has been the main driver for implementing the standard adding the support that you need.

Hence, a more natural and flexible implementation can be found in using the web based APIs focusing on desktop platforms.

For native WebRTC, these parts are not on pair with the web standard, and other means are needed instead. Also, the native

stack is more used on mobile platforms where device handling can be done independently and where a user normally has less

devices to choose from.

Hence, my current recommendation for native WebRTC is to use the external ADM path and accept that device handling will be less

flexible that way. Or alternatively, focus on web based APIs instead.

PS it is not clear to me what type of use case requires support for more than one active recording device.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAHgZEq7dzYU8oZnRTxtRtmgucp2V_h4YAksxOJ_RE0bwMh11Xw%40mail.gmail.com.

[Alexandre GOUAILLARD]

hi henrik,

Thank you again for your answer.

Here is the use case as presented before: "as a C++ developper I would like to be able to capture/record the audio frames  from my webcam/mike, my system audio (on windows), and one acquisition device (e.g. a digital input card from audio equipment) and add them three as different audio tracks to a peerConnection, like I can do in a web application".

[henrik] My own personal reply is Chrome has been the main driver for implementing the standard adding the support that you need. Hence, a more natural and flexible implementation can be found in using the web based APIs focusing on desktop platforms.

[alex] Fair enough. Can you just point us to the part that connects the audio_source in chrome to the VoE in webrtc? Now that the two webrtc trees (chrome and standalone) are in sync, life is good. We could not figure out how the actual audio frames were being pushed through.

[henrik] For native WebRTC, these parts are not on pair with the web standard, and other means are needed instead. Also, the native stack is more used on mobile platforms where device handling can be done independently and where a user normally has less devices to choose from. Hence, my current recommendation for native WebRTC is to use the external ADM path and accept that device handling will be less flexible that way.

[alex] Thank you for confirming that going through the adm is the preferred way for libwebrtc stand alone. Now, is there a way (we would have missed) to change the deviceId after the creation of an ADM (1), respectively a peerconnectifonfactory (2), respectively a peerconnection (3) instance.

thanks again.

best regards,

Dr. Alex.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAD2mqc7B98SL8EpmVba540bjw8mPzzXc36WUDLWDecy5cf2aTA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Kaiduan Xie]

Check the two files,

https://cs.chromium.org/chromium/src/remoting/protocol/webrtc_audio_module.h

https://cs.chromium.org/chromium/src/remoting/protocol/webrtc_audio_module.cc

/Kaiduan

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAHgZEq7wwJRFxE7sRhUgYv%2BDAZOK5pcpprsvHvFDcA67bjA5%3Dg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Founder of Goodstartsoft

https://www.goodstartsoft.com

[Sergio Garcia Murillo]

Yes, basically Chrome passes an empty/dummy audio device module to the peerconnection, and then override the LocalAudioSource to push data to the audio source sink set by the audio track.

BR
Sergio

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CACKRbQdpxGeq3RArSGPK%2BLoN5TUA2oDcPuyTj9vO7nnnFh0AhA%40mail.gmail.com.

[Kaiduan Xie]

This may also help,

https://cs.chromium.org/chromium/src/content/renderer/media/webrtc_audio_device_impl.h?dr=CSs

/Kaiduan

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CACKRbQdpxGeq3RArSGPK%2BLoN5TUA2oDcPuyTj9vO7nnnFh0AhA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/b234daa4-9a55-5c8b-6cb0-4b9e53c81adb%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Henrik Andreasson]

I don't have all the answers and the topic is complex. Using an external ADM in native WebRTC for desktop is not a complete replacement for the existing web-based implementation

parts. Instead, it makes it possible to get more control of the audio streams but in a non standardized way.

An injected ADM is connected and used by VoE here. Today, only default devices are utilized but it should be possible to switch device if the external ADM supports dynamic device switching.

The existing ADM implementation in native WebRTC for desktop serves mainly as an example. The audio parts in Chrome are better maintained and tested.

On Wed, Oct 25, 2017 at 10:44 PM, Kaiduan Xie <kaid...@gmail.com> wrote:

This may also help,

https://cs.chromium.org/chromium/src/content/renderer/media/webrtc_audio_device_impl.h?dr=CSs

/Kaiduan

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CACKRbQcz477NiH1zNgC_OUeiLDJ_gRqq8pAVz7RU0%2BnP1Bvf5Q%40mail.gmail.com.