Custom audio source (alongside default one)

[Jerome Humbert]

Hi all,

I am trying to implement a custom audio source, that is the ability to inject some custom raw PCM audio data (obtained/generated in an opaque way) as a source for one or more audio tracks. The goal is to be able to select this custom audio source per track, so it should not replace the default audio source (microphone) but instead be an alternative, as some other tracks might still want to use the microphone in parallel, or even use multiple microphones.

Looking around it seems that the only approach documented so far is to create a custom audio device module (ADM), and inject it into the peer connection factory.

https://groups.google.com/forum/#!searchin/discuss-webrtc/custom$20audio$20source%7Csort:date/discuss-webrtc/O_W0BuBCfWM/TSoSILp0AwAJ

https://groups.google.com/forum/#!searchin/discuss-webrtc/custom$20audio$20source|sort:date/discuss-webrtc/WuVFzbd9uH4/wptfx-HCBwAJ

https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/audio_device/include/test_audio_device.h;bpv=1;bpt=1;l=31?originalUrl=https:%2F%2Fcs.chromium.org%2F

Unfortunately that solution doesn't work for us, since this replaces the default microphone-based ADM, so makes ALL tracks use that ADM instead of the default one. We need to be able to select per track which source to use.

I had originally thought that the webrtc::AudioSourceInterface would have been the way to go, in the same way that we currently have implemented custom video sources by implementing the webrtc::VideoTrackSourceInterface (custom video sources works well). However despite the documentation saying webrtc::AudioSourceInterface can be shared among multiple tracks, in practice there is no audio data manipulation in this interface, and in fact I am wondering what is the purpose of that interface in the first place if tracks ignore it and simply pull their audio data from the (unique) ADM configured on the peer connection?

So, is it currently impossible to have 2 different audio sources and have tracks use both at the same time? (some tracks one, some tracks the other)

The only way out I see is to implement a custom "dispatch" ADM which knows about all tracks and can dynamically connect them with the correct source. That seems like a lot of work, and I am surprised there is nothing else simpler to achieve that feature.

Thanks

[Pratim Mallick]

Hi Jerome

I am also looking to stream the data from microphone as well as from a local mp3 file over webrtc. Did you find any way?

[guest271314]

On Linux with PulseAudio you can do something like

$ pactl load-module module-remap-source \ master="$(pactl info | sed -En 's/Default Sink: (.*)/\1/p')" \ # set source, sink here, e.g., MP3 and microphone sources (files,streams) source_name=virtmic source_properties=device.description=Virtual_Microphone

which above sets master to default monitor device (which Chrome deliberately restricts support for capturing), where "Virtual_Microphone" will be

listed and set at navigator.mediaDevices.getUserMedia(), see Chromium does not support capture of monitor devices by default on the front-end https://github.com/guest271314/SpeechSynthesisRecorder/issues/17.

There is a PipeWire version at Screenshare with audio on Discord with Linux https://github.com/edisionnano/Screenshare-with-audio-on-Discord-with-Linux#messing-with-audio.

With Data Channel you can stream arbitrary data, raw PCM, Opus, MP3, etc.

Alternatively, get PCM, Opus, MP3, etc. data by any means any use MediaStreamTrackGenerator to one or more MediaStreamTrack e.g., again, at front-end with back-end being Native Messaging and parec https://github.com/guest271314/captureSystemAudio#web-accessible-resources-transferable-streams-media-capture-transform-breakout-box. 

Web Audio API MediaStreamAudioDestinationNode merges all connected audio nodes into a single stream. Perhaps the C++ Chromium implementations of those interfaces might provide solutions for what you are trying to achieve. 

[Joe]

I have a similar problem. I want to put two types of audio in a PeerConnection, so I init two PeerConnectionFactory and createAudioTrack respectively, and then addTrack in the PeerConnection. But the two audio tracks received at the other end are both the same type of audio.

After several years, is there a solution to this problem?

Thanks in advance.

guest271314 在 2022年1月29日 星期六上午9:38:07 [UTC+8] 的信中寫道：

[guest271314]

You can utilize WebRTC Data Channels, send the audio as ArrayBuffer's then reconstruct the audio on the other side using TypedArray and WebCodecs and Web Audio API and/or MediaStreamTrackGenerator if you need a MediaStreamTrack.

--
This list falls under the WebRTC Code of Conduct - https://webrtc.org/support/code-of-conduct.
---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/3H4vwWgNuQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/3492561e-1e20-4e48-acf4-38634973ef3fn%40googlegroups.com.