Accessing PeerConnection Audio Stream (transcode, stream, read/write disk)

[Ricky B.]

Hello,

I would like to code an application which accesses the audio stream of a peer connection (using native c++).

On the side of the application I do not want to use a local audio device but use the incoming audio to transcode/record to disk/stream it elsewhere where the outgoing audio is read from disk/another stream/transcoded from elsewhere.

The signalling and P2P connection establishing already works fine.

I already know the sample application from radioman (https://github.com/radioman/WebRtc.NET) but it just handles a custom rendered video stream, not audio data.

There are several similar questions on the net but many quite old and I would like to use the correct up-to-date approach.

I am not sure if I need to register an AudioCallback with the ADM (AudioDeviceModule), create a complete own implementation of the AudioDeviceModule (which could then skip the access to the hardware but thats not required) or if there is a way to intercept the audio stream (read and write) from the mediastream object on the peerconnection (Sink and Observer?).

Any hint would be appreciated, thanks!

[radioman . lt]

check Conductor::OnAddStream event, GetAudioTracks(), make audioPlayer -> AudioTrackInterface -> AudioTrackSinkInterface -> virtual void OnData(const void* audio_data,

                      int bits_per_sample,   int sample_rate,  size_t number_of_channels,  size_t number_of_frames) = 0;

use Native::VideoRenderer as demo for implementation

[Ricky B.]

Thanks for your response. I am already trying to use the sink on the AudioTrack with code like this:

void TestAudioTrackSink::OnData(const void* audio_data, int bits_per_sample, int sample_rate, size_t number_of_channels, size_t number_of_frames) {
    size_t valueCount = number_of_channels * number_of_frames;
    int16_t *_data = (int16_t*)audio_data;

    f.write((char*)&_data, sizeof(int16_t) * valueCount);
    f.flush();
}

f is a ofstream here. But when I import the raw data in an editor like Audacity I am not getting the correct audio (16bit PCM, channels and sample rate like the OnData method says).

Unfortunately the sink lacks documentation. Also how about the other direction, to write a file to the stream towards the browser?

[Ricky B.]

I wanted to add, getting the data streamed to me now works with the AudioTrackSinkInterface like this:

void TestAudioTrackSink::OnData(const void* audio_data, int bits_per_sample, int sample_rate,
        size_t number_of_channels, size_t number_of_frames) {

        size_t number_of_bytes = number_of_channels * number_of_frames * sizeof(int16_t);        
        f.write(reinterpret_cast<const char*>(audio_data), number_of_bytes);
        f.flush();
}

Is this the right thing to do?

How can I access the outgoing stream to the peer? For example to stream an audio file to him instead of my microphone.

[radioman . lt]

..somehow the event isnt called for me, i wonder what i'm missing ;/

[Ricky B.]

I am adding the sink to the track when the OnAddStream event is fired. Both peers must have at least one media stream, without also no ICE candidates get generated.

[Ricky B.]

I took a look at your code on GitHub and I believe you are adding a sink to the outgoing stream and also you have not set up local audio devices. Maybe that's a clue?

To record the incoming audio I have added a sink to the incoming stream (OnAddStream event). For stream a file for the outgoing stream I do not see anything on that track/stream object. I get the feeling I will need to implement my own AudioDeviceModule to achieve this.

[radioman . lt]

i do the same OnAddStream:

webrtc::AudioTrackVector atracks = stream->GetAudioTracks(); if (!atracks.empty()) { webrtc::AudioTrackInterface* track = atracks[0]; remote_audio.reset(new Native::AudioRenderer(*this, true, track)); }

there is one track in that list, i call AddSink, and i can hear the voice from the other end, but no events are fired at all ;/

On Thursday, November 3, 2016 at 7:09:49 PM UTC+2, Ricky B. wrote:

[Ricky B.]

Okay, major update.

As said before, getting the received data from the remote audio track by adding an AudioTrackSink to that track worked for me. I can get raw data that way.

For sending data to the peer which does not come from my audio hardware but a raw audio file (or other source not controlled by the native webrtc library) I could get it to work by supplying my own AudioDeviceModule. I have build it up based on the FakeAudioDeviceModule (webrtc/modules/audio_device/include/fake_audio_device.h) and FakeAudioDevice (webrtc/test/fake_audio_device.h) might be interesting too. Many of the methods you have to override are unused/empty.

I have build up mine to provide 1 recording and 1 playback device named "Virtual Input Device" and "Virtual Output Device" - but that interface parts might be skipped as well I guess.

The FakeAudioDeviceModule spins up a processing thread when playout or recording is started. There I can write data to the buffer before AudioTransportInterface::RecordedDataIsAvailable is called (which is provided by the voice engine and does all the webrtc magic to transcode and transport the raw audio to the peer).

Also I found out I could do it the other way around - when processing received data from the transport I can write it to a file after the call to AudioTransportInterface::NeedMorePlayData - so the sink is not necessary if you would like to skip regular audio hardware and handle all the bits and bytes by yourself.

[Jason Thomas]

For posterity, and since I have faced this multiple times and later forgot/lost the information here, the entire RTP receive, decode, pipeline in WebRTC eventually delivers decoded PCM packets to webrtc/modules/audio_coding/neteq/neteq_impl.cc, into a PacketBuffer.

They then just sit in the packet_buffer and wait for something to come along and request/pull packets out.

This only happens in webrtc/voice_engine/channel.cc in 'MixerParticipant::AudioFrameInfo Channel::GetAudioFrameWithMuted' and 'AudioMixer::Source::AudioFrameInfo Channel::GetAudioFrameWithInfo'

You will notice that in GetAudioFrameWithMuted, this is where the audio_sink_->OnData(data) callback is made, that will eventually propagate to delivering frames to our registered AudioSinkInterface.

So, what is responsible for calling GetAudioFrameWIthMuted, or GetAudioFrameWithInfo?

Basically the mixer module:

webrtc/modules/audio_mixer/audio_mixer_impl.cc: AudioFrameList AudioMixerImpl::GetAudioFromSources

And what configures the mixer, and animates its requests for more frames?

webrtc/audio/audio_receive_stream.cc:

void AudioReceiveStream::Start() {
  RTC_DCHECK_RUN_ON(&thread_checker_);
  if (playing_) {
    return;
  }


  int error = SetVoiceEnginePlayout(true);
  cout << "AudioReceiveStream::Start SetVoiceEnginePlayout(true) error: " << error << endl;


  if (error != 0) {
    LOG(LS_ERROR) << "AudioReceiveStream::Start failed with error: " << error;
    return;
  }


  cout << "AudioReceiveStream::Start mixer()->AddSource(this)" << endl;
  if (!audio_state()->mixer()->AddSource(this)) {
    LOG(LS_ERROR) << "Failed to add source to mixer.";
    SetVoiceEnginePlayout(false);
    return;
  }

   playing_ = true;

}

So, basically the way this code reads is, it attempts to start voice playout via the voice engine, and if it fails, it never attaches the mixer, which then never is around

to deliver audio samples to our OnData callback.

The remedy is as Ricky B. says, to provide your own AudioDeviceModule.

Override PeerConnectionFactory's default constructor, and replace the call to CreatePeerConnectionFactory with one that attaches a FakeAudioCaptureModule:

rtc::scoped_refptr<webrtc::AudioDeviceModule> adm = FakeAudioCaptureModule::Create();


The source for FakeAudioCaptureModule can be found here:
./webrtc/api/test/fakeaudiocapturemodule.cc

This will ultimately break normal playout, but probably a proxy could be made to wrap the default AudioDeviceModule, and basically if it fails to initialize, still allow the rest of the audio processing pipeline to continue so samples can be captured, and libwebrtc can continue to be used just as a library independent of its rendering features.

- Jason Thomas.

[Wesley]

Hi Ricky B,

Could you share your virtual ADM implement?

Any hint would be appreciated, thanks!