iOS: Accessing raw audio buffers

[Eddie Sullivan]

Hi. In our iOS app, we have a need to access video and audio directly, while at the same time sending the A/V out over WebRTC. For the video side of things, I can create a custom implementation of RTCVideoCapturer which lets us tap into the AVCaptureSession's video output. Is there a way to do the same thing for audio? From looking at the SDK source code, I see that the low-level Audio Toolbox library is used deep inside the C++ source code, but it's not clear how to tap into the flow of audio buffers. Ideally, I'd like to accomplish this using the Objective C/Swift interface, but I can dive down into the native C++ if that's the only option. Thanks. Eddie Sullivan

[Alin Radut]

Hello!

It depends whether you want to perform some processing on the audio buffers before you send them out or not. If all you want is to save them somewhere you can do this by modifying AudioDeviceIOS::OnDeliverRecordedData and AudioDeviceIOS::OnGetPlayoutData to forward the AudioBufferLists from your microphone and remote side back into your app.

The easiest way to do this would be by modifying the RTCAudioSessionDelegate and using the RTCAudioSession singleton to pass the buffers to where you need them. Credit where credit's due: https://groups.google.com/d/msg/discuss-webrtc/CvMCPNIIyqY/GhGBhmonCwAJ

Alin

[Eddie Sullivan]

Thanks Alin, and especially thanks for the link, because I was indeed thinking I could use AddSink as the linked poster unsuccessfully tried. I was hoping not to have to modify the SDK's source code, but if that's the only solution then I'll take a look.

I imagine there might be general use for such a capability, so if my client is on board I'll try to do it in a general way and submit a patch.

Thanks.

-Eddie