Audio cuts in Android set-top-box

[Jorge Vinaches]

Hi.

I am developing a video conference app between 2 Android devices.

The problem is, when I make a video-call between an Android mobile and an Android set-top-box, the audio has a lot of cuts and you can't understand anything.
Otherwise, if you try to make a video call between 2 Android mobiles, the sound is clear and everything seems to work nice ... because of that I think the problem is in the set-top-box side.

The audio cuts are small but repeated, for example, if I want to say "Hello, my name is Jorge", the audio sound will be "He.. my....me is....ge"

I've tried modifying the Aecm modes (by default is kAecmSpeakerphone), and modifying Ns modes also, but the audio cuts are still there.

More information:

• I'm developing with the webrtc revision 7700.
• The Android set-top-box is a Rikomagic MK902 with Android 4.4.2.
• The Android mobiles tested are: Samsung Galaxy Nexus, Nexus 5, Motorola MotoG, Samsung Galaxy Note 2.
  
• The Internet connection is local, with 100/10 mb of speed (so the problem is not a low internet speed I think).
• Developing with webrtc in OSX, with Eclipse using Android 4.2 platform (API level 17).
  

Repeat: between mobile devices the sound is clear, the problem comes with the set-top-box. Unfortunately, I need to use a set-top-box, so I can't avoid the problem just saying "Don't use a set-top-box".

If you need more information, please let me know because I need to solve this issue.

Thanks a lot.

Jorge.

[Vikas]

You might want to check if its related to performance (CPU spiking) on the device. Does making only an audio call also display the same problem?

/Vikas

[Henrik Andreasson]

The set-top-box should in theory be capable of performing full-duplex audio/video calls but there might be other bottlenecks on this particular device. I would start by following Vikas advice. Also, can you make a loopback call on the set-top-box?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aaroncray]

You probably already knew this but in case ya didn't: Often with set top boxes the cpu can't handle the encode/decode of video data and that is offloaded onto the GPU. This is why in many posts people are asking for the h.264 video codec support because it is already optimized in many cases for GPU offloading encoding and decoding. So the first thing I would do if I wanted it to work well for a set top box scenario is check the graphics chipsets and see if they have codec support for GPU offloading the encode/decode (OpenGL support). Then I'd force the OFFER to only offer h.264 or vp8/vp9 iff that now has GPU offloading support (don't think it does?)  

Also make sure you aren't using the debug build of webrtc and expecting no choppiness cause that will cause choppiness if your clock cycles are under strain from the video encoding/decoding. I get choppiness on an iPhone 6 and my android test phones (I have 12) with debug builds which is expected cause well... debug build is usually built with no optimizations for debugging purposes.

If you get it working well on a low powered device (set-top box) i'm sure everyone would dig hearing if it works or you had to add any changes manually (I would). Happy hacking!

[Ben Weekes]

In opensles_input.h edit these constants as follows:

    kNumOpenSlBuffers = 1,
    kNum10MsToBuffer = 40,

[Jorge Vinaches]

Thanks for your responses.

I have been digging into the problem and now I'm going to try to explain what I've found.

First of all, in response to Vikas, I've checked the CPU performance and with an audio/video call the CPU usage is around 70%. I've made an audio call only, and the CPU usage is 30%...but the audio cuts are still there, none improvements.

In response to Henrik, a loopback call generates lots of echo and I can't check the audio cuts without hurting my ears with the echo. Please, tell me what do you want to check with the loopback call to think about how to check it by other way.

In response to aaroncry. Well, this is maybe relevant information also to everyone. I am using VP8 codec for video and PCMU for audio. In relation to use debug build...yes, I use that. Tomorrow I am going to check this just in case.

So...reading more carefully logs and exceptions, I noticed some new information:

In set-top-box side, I have a lot of warnings like this:

Warning(webrtcvideoengine.cc:1554): webrtc: Audio overrun

Warning(webrtcvideoengine.cc:1554): webrtc: Audio underrun

The trace log is full of this kind of messages, only in the set-top-box side.

Otherwise, in mobile side, I noticed I have an Error:

Error(webrtcvideoengine.cc:1550): webrtc: (acm_receiver.cc:161): "AcmReceiver::SetExtraDelay" failed: delay_ms=3249

Error(webrtcvideoengine.cc:1550): webrtc: SetMinimumPlayoutDelay() failed to set min playout delay (error=10027)

Error(webrtcvideoengine.cc:1550): webrtc: (vie_sync_module.cc:165): Error setting voice delay.

This error is weird... It just happen while makes a call between the set-top-box and a mobile device. If the call is between mobiles, this error never appears. The error appears arround one minute after init the call, and it start when the delay is over 750ms. The error appears only in the mobiles, not in the set-top-box.

Digging into this problem, I've found this issue:

http://code.google.com/p/webrtc/issues/detail?id=2982

with similar audio problems, so I've tried to use their solution, that it's the same as Ben said..and I've got some improvements.

Using that configuration the audio from set-top-box to mobile seems to work nice, otherwise, in the other direction, from mobile to set-top-box continues with the audio cuts.

In relation with the "Warnings" and "Errors". With that configuracion the error does not appear anymore, and the warning just show "audio underrun", the "audio overrrun" has dissappear.

I don't know exactly what this constants mean and how it work, so I'm going to dig into the code to know what I'm modifying when I change these constants.

Hope this new information help to solve this.

I really apreciate your responses. Thanks.

Jorge.

[Jorge Vinaches]

I'm sorry, I am a liar:

In relation with the "Warnings" and "Errors". With that configuration the error does not appear anymore, and the warning just show "audio underrun", the "audio overrrun" has disappear.

That was false. The error has not disappeared, just appears a bit later.

The configuration was:

kNumOpenSlBuffers = 1,
kNum10MsToBuffer = 40,

So, without that configuration:

• Set-top-box: Show a lot of "audio overrun" and "audio underrun"
• Mobile device: Show some "audio overrun" (2 warnings each 1 or 2 seconds), and that error I've told you.
• Audio cuts in both directions.

With that configuration:

• Set-top-box: "audio overrun" disappeared, otherwise "audio underrun" continues.
• Mobile devices: "audio overrun" disappeared, message error continues but is shown a bit later.
• Audio cuts just from mobile to set-top-box.

Another error I've noticed:

Error(webrtcvideoengine.cc:1554): webrtc: (voe_audio_processing_impl.cc:1001): virtual int webrtc::VoEAudioProcessingImpl::SetTypingDetectionStatus(bool): not supported

This error is shown in both sides, set-top-box and mobile device.

I don't know if this is relevant, but I prefer show you what for me is weird.

Thanks. Jorge.

[Jorge Vinaches]

Some news:

When I build webrtc in release mode, the performance is the same than applying this constants:

kNumOpenSlBuffers = 1,
kNum10MsToBuffer = 40,

The "audio overrun" disappears and the sound from set-top-box to mobile is good. In the other direction don't.

So, by now, I'm going to continue developing with the release build and that constants by default (kNumOpenSlBuffers = 2; kNum10MsToBuffer = 3).

Jorge.

[Henrik Andreasson]

Native WebRTC can be built with two different audio backends. If you have issues with the OpenSL version you could try the alternative (pure Java based). Add enable_android_opensl=0 for GYP_DEFINES, run gclient runhooks and compare performance.

The: Error(webrtcvideoengine.cc:1554): webrtc: (voe_audio_processing_impl.cc:1001): virtual int webrtc::VoEAudioProcessingImpl::SetTypingDetectionStatus(bool): not supported can be ignored.

[Henrik Andreasson]

And you need a headset to avoid the echo in loopback (not sure if that is possible in this case).

[Jorge Vinaches]

Thanks Henrik for your response. I don't know if I am doing something wrong, but I don't see any difference between use the enable_android_opensl=0 flag and not. It is weird because I continue seeing the "audio underrun" log, but that log is fom "opensles_output.cc", and if I am disabling opensl, I shouldn't see that log...

My GYP_DEFINES is "OS=android enable_android_opensl=0", then I run gclient runhooks and build again with ninja (release build). But when I check the changes, the performance is the same and that log is still there...so I thing I am missing something...

Otherwise, I've seen SSL problems in my trace log:

 I/libjingle(15445): Jingle:Channel[audio|1|__]: DtlsTransportChannelWrapper: Started DTLS handshake

 I/libjingle(15445): Jingle:Conn[audio:q06UF9cx:1:0:local:tcp:192.168.1.56:38997->:1:1518149375:local:tcp:192.168.1.42:37375|C-xW|6520401919104269823|-]: Connection deleted due to read or write timeout

 I/libjingle(15445): Jingle:Channel[audio|1|RW]: Removed connection (2 remaining)

Maybe that does not mean anything...

I am going to attach the set-top-box log, and the mobile log (motorola moto g), related to the same call. Maybe you will be able to see something wrong in the logs that I can't see.

That call was make with opensl enabled.

Thanks. 

Jorge. 

[Henrik Andreasson]

Setting enable_android_opensl=0 should work. I have verified it locally. As a simple test, I added an invalid line before #280 here:

https://code.google.com/p/webrtc/source/browse/trunk/webrtc/modules/audio_device/audio_device_impl.cc#280

and verified that I can still build. Hence, WEBRTC_ANDROID_OPENSLES is in fact undefined as it should. 

The other error message are most likely irrelevant. 

--

[Jorge Vinaches]

Yes, you were right. It was my fault building wrong the sources.

Finally I built without opensl and everything work nice.

Audio between set-top-box and mobiles sounds really great.

Thanks a lot! 

Jorge.

[Henrik Andreasson]

Glad to hear that it works well. Seems like we have to polish our Open SL implementation. Thanks for all feedback.