H264 support in webrtc

[Ravindra Bharathi]

Greetings
What is the status of H264 support in WebRTC ?
I tried the latest code for Android and it seems like the offer SDP still contains only VP8 .
Would be great if someone can share some information on whether H264 support would be enabled soon or if its already there and I'm perhaps missing something in enabling H264 support.

Thanks

[gla...@webrtc.org]

For standalone WebRTC Android build H.264 codec should be supported if Android device supports H.264 hardware encoding and decoding. If there is no hardware H.264 codec then only VP8 will be used.

You can try to compile and run Android AppRTCDemo application - it has an option to use H.264 codec as default video codec.

[Ravindra Bharathi]

Thanks for the answer. Figured that one out from https://code.google.com/p/webrtc/source/detail?r=8465
Yes I was able to use the latest code to get the apprtc demo for Android working with H264 set as preferred codec.
Works for kitkat and newer versions .
I assume only ARM based devices would be supported right now . I could be wrong though

[gla...@webrtc.org]

Right now H.264 hardware codec is supported for Qualcomm platforms only - WebRTC code on Android makes the decision to enable/disable H.264 hardware based on media codec OpenMax component name. If there will be x86 devices with good quality H.264 codec we can extend this list of course.

[Ravindra Bharathi]

Thanks for this information. When using H264 , does the peer that offers video (a browser or a camera) to an android webrtc client need to limit the video size , etc for it to seamlessly work on Android devices ? What we are seeing is that the webrtc session using h264 is started but the video seems frozen . Any insights will help.

Thanks

[Ravindra Bharathi]

Ok. it had to do with profile   ... The frozen video issue disappeared after encoding the stream with h264 baseline profile . The video playing is not as seamless as I'd like it to be but doesn't just stop like before.
Hopefully this information is of help to some else facing the same issues

[Kang Hua]

Hi:

    if mtk \ mstar chip also support h.264 , can i extend supportedH264HwCodecPrefixes ? let it use h.264 hareware power?

在 2015年3月26日星期四 UTC+8下午12:31:05，gla...@webrtc.org写道：

[gla...@webrtc.org]

Yes, you can extend supportedH264CodecPrefixes and ideally if MTK supports yuv420 or nv12 formats it should work. 

[Martin]

do you know where is this list in the code and what one has to do to enable a new component on non-android devices/systems?

[Peter Boström]

What does that mean? What do you want to do?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/9e41961d-d94c-41c1-939d-25f150585b02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Boström]

For Android those prefixes are here for the decoder and encoder respectively. This is for MediaCodec based (often hardware) codecs. H264 support through OpenH264 in Chromium is tracked here. If there's no existing implementation for the encoder/decoder pair you want to use you'll have to implement webrtc::VideoEncoder/webrtc::VideoDecoder wrappers around them.

These instances are created here and here when using PeerConnection, and an external video-encoder factory provided from outside can be used to support more encoder-decoder pairs (this is how MediaCodec gets used today, though I'm not sure where the video-encoder factory is provided).

Hope that helps,

- Peter

[Martin]

Thanks Peter, your answer put some light on what I would like to do.

What I would like to do is to use an HW H264-baseline encoder present on the ARM-based SoC I am using which is supported via OpenMAX.  Is there a way to reuse/extend the video-encoder factory the MediaCodec is providing to the peer connection in order to create/use this encoder based on its OpenMAX name only?

Do you think that is possible?

Thanks

[Peter Boström]

I'd replace it, it looks like we're using MediaCodecVideoEncoderFactory on Android, and if you're neither using Android or MediaCodec I wouldn't think that makes too much sense.

But more specifically if you're looking into H264 then maybe you can implement H264Encoder::IsSupported and H264Encoder::Create locally. Right now it's implemented for VideoToolbox when building with special flags on Mac/iOS.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/abfdc60a-4aa8-437e-b819-7460ca1764be%40googlegroups.com.

[Martin]

There is one thing not clear in the VideoEncoder interface:

what calls the registered callback
     EncodedImageCallback::Encoded // Callback function which is called when an image has been encoded.

is it supposed to be called by the framework? If yes, how does it know when an image has been encoded? When exactly? It's a bit confusing. Comments and the VideoToolBox example do not help. In the VideoToolBox I see this comment:

    // This is the callback function that VideoToolbox calls when encode is
    // complete.
       void VTCompressionOutputCallback(void* encoder, ...

What does "when encode is complete" exactly? I tried to search for the point in which that callback is explicitly called in the VideoToolBox encoder (through the callback pointer) , but I did not find anything.

[Peter Boström]

When something (typically a keyframe or delta frame) has been encoded and should be packetized and transmitted. Our software VP8 encoder does this here. In VideoToolbox we know that encoding of a frame is complete because that's what the callback means?

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/22759b0b-7135-4a01-9c0f-dd827096c209%40googlegroups.com.

[Peter Boström]

The callback is specified here, but the VideoToolbox implementation is outside of our control. That's why you can't find the callback being called from anywhere.

[Boyuan Liu]

Hello,

i am a freshman in webrtc, currently, we try to make a conference application under webrtc, 

in order to support H.264, we add mstar / mtk openmax items for both decoder and encoder array , i.e. supportedH264HwCodecPrefixes. 

Unfortunately, when connect from mtk platform to mstar platform or vice versa, we can not display the video data. 

What may the problem be ? Is it probably the two platform not compatable with each other ?

p.s. mtk to mtk and mstar to mstar plays well

Thanks a lot 

[Martin]

Hi,

what's the purpose of bool H264Encoder::IsSupported()

if an encoder has to/can be already specified when creating a peerconnectionfactory?

Thanks

[Peter Boström]

To decide whether or not to offer it in SDP we check if the codec is internally supported.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2c1520da-fc22-4db0-b6bb-42d1e7bc6f27%40googlegroups.com.

[Martin]

Thanks for your answer,

now I am in a situation where I locally modified ::IsSupported() and ::create(). When offering,  I can see VP8 and H264 in the SDP, but VP8 is always preferred over H264, so I cannot test the latter. Do you have any suggestions how to make use of the H264 encoder during the negotiation without explicitly disabling VP8 in the code?

[Peter Boström]

Reorder them within SDP so that H264 appears before VP8 in the list of supported codecs. If I recall correctly you can just move this line further up so that it's added before VP8: https://chromium.googlesource.com/external/webrtc/+/89ef6cc13e8aa9f16b212dd82124f4297a1f7385/talk/media/webrtc/webrtcvideoengine2.cc#349

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/8805e294-73ec-4b5d-acc8-7780ea71082b%40googlegroups.com.

[Harald Alvestrand]

The proper change is to change the order on the m-line, not the order of declarations (the order of declarations is suppposed to have no effect).

So, if you have:

m=video ..... 97 98

a=rtpmap:97 vp8....

a=rtpamp:98 h264....

you should change the first line to

m=video .... 98 97

before sending the SDP to your peer.

That should be enough.

(The W3C WG is discussing adding an API to let you do this, so that you can do it without munging SDP. Not yet finalized.)

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAGFX3sHJmsud8Sm8p%2BdkMgaUaiyjAmcJaUTAm6%3DK8eyqbEvcmw%40mail.gmail.com.

[Martin]

Hi Peter,

there must be something more to have the H264 encoder instantiated. I appropriately implemented bool IsH264CodecSupported() and H264Encoder::Create() in h264.cc. According to what you suggested, I also have H264 in the SDP as the only possible codec now.

However, while I can see IsH264CodecSupported() being called at some point,  H264Encoder::Create() is never called (therefore my encoder is never instantiated).

What do I have to check/do to make the framework call  the H264Encoder::create() static member function in order to instantiate my encoder?

[Peter Boström]

Since you removed all other possible codecs I've no idea why it's not instantiating that one. I'd printf debug if I were you. AppRTCDemo on iOS can use that path (if special flags are set to build and prefer that codec).

On my phone, hope that brief answer is enough. Check Harald's response as well. :)

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/7d3ad918-3d12-48b4-aaf8-69916bd7f038%40googlegroups.com.

[Martin]

I debugged this issue in every possible way, including logging. H264Encoder::create() is not called unfortunately.

Unfortunately I do not have the hardware to test the VideoToolBox encoder, but my implementation is basically the same. This is the SDP offer I obtain when removing all the codecs except H264 in webrtcvideoengine2.cc.  Harald's requirement seems to be satisfied. Should not at this point the encoder be instantiated?

v=0
o=- 9013649586430106420 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE video
a=msid-semantic: WMS stream_label
m=video 9 UDP/TLS/RTP/SAVPF 107
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:IwPYOdQvh6rn8/uw
a=ice-pwd:y3Zc1Rx8IpK1CH9AzoxvcaHb
a=fingerprint:sha-256 2F:FD:06:F3:94:8F:F4:B2:E3:53:C6:87:49:F7:BC:82:96:AB:C2:D3:C2:9A:DE:FE:13:E3:A2:90:03:7A:46:6D
a=setup:actpass
a=mid:video
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 urn:3gpp:video-orientation
a=sendonly
a=rtcp-mux
a=rtpmap:107 H264/90000
a=rtcp-fb:107 ccm fir
a=rtcp-fb:107 nack
a=rtcp-fb:107 nack pli
a=rtcp-fb:107 goog-remb
a=ssrc:3394908450 cname:Wxa32lf9hFbNv46I
a=ssrc:3394908450 msid:stream_label video_label
a=ssrc:3394908450 mslabel:stream_label
a=ssrc:3394908450 label:video_label   

[Martin]

Ok. After having spent the whole day on this approach, I think I will give up. There clearly is something beyond my knowledge of the library internals. I am not even sure supporting the encoder internally is what I want after all. I'll try by passing the video encoder factory to the peer connection factory instead.

My question is whether passing the factory for the video encoder only - and not for the video decoder - would work and whether my encoder would override or have an higher priority than the internal ones.

Thanks

[Peter Boström]

It would prefer your hardware one over internal ones. Support for a codec is symmetric, so you either support both encoding/decoding or none. Filed bug here to track separating these.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/5ebda000-5c5e-4204-9472-2e5e7deeae65%40googlegroups.com.

[Martin]

Hi Pete,

still no luck. I created both the VideoEncoderFactory and VideoDecoderFactory and added the H264 codec type to them. I modified DefaultVideoCodecList() to make sure I saw H264 as only possible codec in the SDP. I passed the factories to createpeerconnectionfactory(...).

However, also in this case, there is just no way to see my encoder being instantiated, because (the overridden)  cricket::WebRtcVideoEncoderFactory::CreateVideoEncoder() is *never called*. What is the reason now I do not know.

Do you have any idea? I know there is fakewebrtcvideoengine.h containing fake factories, but I wonder if they have ever been tested with createpeerconnectionfactory(...). to know if the framework is working as expected.

[Peter Boström]

It's instantiated in Chromium here: https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/media/webrtc/peer_connection_dependency_factory.cc&q=createpeerconnectionfactory&sq=package:chromium&type=cs&l=414 and used on ChromeOS among others. It's also used in AppRTCDemo on Android.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/3137441e-e8ef-4821-9f50-838ce8d605e1%40googlegroups.com.

[Martin]

Hi Peter,

yes, that is the way I was/am using the API. It's REALLY simple. However, I spent another day double checking and debugging all the stuff (the code has been taken from master recently), but I still do not see the video encoder factory being called anywhere, although I am using the createpeerconnectionfactory properly and H264 is the only codec enabled and appearing in the SDP. What happens is that the peer connection is created as usual (offer), and I also see that YUV420 frames are captured from the device, but they cannot go through my encoder as it is simply not instantiated. I am building with the "OS=linux target_arch=arm [...]" flags. What should I check now? Should I enable some magic flags? I have used the API in all the possible combinations! The fact that this also failed when I initially tried to enable encoder/decoder internally makes me really suspect there's something more to do apart from properly using the API.

[Martin]

never mind, I found the problem. thanks again.

[Peter Boström]

Mind sharing what it was? :)

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/ef54b4e5-3c60-42b1-b22d-e4065ce87766%40googlegroups.com.

[Martin]

Hi Peter,

I was wrong, sorry. "don't count your chickens before they've hatched" :)

It turns out that I got the cricket::WebRtcVideoEncoderFactory::CreateVideoEncoder(type)  called, yes, but the type there was VP8, not H264, never H264. This is because, you know, I have tried for tens or more times (i.e. by rebuilding with differrent flags, reimplementing the encoders internally and/or externally, simulating all the code you posted so far, etc...) , and this is one case where I had all the codecs listed in the SDP (VP8 first, H264 second). So, what happened is that CreateVideoEncoder() was called, as I said, but returned nullptr, as the 'VP8' codec was not supported by my encoder factory, at that point the framework switched to using  the  internal VP8 software encoder for some reasons. This is why I was under the false impression (to my BIG surprise) that the thing was starting to work for some reasons...

Peter, unless you have any other idea about what is possibly wrong in what I have been doing for days now   and/or want me to try anything else (just ask), I think I'll have to resign, as the API does not give me the results I am expecting.

[Peter Boström]

Are you implementing ::codecs() + does that one get called + does that one contain H264 ?

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/eced416c-73ef-44fe-b3c4-7c4363365164%40googlegroups.com.

[Martin]

Hi Peter,

Yes, I implemented ::codecs().

It's called several times during one offer and, yes, contains  H264. It's the only member function being called. Here is some log which shows when ::codecs() is called (under <debug> there is my stuff):


(webrtcvideoengine2.cc:556): WebRtcVideoEngine2::WebRtcVideoEngine2()
<debug> factory.h, codecs, 64: Returning supported codec list for encoder...
<debug> factory.h, codecs, 66:   codec: H264
<debug> factory.h, codecs, 64: Returning supported codec list for encoder...
<debug> factory.h, codecs, 66:   codec: H264
(webrtcvoiceengine.cc:526): WebRtcVoiceEngine::Init


....

(webrtcvideoengine2.cc:899): SetRecvCodecs: {VideoCodec[120:H264:1280:720:30:0]}
<debug> factory.h, codecs, 64: Returning supported codec list for encoder...
<debug> factory.h, codecs, 66:   codec: H264
(webrtcvideoengine2.cc:925): Changing recv codecs from {} to {VideoCodec[120:H264:1280:720:30:0]}

...

(webrtcsession.cc:923): Session:7363693225198545553 Old state:STATE_INIT New state:STATE_SENTOFFER
(channel.cc:1730): Setting local video description
(webrtcvideoengine2.cc:899): SetRecvCodecs: {VideoCodec[120:H264:1280:720:30:0]}
<debug> factory.h, codecs, 64: Returning supported codec list for encoder...
<debug> factory.h, codecs, 66:   codec: H264
(webrtcvideoengine2.cc:920): Ignoring call to SetRecvCodecs because codecs haven't changed.

Any idea?

[Peter Boström]

Looks like it's choosing the right codec to set even then. I'd set a breakpoint and debug from SetRecvCodecs?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/99c47d77-22df-468b-9361-9d2df6364f05%40googlegroups.com.

[Martin]

Hi Peter,

I suspect the encoder is never created because the following condition is false:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/libjingle/source/talk/media/webrtc/webrtcvideoengine2.cc&rcl=1447346049&l=1758

because its caller WebRtcVideoChannel2::SetSendParameters()  is never called:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/libjingle/source/talk/media/webrtc/webrtcvideoengine2.cc&l=866&cl=GROK&rcl=1447346049

do you know why?

[Martin]

Peter, as I said before, this is an offer-only connection. I do not if this can help, but if I am interpreting the call stack correctly, it seems that SetSendParamters() (mentioned in my previous post) would be called when setting the codec for the remote video only, but in my case the remote video stream does not exist and the remote sdp is essentially empty, so my external encoder for local video can not be instantiated In other words it looks like the local encoder depends  on the presence of a remote video stream somehow. Does this make any sense or am I just misreading the code?

[Peter Boström]

I think that makes sense, the model at this level assumes that the remote side will announce what it supports and then we'd pick one to always send something that the remote side supports. I'd guess you're not sending video at this point (and not just from the wrong encoder). I'm not as familiar with the configuration above this level (offer-only etc.), so I'd defer to someone else for how that is supposed to work.

I guess that "If you don't have a receiver what's the point of encoding?" is the design here.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/7e792b70-6ff6-48a9-adfd-d9e038b78850%40googlegroups.com.

[Martin]

Hi Peter, thanks for you answer, but I am not sure I fully understood what you meant by "If you don't have a receiver what's the point of encoding?". Probably because I was not clear enough. Let me explain my scenario again.

A peer connection is created between two peers (the answering peer is a browser) where the offerer sets both OfferToReceiveVideo and OfferToReceiveAudio to FALSE. Everything works perfectly when using the default software encoder on the offerer side - that is, using the CreatePeerconnectionFactory() version which takes NO arguments. The video stream in this case is one-way. Now I am just trying to replace the default VP8 encoder with my hw H264 encoder by passing the Encoder/Decoder factories to the multi-arg version of CreatePeerconnectionFactory(), but here are the problems I 've been describing so far, and I still fail to understand the reasons why this would not work as before..

I hope this is more clear now..

[Harald Alvestrand]

Can you explain the words "offer only connection"?

The usual setup involves sending the offer, getting an answer, and using both pieces to configure the codecs. If you don't process the answer at the sender, you might be hitting a code path that's never used otherwise.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/c5dccf64-d90e-470d-92a0-79fad8fb80d2%40googlegroups.com.

[Martin]

Hi Harald,

probably you did not get my previous answer to Peter yet.
Anyway, by "offer-only" I meant to say that I set both OfferToReceiveVideo and OfferToReceiveAudio to false. This produces a one-way video stream from the offerer to the answerer and the normal local/remote sdp handshake happens as usual.

The problems arise when trying to pass my encoder/decoder factories to have MY hardware encoder used.

[Peter Boström]

But do you still process the answer from the browser on your iOS client? The browser should respond with what they want to receive (based on your offer) and that should be your remote video stream. I'm confused since this part of the stack/APIs are outside my comfort zone.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/8e98aa84-74f2-4da5-8110-71879cf2888a%40googlegroups.com.

[Peter Boström]

Also, if you're offering H264 Chromium probably won't respond that it accepts it until https://code.google.com/p/chromium/issues/detail?id=500605 lands, is that possibly what you're hitting?

[Martin]

That  is firefox. I was aware Chrome does not support H264 yet. The answer is processed as usual, as the code path is the same in my client. The only thing different between the working case from the NOT working case is that I am passing the Encoder/Factories to CreatePeerConnectionFactory now. This will probably hit a different path in the WebRTC framwork... but I would expect it to work in any case.

I'll try to analyse the remote SDP in the two cases above to see if there are any differences which would lead these problems somehow, although, as I said, whatever the content of the answer is , it's processed by the same webrtc APIs

[Peter Boström]

Then you're possibly hitting https://code.google.com/p/chromium/issues/detail?id=543540 and https://groups.google.com/d/msg/discuss-webrtc/jjUKXC7RJVI/GQRw7VKACgAJ.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/c9993e63-4929-4b1a-94f8-1306f7405a45%40googlegroups.com.

[Martin]

Interesting. I'll try the second hack..

[Devin Heitmueller]

I ran into a similar issue when hacking an H.264 encoder into a send-only webrtc client.  In the end the encoder wasn't being spawned because I had to advertise both support for H.264 encode *and* decode in the video_codec/h264/foo.c plugin (due to some faulty assumptions in the webrtc framework that both would always be present).  Definitely check the codecs in the SDP offer being sent and see if H.264 is being offered up.  If you don't even see it being offered up in the SDP then I would add debug to webrtcvideoengine2.cc since it's an issue with advertising the supported codecs.

Devin

[Devin Heitmueller]

Also worth noting that I definitely hit the issue Peter described above, but that was further down the negotiation.  In that case the SDP appeared in the offer but because it was malformed Firefox would never select it in its answer.

Devin

[Martin]

It works when implementing the hack. But is this a bug in the current webrtc implementation is it just because full support for h264 is not ready yet?

[Peter Boström]

I think this bug is due to whoever put H264 support in just called it H264 without profiles and stuff. The crbug could've been filed against webrtc, but I believe it covers what you need fixed for Firefox interoperability.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/ac66aad0-7cb7-404c-a80e-84a72072dc2d%40googlegroups.com.

[Martin]

Hi Peter, guys,

Ok. I implemented the H264 encoder for my hardware and I use it for high resolutions (greater than the default 640x480) in my LAN.

I noticed that the WebRTC framework periodically asks to update the bitrate with new values, but the values I am observing are strange for two reasons:

1. The initial bitrate is too low 300kbps for this resolution, which gives very poor image quality
2. The updated values never reach an acceptable value (e.g. 1500kbps/s) considering I have plenty of available bandwidth
3. The values being updated sometimes follow a strange patter, for example:
     300k, then abruptly down to 80k, then 90k, 100k,..... up to 800k in few minutes (too much time and, as I said, in any case never the expected 1500kbps to get a decent quality)

I have few questions:

1. What are the new bitrate update requests based on? Is the resolution taken in consideration?
2. Why do the asked bitrates apparently never reach the 1500kbps or more
3. What is causing the above pattern?
4. Is there a way to increase the initial bitrate and have a more precise estimation?
5. As opposite to 4, is there a way to disable this bitrate estimation at all. What are side effects of ignoring these update requests (assuming I stream at a constant bitrate) - in my tests I could not see any issue when ignoring these requests?
6. If I need to modify the SDP somehow, can I do this via the Jsep C++ interface? (for example, I see a VideoContentDescription::set_bandwidth())

Thanks in advance for your help.

[Peter Boström]

As a partial response we're considering choosing a smarter initial resolution, so even though we might be trying to send HD, it'd start with VGA or something before the bitrate ramps up enough to have good quality. It hasn't made the cut yet though. As a workaround you could choose to downscale by 2x until your estimate reaches 500k, or some other constant. We're doing QP-based downscaling on Android, which means that when quality is estimated to be bad (high quantization) we scale down before encoding, which looks a lot better on our implementations. Perhaps the corresponding CL could give you hints to do the same in your implementation. This doesn't help with the initial estimate, but you can get out of the "VGA looks crappy" situation at least.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/a69c1a3c-7122-421e-b6df-3b4848d96026%40googlegroups.com.