Issue recording H264 to MP4 file

[eden...@gmail.com]

Hello,

I am currently recording a H264 stream to MP4 file in order to prevent kurento's Transcoding.

However when playing back the recorded file after a certain period of time the video freezes. (audio is fine).

Having looked at the logs it seems that recording task stopped due to a "not-negotiated" error:

kurento_1  | 0:18:19.932820327     1 0x5614701ca800 WARN                   qtmux gstqtmux.c:4123:gst_qt_mux_video_sink_set_caps:<mp4mux6> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au,

 codec_data=(buffer)0142c029ffe3000f6742c0298c8d40b87acbc03c2211a8000f6742c0296323503c0a2403c2211a80000f6742c02920c8d40f028900f08846a003000468ce3c800004686ce3c8000568210e3c80, level=(string)4.1, profile=(string)constrained-baseline, widt

h=(int)360, height=(int)480, framerate=(fraction)0/1, parsed=(boolean)true

kurento_1  | 0:18:19.932880876     1 0x5614701ca800 WARN                   qtmux gstqtmux.c:4123:gst_qt_mux_video_sink_set_caps:<mp4mux6> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au,

 codec_data=(buffer)0142c029ffe3000f6742c0298c8d40b87acbc03c2211a8000f6742c0296323503c0a2403c2211a80000f6742c02920c8d40f028900f08846a003000468ce3c800004686ce3c8000568210e3c80, level=(string)4.1, profile=(string)constrained-baseline, widt

h=(int)360, height=(int)480, framerate=(fraction)0/1, parsed=(boolean)true

kurento_1  | 0:18:19.932939941     1 0x5614701ca800 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<videoSrc> error: Internal data flow error.

kurento_1  | 0:18:19.932948454     1 0x5614701ca800 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<videoSrc> error: streaming task paused, reason not-negotiated (-4)

kurento_1  | 0:18:19.932995024     1 0x5614701ca800 WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'gerror' of type 'GError'

kurento_1  | 0:18:19.932989061     1 0x5614701ca800 ERROR       recorderendpoint kmsrecorderendpoint.c:1968:bus_sync_signal_handler:<kmsrecorderendpoint6> Message error message: 0x7fafb000ec20, time 99:99:99.999999999, seq-num 10989, ele

ment 'videoSrc', GstMessageError, gerror=(GError)NULL, debug=(string)"gstbasesrc.c\(2948\):\ gst_base_src_loop\ \(\):\ /GstPipeline:pipeline13/GstAppSrc:videoSrc:\012streaming\ task\ paused\,\ reason\ not-negotiated\ \(-4\)";

kurento_1  | 0:18:19.933032204     1 0x5614701ca800 WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'gerror' of type 'GError'

kurento_1  | 0:18:19.933028885     1 0x5614701ca800 ERROR       recorderendpoint kmsrecorderendpoint.c:1972:bus_sync_signal_handler:<kmsrecorderendpoint6> Error: error message: 0x7fafb000ec20, time 99:99:99.999999999, seq-num 10989, elem

ent 'videoSrc', GstMessageError, gerror=(GError)NULL, debug=(string)"gstbasesrc.c\(2948\):\ gst_base_src_loop\ \(\):\ /GstPipeline:pipeline13/GstAppSrc:videoSrc:\012streaming\ task\ paused\,\ reason\ not-negotiated\ \(-4\)";

kurento_1  | 0:18:19.933122004     1 0x5614701ca800 WARN                   qtmux gstqtmux.c:4123:gst_qt_mux_video_sink_set_caps:<mp4mux6> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au,

 codec_data=(buffer)0142c029ffe3000f6742c0298c8d40b87acbc03c2211a8000f6742c0296323503c0a2403c2211a80000f6742c02920c8d40f028900f08846a003000468ce3c800004686ce3c8000568210e3c80, level=(string)4.1, profile=(string)constrained-baseline, widt

h=(int)360, height=(int)480, framerate=(fraction)0/1, parsed=(boolean)true

kurento_1  | 0:18:19.933170464     1 0x5614701ca800 WARN                   qtmux gstqtmux.c:4123:gst_qt_mux_video_sink_set_caps:<mp4mux6> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au,

 codec_data=(buffer)0142c029ffe3000f6742c0298c8d40b87acbc03c2211a8000f6742c0296323503c0a2403c2211a80000f6742c02920c8d40f028900f08846a003000468ce3c800004686ce3c8000568210e3c80, level=(string)4.1, profile=(string)constrained-baseline, widt

h=(int)360, height=(int)480, framerate=(fraction)0/1, parsed=(boolean)true

kurento_1  | 0:18:19.933207800     1 0x5614701ca800 DEBUG                  qtmux gstqtmux.c:3200:gst_qt_mux_add_buffer: dts: 0:00:30.807659672 pts: 0:00:30.807659672 timebase_dts: 1478768 pts_offset: 0

kurento_1  | 0:18:19.935115760     1 0x56147015ae90 ERROR   KurentoMediaPipelineImpl MediaPipelineImpl.cpp:69:processBusMessage: Error code 1: 'Internal data flow error.', element: kmsrecorderendpoint6, parent: pipeline12

kurento_1  | 0:18:19.935145691     1 0x56147015ae90 ERROR   KurentoMediaPipelineImpl MediaPipelineImpl.cpp:72:processBusMessage: Debugging info: gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline13/GstAppSrc:videoSrc:

kurento_1  | streaming task paused, reason not-negotiated (-4)

kurento_1  | 0:18:19.935189564     1 0x56147015ae90 ERROR   KurentoMediaElementImpl MediaElementImpl.cpp:262:processBusMessage: Error code 1: 'Internal data flow error.', element: kmsrecorderendpoint6, parent: kmsrecorderendpoint6

kurento_1  | 0:18:19.935201869     1 0x56147015ae90 ERROR   KurentoMediaElementImpl MediaElementImpl.cpp:265:processBusMessage: Debugging info: gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline13/GstAppSrc:videoSrc:

kurento_1  | streaming task paused, reason not-negotiated (-4)

Is there a way to solve this issue?

[Carlos]

A couple of hints in case it helps:

• If you want to use the recorder you'll also need to specify 'MP4' in the 'mediaProfile' option.
• Take a look to the bitrate and the H264 profile of the recorded video. I've had similar issues when trying to record from low quality cameras through a narrow bandwidth (such as a phone connection). Apparently when starting to record the video the browser selects a Baseline profile and the bitrate is high. So when the browser detects that the bandwidth is not very good (after 6 seconds or so) it changes the profile to High to get lower bitrate. That's fine for videoconferencing (that's WebRTC for), but for video recording, you cannot record a MP4 file with different profiles, so the video stops recording, while audio continues. When using HD cameras it doesn't happen because selected profile is High from the beginning.

Regards

[eden...@gmail.com]

Hi Carlos,

Thanks for the feedback.

• If you want to use the recorder you'll also need to specify 'MP4' in the 'mediaProfile' option.

I currently am specifying MP4 as the media profile.

The problem mostly occurs with Safari browser, I have never had an issue with recording when using Chrome/Firefox.

• Take a look to the bitrate and the H264 profile of the recorded video. I've had similar issues when trying to record from low quality cameras through a narrow bandwidth (such as a phone connection). Apparently when starting to record the video the browser selects a Baseline profile and the bitrate is high. So when the browser detects that the bandwidth is not very good (after 6 seconds or so) it changes the profile to High to get lower bitrate. That's fine for videoconferencing (that's WebRTC for), but for video recording, you cannot record a MP4 file with different profiles, so the video stops recording, while audio continues. When using HD cameras it doesn't happen because selected profile is High from the beginning.

 Thanks for the detailed explanation, I am currently limiting the bitrate on the endpoint objects, I'll try increasing the value and see if I get different results.

Just one question about the above, if the client is recording via a not so good network is there a chance that the H264 profile will change and the recording will fail?

Thanks for the help.

Regards.

[Carlos]

That's actually what happened to me, so indeed it can happen. In my case it was with low quality cameras. But I was forcing a high bitrate. So you could have similar issues on the opposite way: allowing low bitrate with high quality cameras with not good network. That's the browser starts with a low bitrate, but being able to improve quality it changes to high bitrate. In that change, it may also change the profile.

All of this is just guessing by the way. I've not actually proven that the browser behaves this way, it's just observation.

[Carlos]

Hi,

I've opened a bug report here about this:

https://github.com/Kurento/bugtracker/issues/359

You maybe want to contribute with your research.

Regards.

[Juan Navarro]

Hi,

additionally, support for Matroska container (MKV) has been added and can be tested in nightly builds, you may want to check it out to see if it makes any difference.

[Carlos]

Hi Juan,

I'll do the test and let you know, although that would mean to change at least the container once it's recorded. I think I could do that also with WEBM.

Anyway I'd be grate if you could guide me to find the problem and fix it so I can send a pull request.

Thanks.

Regards

[Carlos]

Hi Juan,

An update. It seems like this bug is happening also when recording under Windows 7. I think since some years Chrome is not supporting H264 and instead taking the codec from the Operative System. In that case, the H264 codec packed with Windows 7 is making Kurento Recorder to fail. What do you think?

Regards.

[Juan Navarro]

Hi Carlos,

I'm not sure of how Chrome handles its H.264 support under Windows; in Linux it is supposed to use its own FFmpeg that comes embedded with the installation, but in Windows 7 I wasn't able to find any libav* libraries under the Chrome install directory, so most probably you are right about this. However I haven't had time to research this and find some official source that can confirm this; maybe you could ask in the discuss-webrtc group to have a confirmation from one of Chrome devs...

Regarding the failure in Recorder, I haven't observed before (nor I have been reported) about the browser changing H.264 profile in the middle of the stream, but I think it might be possible that it happens under some conditions, and if that's the case then yes, it's surely a reason for having a Recorder error. This would also be an interesting bit of information to have from some dev who knows how Chrome behaves in such conditions... please if you do more research let us know about this.

If you are investigating an undesired video transcoding, keep in mind the log messages that might provide some useful information: https://doc-kurento.readthedocs.io/en/latest/user/troubleshooting.html#cpu-usage-grows-too-high
(note the messages described in that link apply only to Kurento 6.10)

Regards

[Carlos]

Hi Juan,

I've updated the bug report with some media constraints that we use and with a couple of video samples.

I've just asked in the group you said, my post just need to be approved by the moderators.

Also I'll try to take some logs with 6.10.0 about the transcoding and let you know.

Regards.

[Carlos]

Hi,

Just update here about one of the post, where I'm asking about H264 with Chrome: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/discuss-webrtc/4ooLmFIRxS8/GAIAuO6ICAAJ

Regards

[Juan Navarro]

Hi Carlos, the profile-level-id is just a description of the type of H.264 stream that is sent, but it says nothing about the H.264 encoder that is used to generate that stream.

I think that it's impossible to know just from the SDP what is the H.264 encoder used, as it doesn't seem to exist a field in which the stream codifies some information about its encoder.

What you saw in your SDP messages is a pretty usual line:

H.264 encoding/decoding profile

By default, Chrome uses this line in the SDP Offer for an H.264 media:

    a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f

`profile-level-id` is an SDP attribute, defined in [RFC 6184] as the hexadecimal representation of the *Sequence parameter set* from the H.264 Specification. The value **42e01f** decomposes as the following parameters:
- `profile_idc` = 0x42 = 66
- `profile-iop` = 0xE0 = 1110_0000
- `level_idc` = 0x1F = 31

[RFC 6184]: https://tools.ietf.org/html/rfc6184

According to the definitions from ISO/IEC 14496-10 (Annex A.2.1), a profile_idc of 66 with constraint_set0_flag and constraint_set1_flag = 1 corresponds to the H.264 Constrained Baseline profile; and level_idc = 31 which means Level 3.1.

When profile-level-id=42001f, it describes H.264 Baseline Level 3.1 (unconstrained).

Similarly, it's possible to "translate" other profile-level-id values.

You can read more detailed technical information about H.264 in here:

https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html

profile-level-id corresponds to these 3 fields of the SPS:

AVCProfileIndication (profile_idc)
profile_compatibility (constraint_setx_flag)
AVCLevelIndication (level_idc)

But in any case, I think that Firefox used the OpenH264 encoder because Cisco (its creator) provides it free from royalties for the community, so projects like Kurento or Firefox make use of it; but Chrome was using a custom version of FFmpeg and with it, the x264 encoder. But I'll wait for more replies in your thread in discuss-webrtc that might confirm (or deny) this, because it's an interesting trivia detail to know about (and improve our own technical documentation!). It's also interesting that Windows 10 doesn't show the same problem...