MediaSource API works in Firefox but not in Chrome

[Chetan Naik]

Hi All,

I am trying to stream a WebM format video being generated using gstreamer and individual frame being sent over websockets. A typical byte arrangement of webm file is like this (you may be already familiar with this).

EBML (head size: 12 bytes, data: 16 bytes, pos: 0, '0x0')
    DocType (head size: 3 bytes, data: 5 bytes, pos: 12L, '0xcL') : 'webm\x00'
    DocTypeVersion (head size: 3 bytes, data: 1 bytes, pos: 20L, '0x14L') : 2
    DocTypeReadVersion (head size: 3 bytes, data: 1 bytes, pos: 24L, '0x18L') : 2
...
...
SegmentInfo (head size: 12 bytes, data: 91 bytes, pos: 192L, '0xc0L')
    TimecodeScale (head size: 4 bytes, data: 3 bytes, pos: 204L, '0xccL') : 1000000
    Duration (head size: 3 bytes, data: 8 bytes, pos: 211L, '0xd3L') : 0.0
    MuxingApp (head size: 3 bytes, data: 31 bytes, pos: 222L, '0xdeL') : 'GStreamer plugin version 1.2.4\x00'
    WritingApp (head size: 3 bytes, data: 25 bytes, pos: 256L, '0x100L') : 'GStreamer Matroska muxer\x00'
    DateUTC (head size: 3 bytes, data: 8 bytes, pos: 284L, '0x11cL') : 447902803000000000L
Video (head size: 9 bytes, data: 8 bytes, pos: 295L, '0x127L')
    Pixel Width (head size: 2 bytes, data: 2 bytes, pos: 351L, '0x15fL') : 640
    Pixel Height (head size: 2 bytes, data: 2 bytes, pos: 355L, '0x163L') : 480
    Codec Id (head size: 2 bytes, data: 6 bytes, pos: 359L, '0x167L') : 'V_VP8\x00'
Cluster (head size: 12 bytes, data: 72057594037927935L bytes, pos: 367L, '0x16fL')
    TimeCode (head size: 2 bytes, data: 2 bytes, pos: 379L, '0x17bL') : 1514
    SimpleBlock (head size: 4 bytes, data: 44618 bytes, pos: 383L, '0x17fL') : 'binary'
      track number : 1, keyframe : True, invisible : 'no', discardable : 'no'
      lace : 'no lacing', time code : 0, time code(absolute) : 1514
    SimpleBlock (head size: 3 bytes, data: 793 bytes, pos: 45005L, '0xafcdL') : 'binary'
      track number : 1, keyframe : False, invisible : 'no', discardable : 'no'
      lace : 'no lacing', time code : 27, time code(absolute) : 1541
<<conitnued....>>




What I see, the absolute time code and relative time code is being written correctly when I redirect the gstreamer output to filesink. The same gstreamer pipeline is used to extract the byte sequences (samples). These samples are then transmitted over websocket and received on client side using MediaSource API.

My implementation of client javascript is described here. When I run the client in Firefox, the video runs smoothly without any glitches. But on Chrome, the video freezes after some time or at the beginning.

I tried modifying sourceBuffer.mode = "sequence" or "segments", none of the options work on Chrome, whereas the video feed on Firefox is totally unaffected by any value of "sourceBuffer.mode". The description of these modes is here. (I am assuming that the MediaSource API works the same way on IE and Firefox, as no documentation available on Mozilla website).

Also, mediaSource.duration is Infinity/NaN in Chrome and Firefox both.

No matter which way I try, the live feed on chrome is not at all working, whereas Firefox displays smooth video. Any suggestions, why this could be happening?

[Chetan Naik]

UPDATES: I upgraded to Chrome version 41 which gives more details on chrome://media-internals. The message that is shown is:

render_id: 23
player_id: 1
pipeline_state: kStopped
EVENT: WEBMEDIAPLAYER_DESTROYED
url: blob:http%3A//localhost%3A8080/172f68c8-9ff3-4983-9dcb-    396b3f843752
found_video_stream: true
video_codec_name: vp8
duration: unknown
video_dds: false
video_decoder: FFmpegVideoDecoder
error: Append: stream parsing failed. Data size=2283         append_window_start=0 append_window_end=inf
pipeline_error: pipeline: decode error


Timestamp   Property    Value
00:00:00 00 pipeline_state  kCreated
00:00:00 00 EVENT   PIPELINE_CREATED
00:00:00 00 EVENT   WEBMEDIAPLAYER_CREATED
00:00:00 00 url blob:http%3A//localhost%3A8080/172f68c8-9ff3-4983-9dcb-396b3f843752
00:00:00 00 pipeline_state  kInitDemuxer
00:00:01 687    found_video_stream  true
00:00:01 692    video_codec_name    vp8
00:00:01 692    duration    unknown
00:00:01 692    pipeline_state  kInitRenderer
00:00:01 694    video_dds   false
00:00:01 694    video_decoder   FFmpegVideoDecoder
00:00:01 695    pipeline_state  kPlaying
00:00:10 989    EVENT   PLAY
00:00:11 276    error   Got a block with a timecode before the previous block.
00:00:11 276    error   Append: stream parsing failed. Data size=2283 append_window_start=0 append_window_end=inf
00:00:11 276    pipeline_error  pipeline: decode error
00:00:11 276    pipeline_state  kStopping
00:00:11 277    pipeline_state  kStopped
00:01:14 239    EVENT   WEBMEDIAPLAYER_DESTROYED

How to fix or calculate the "append_window_end"???

[PhistucK]

Looks like GStreamer is generating improper streams/containers/frames.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-html...@chromium.org.
To post to this group, send email to chromiu...@chromium.org.
Visit this group at http://groups.google.com/a/chromium.org/group/chromium-html5/.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[chcunnin...@chromium.org]

I agree with PhistucK ... looks like the webm file is either not valid, or you are appending parts of the file out of order. 

From the WebM guidelines (http://www.webmproject.org/docs/container/):

All absolute (block + cluster) timecodes MUST be monotonically increasing.

For MSE, once start of a cluster is appended, you must append the remainder of that cluster, with each block inside the cluster in order (increasing) by timecode. 

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-html5+unsubscribe@chromium.org.

[Chetan Naik]

Yes, but then the Firefox is able to play the same broadcast, without any glitches. why could that be?

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-html...@chromium.org.

[PhistucK]

Different codecs or implementation.

It may be a Firefox bug or a Chrome bug.

If you checked your streams and everything is in the right order, like chcunningham@ suggested, then it might be a Chrome issue. Otherwise, you may want to file an issue with Firefox instead (be a good web citizen :)) - 'this thing works but it should not' type of thing.

☆PhistucK

[Matthew Wolenetz]

Update: There is now a crbug at least tracking this problem. Independently, I suspect incorrect gstreamer webm muxing (see https://code.google.com/p/chromium/issues/detail?id=469457#c6).

+1 to PhistucK's request to file "This thing works but it should not" as a Firefox issue.

[Chetan Naik]

Hi PhistucK & Matthew,

Thanks for helping and investigating. Your conclusion may be correct that Firefox is ignoring this particular condition of monotonically increasing timestamps.

Actually if I print sourceBuffer.buffered.start(0) and sourceBuffer.buffered.end(0), after every update of sourceBuffer, I get the following:

WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.481
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.55
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.62
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.69
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.76
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.83
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.9
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.97

if you notice, every time the buffer is updated, the start time remains 1.454 and the end time is increased depending on length of video data received. I will have to figure out a way to modify the gstreamer "start time" while streaming.

having said so, if I have to modify the time stamps, I will have to manipulate the byte buffer (in Python2.7). Any suggestions, how that can be read and modified? I can alter the time stamps based on frame rate and bitrate.

[Matthew Wolenetz]

I'm not clear: is your example buffered start/end range printout from Firefox or Chromium? I assume Firefox, unless the decode error comes later.

In general, I suggest you focus more on obtaining properly muxed content to feed to MSE API (Chromium, Firefox, IE, etc..) rather than trying to work around the content. It might be good to file another bug against gstreamer "why is gstreamer muxing blocks sometimes with decreasing block timecodes within a webm cluster?" (in addition to a Firefox one "this works but shouldn't", and the existing Chromium https://crbug.com/469457)

More specifically, in your example, what were you expecting the range start/end to be after every append to the SourceBuffer? I'm confused about why the start time might be expected to change.

Matt

[Matthew Wolenetz]

On Thu, Mar 26, 2015 at 3:48 PM, Matthew Wolenetz <wole...@chromium.org> wrote:

I'm not clear: is your example buffered start/end range printout from Firefox or Chromium? I assume Firefox, unless the decode error comes later.

In general, I suggest you focus more on obtaining properly muxed content to feed to MSE API (Chromium, Firefox, IE, etc..) rather than trying to work around the content. It might be good to file another bug against gstreamer "why is gstreamer muxing blocks sometimes with decreasing block timecodes within a webm cluster?" (in addition to a Firefox one "this works but shouldn't", and the existing Chromium https://crbug.com/469457)

More specifically, in your example, what were you expecting the range start/end to be after every append to the SourceBuffer? I'm confused about why the start time might be expected to change.

I see that the crbug has been updated with a similar comment by you. Let's continue this in one place, on the crbug for now.

Matt

[Enrique Ocaña González]

On Thursday, March 26, 2015 at 10:19:54 PM UTC+1, Chetan Naik wrote:

WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.481
WSvideo_chrome.js:70 time_range start: 1.454
WSvideo_chrome.js:71 time_range end: 1.55
[...] 

if you notice, every time the buffer is updated, the start time remains 1.454 and the end time is increased depending on length of video data received. I will have to figure out a way to modify the gstreamer "start time" while streaming.

I've been working on MSE support on WebKit based browsers for a while (not Chromium, but the experience is applicable anyway) and, from my point of view, the way of growing buffered timeranges you're getting is normal. When you append data, it translates into buffered timeranges and they only grow (you're "buffering", after all).

On the other hand, when you start playing, the currentTime goes forward and makes past buffers obsolete. When the memory consumed by the buffers is too high, they are automatically garbage collected. This process works more or less like this (please somebody correct me if I'm mistaken):

A SourceBuffer (SourceBuffer.cpp) delegates ("uses", "relies on") on a WebSourceBuffer interface (WebSourceBuffer.h) implemented by WebSourceBufferImpl (websourcebuffer_impl.cc), which delegates on a ChunkDemuxer (chunk_demuxer.cc), which delegates on a SourceBufferStream (source_buffer_stream.cc). SourceBufferStream::Append() calls SourceBufferStream::GarbageCollectIfNeeded(), who checks if the memory limit has been reached and starts deleting ranges from the future, front and back until the memory is under limits again. SourceBufferStream::GetBufferedTime() will report less ranges now and that info will traverse up layers to ChunkDemuxer::GetBufferedRanges(), WebSourceBufferImpl::buffered() and SourceBuffer::buffered(). That's how the buffered data "expires" automatically.