autobahn websockets and html5 video

Chetan Naik

unread,

Mar 12, 2015, 4:53:52 AM3/12/15

to autob...@googlegroups.com

I am running a websocket server using autobahn|python.

On the server side, I also have a gstreamer pipeline running which I am using to capture webm frames using "appsink". The gstreamer pipeline that is implemented is:

gst-launch-1.0 v4l2src ! video/x-raw,width=640,height=480 ! videoconvert ! vp8enc ! webmmux ! appsink name="sink"

Everytime, I receive a buffer in the appsink, I send it over a websocket as a binary "message" using sendMessage.

def on_new_buffer(appsink):
    global once
    gstsample = appsink.emit('pull-sample')
    gstbuffer = gstsample.get_buffer()
    frame_data = gstbuffer.extract_dup(0,gstbuffer.get_size())
    for c in global_clients:
        c.sendMessage(frame_data,True)
        print("Directly sent: {0} bytes".format(len(frame_data)))

    return False

On the client side, I have a complicated flow of received frame_data blob. There is a FileReader, MediaSource and source buffer. whenever a frame_data is received, it is read as buffer using filereader. if the filereader is busy reading the previous frame_data, it will append it to "buffer_pool". once the frame_data is read as buffer, it is appended to "sourceBuffer". if the "sourceBuffer" is still updating the previous chunk, it will be appended to "sourceBufferpool".

var sourceBuffer;
    var buffer_pool = [];
    var sourceBufferpool = [];

    function setupVideo() {
        window.MediaSource = window.MediaSource || window.WebKitMediaSource;
        if (!!!window.MediaSource) {
            alert('MediaSource API is not available');
        }
        mediaSource = new MediaSource();
        video.src = window.URL.createObjectURL(mediaSource);
        mediaSource.addEventListener('sourceopen', function (e) {
            try {
                sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp8"');
            } catch(e) {
                console.log('Exception calling addSourceBuffer for video', e);
                return;
            }

            //sourceBuffer.addEventListener('updatestart', function(e) { console.log('updatestart: ' + e.target + mediaSource.readyState); });
            //sourceBuffer.addEventListener('updateend', function(e) { console.log('updateend: ' + e.target + mediaSource.readyState); });
            sourceBuffer.addEventListener('error', function(e) { console.log('error: ' + e.target + mediaSource.readyState); });
            sourceBuffer.addEventListener('abort', function(e) { console.log('abort: ' + e.target + mediaSource.readyState); });

            sourceBuffer.addEventListener('update', function() {
                if (sourceBufferpool.length > 0 && !sourceBuffer.updating) {
                    try {
                        sourceBuffer.appendBuffer(sourceBufferpool.shift());
                        console.log('update: pooled buffer appended ' + sourceBufferpool.length + mediaSource.readyState);
                    }catch(e){
                        console.log('Exception calling appendBuffer for video ', e);
                        return;
                    }
                }
            },false)

            if (video.paused) {
                video.play()
            }

            startWSStreaming();
        },false)

        mediaSource.addEventListener('sourceended', function(e) { console.log('sourceended: ' + mediaSource.readyState); });
        mediaSource.addEventListener('sourceclose', function(e) { console.log('sourceclose: ' + mediaSource.readyState); });
        mediaSource.addEventListener('error', function(e) { console.log('error: ' + mediaSource.readyState); });

    }

    function startWSStreaming() {
        var reader = new FileReader();

        reader.onload = function (evt) {
            if (sourceBuffer.updating || sourceBufferpool.length > 0){
                sourceBufferpool.push(new Uint8Array(evt.target.result));
                console.log('update: pooled buffer appended ' + sourceBufferpool.length + mediaSource.readyState);
            }else{
                sourceBuffer.appendBuffer(new Uint8Array(evt.target.result));
                console.log('update: direct buffer appended ' + sourceBufferpool.length + mediaSource.readyState);
            }
        }

        reader.onloadend = function (evt) {
            if (buffer_pool.length > 0) {
                var chunk = new Blob([buffer_pool.shift()], {type: 'video/webm'});
                evt.target.readAsArrayBuffer(chunk);
                console.log('Processed buffer pool: current size ' + buffer_pool.length);
            }
        }

        ws = new WebSocket("ws://localhost:9000/");
        ws.onopen = function () {
            document.getElementById("MSG1").innerHTML = 'Websocket opened <br>';
        }
        ws.onmessage = function(e) {
            myBuffer = e.data;
            if (reader.readyState == 1 || buffer_pool.length > 0) {
                buffer_pool.push(myBuffer);
                console.log('Received buffer pooled: current size ' + buffer_pool.length);
            }else{
                var chunk = new Blob([myBuffer], {type: 'video/webm'});
                reader.readAsArrayBuffer(chunk);
                console.log('First buffer processed');
            }
        }

    }
</script>

now, the end result is, I see only one frame on a browser window and then the video freezes. After checking chrome://media-internals/, I get the following clue:

Timestamp   Property    Value
00:00:00 00 pipeline_state  kCreated
00:00:00 00 EVENT   PIPELINE_CREATED
00:00:00 00 EVENT   WEBMEDIAPLAYER_CREATED
00:00:00 00 url blob:http%3A//localhost%3A8080/09060a78-9759-4fcd-97a2-997121ba6122
00:00:00 00 pipeline_state  kInitDemuxer
00:00:01 668    duration    unknown
00:00:01 669    pipeline_state  kInitVideoRenderer
00:00:01 685    pipeline_state  kPlaying
00:00:03 820    EVENT   PLAY
00:00:04 191    error   Got a block with a timecode before the previous block.
00:00:04 191    pipeline_error  pipeline: decode error
00:00:04 191    pipeline_state  kStopping
00:00:04 192    pipeline_state  kStopped
00:00:28 483    EVENT   WEBMEDIAPLAYER_DESTROYED

so the questions:

why is the video freezing after diplaying just one frame?
is it because of websocket "sendMessage" method, as I am treating webm chunks as distinct messages whereas, this should need treatment as "sendMessageFrameData"?
will I need some sequencing on the arrived frame_data so that they are received in the order they were sent?

please help!

Tobias Oberstein

unread,

Mar 15, 2015, 10:12:14 AM3/15/15

to autob...@googlegroups.com

Hi Chetan,

rgd your questions:

2) using sendMessage() is fine (do not use the sendMessageFrameData() API - this is for _WebSocket_ frames, not video frames .. 99.9% of all users don't need to go to that level).
3) messages sent via sendMessage() are received in-order on the browser side

Hope this helps,
/Tobias

Chetan Naik

unread,

Mar 16, 2015, 9:35:13 PM3/16/15

to autob...@googlegroups.com

Thanks Tobias! for your response.

I just happened to check that the code I wrote is working smoothly when I connect the client using Mozilla Firefox. the video feed is smooth. but the problem is with Chrome. For more details, please see http://stackoverflow.com/questions/29089786/mediasource-api-not-working-on-chrome-but-works-smoothly-on-firefox

In conclusion, there is no issue from the autobahn client (to feed live video chunks), autobahn server (to forward the websocket messages to the connected clients) and the JS client. The issue is on Chrome side. I will post my question on chrome community.

To let you know, i am using a python client with IPushProducer implementation and gstreamer pipeline to send the video. On the server side, again the IPushProducer model is used to forward the video frames for each of the connected client. I have become fan of Autobahn now. due to its simplicity and ease of developing the application. :)

Sylvain Hellegouarch

unread,

Mar 17, 2015, 4:10:09 PM3/17/15

to autob...@googlegroups.com

Hi Chetan,

2015-03-17 2:35 GMT+01:00 Chetan Naik <cheta...@gmail.com>:

Thanks Tobias! for your response.

I just happened to check that the code I wrote is working smoothly when I connect the client using Mozilla Firefox. the video feed is smooth. but the problem is with Chrome. For more details, please see http://stackoverflow.com/questions/29089786/mediasource-api-not-working-on-chrome-but-works-smoothly-on-firefox

In conclusion, there is no issue from the autobahn client (to feed live video chunks), autobahn server (to forward the websocket messages to the connected clients) and the JS client. The issue is on Chrome side. I will post my question on chrome community.

To let you know, i am using a python client with IPushProducer implementation and gstreamer pipeline to send the video. On the server side, again the IPushProducer model is used to forward the video frames for each of the connected client. I have become fan of Autobahn now. due to its simplicity and ease of developing the application. :)

This really looks fun. Will you share your code somewhere perhaps?

Thanks,

--

- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

Chetan Naik

unread,

Mar 22, 2015, 6:16:15 AM3/22/15

to autob...@googlegroups.com

https://code.google.com/p/chromium/issues/detail?id=469457&colspec=ID%20Pri%20M%20Week%20ReleaseBlock%20Cr%20Status%20Owner%20Summary%20OS%20Modified

I have created a new issue here with a JS client code attached...

Sylvain Hellegouarch

unread,

Mar 22, 2015, 6:19:09 AM3/22/15

to autob...@googlegroups.com

Thanks Chetan!

Is there a chance you shared the Python side as well?

--
You received this message because you are subscribed to the Google Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/198c105b-261c-4d03-ab37-082faa714c0b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Chetan Naik

unread,

Mar 22, 2015, 6:50:10 AM3/22/15

to autob...@googlegroups.com

OK, the overall architecture is like:

python websocket client <-----> python websocket server <-----> websocket client on browser

whenever, the browser client requests a live feed, the python websocket client will start running a gstreamer pipeline, and pass it on to the webserver. At the moment, this can work only with one python client and one browser client connected.

to simplify, I can send you a python implementation where websocket server hosts a web server on port 8020, and streams a video directly to websocket browser client, instead of getting the feed from a python client.

Reply all

Reply to author

Forward

Message has been deleted