about dynamic muxing for vpx/vorbis encoded data on buffer basis, instead of file basis

244 views
Skip to first unread message

Joony

unread,
Aug 7, 2012, 11:49:42 PM8/7/12
to WebM Discussion
Hi. All.

I completed only live webm video streaming in connection with steram-m
server project with no ffmpeg. Now, I started to work on dynamic webm
muxing vpx/vorbis encoded stream from webcam/microphone. But all
examples that comes with vpx/vorbis and others in webm project, most
of them are file-based. for webm only video streamin, I referred to
libwebmvv and libvpx-v1.1.0\libmkv.

while I was keeping to read some on webm project site, I found an
interesting code repository in Webm code repository in the following.

git clone https://code.google.com/p/webm.webmlive/

after checked out, I looked into the code directory. there is no
README file to understand get general idea of the code functionality.

My question is quite simple. I would like to get help about how to
approach for dynamic muxing for both vorbis/vpx encoded data or maybe
some good starting points. I hope that I can hear some from anyone.

Oleksij Rempel

unread,
Aug 8, 2012, 9:21:12 AM8/8/12
to webm-d...@webmproject.org
What do you mean with dynamic muxing?

Hyoung joon Jun

unread,
Aug 8, 2012, 10:06:14 AM8/8/12
to webm-d...@webmproject.org
I mean more likely muxing vorbis/vpx encoded data dynamically in real time on the buffer. is there any suggested starting point I can approach for that ?

Thanks & regards
joon



On Wed, Aug 8, 2012 at 9:21 PM, Oleksij Rempel <lexa....@gmail.com> wrote:
What do you mean with dynamic muxing?

--
You received this message because you are subscribed to the Google Groups "WebM Discussion" group.
To post to this group, send email to webm-d...@webmproject.org.
To unsubscribe from this group, send email to webm-discuss...@webmproject.org.
For more options, visit this group at http://groups.google.com/a/webmproject.org/group/webm-discuss/?hl=en.




--
Thank you
Best regards
joon


Tom Finegan

unread,
Aug 8, 2012, 11:38:32 AM8/8/12
to webm-d...@webmproject.org
The v2 branch of webmlive can be used as an example of live vp8/vorbis encoding, but the final patch for audio support hasn't landed yet (and won't in the near future, because I've been working on other projects). 

In brief, webmlive works like this:

1) DirectShow feeds uncompressed audio and video samples to the audio and video sink filters.
2) The a/v sink filters push data into buffer pool objects.
3) The a/v encode thread pulls data from the buffer pools, encodes the data using libvpx and libvorbis, and feeds the compressed samples to libwebm.
4) (still in the encode thread) When a cluster is completed, the encode thread hands the completed cluster off to a data sink interface attached to a HTTP uploader that runs in another thread.

webmlive v2 gitweb: 

audio encoding patchset (still under review, not perfect):

On the other hand, the master branch contains a working live encoding example. I wouldn't use that, though. It's a prototype version that relies completely on DirectShow, and uses a file reader object to read data written to disk after encode/mux/writing to disk takes place within the DirectShow filter graph.

Joony

unread,
Aug 23, 2012, 9:26:58 AM8/23/12
to WebM Discussion
Hi. Tom and All

Thanks for your kind and detailed explanation. After carefully reading
your comment, although webmlive is working prototype with limitation
and depedency on directshow, I think I can use some of webmlive
classes for my project. Since I need a webm parser to buffer, not
file, I did little modification and added "WebmBufferParser" and
"WebmBufferReader" class to my project code; mostly about "access
specifier" classes and pointer to keep track of segment, tracks and so
on.


the following is my approach

webmlive::WebmBufferParser videoBufferParser;
webmlive::WebmBufferWriter writer; // curretnly abstract class object.

while(1) {
....
/* get vorbis encoded data from real-time buffer */
....
/* get vpx encoded data from real-time buffer */
getVideoStreamBuf( &retPtrToVideo, &videoLength);
if (retPtrToVideo ) {
  for( int i =0; i < videoLength; ++i)
     videoBuffer.push_back( (uint8)retPtrToVideo[i] );
  if ( retPtrToVideo )
     videoBufferParser.Parse( videoBuffer, &videoLength );
... // some other codes to related to muxing.
... // I also added buffer deletion function. but currently commented
out to look into "Parse"
}


while I walk through the code, it looks that key processes about
"Parse" function can be described in the following

1. SetBufferWindow(&buf[0], buf.size(), total_bytes_parsed_)
2. WebmBufferParser::ParseSegmentHeaders(int32* ptr_element_size)
3. WebmBufferParser::ParseCluster(int32* ptr_element_size) {
      ...
      status = segment_->LoadCluster(current_pos, length);
      if (status) {
         return kNeedMoreData; // returns all the time although new
cluster starts again
      }
      ...
   }


"Segment::LoadCluster" function invokes "Segment::DoLoadCluster". The
function returns non-zero value all the time along with my code.
While I am reading "DoLoadCluster" function, there is commented out
code "~~~ to support live webm" and it keeps returing
"E_FILE_FORMAT_INVALID" although my code generates new cluster every
33 seconds [webm mux guideline]. The file dumped from the buffer play
well in VLC player even though it does not carry the size of segment
for live webm.


long Segment::DoLoadCluster(
    long long& pos,
    long& len)
{
...
#if 0  //we must handle this to support live webm
        if (size == unknown_size)
            return E_FILE_FORMAT_INVALID;  //TODO: allow this
#endif
...
        if (id != 0x0F43B675)  //Cluster ID
        {
            if (size == unknown_size)
                return E_FILE_FORMAT_INVALID;  //TODO: liberalize
            m_pos = pos + size;  //consume payload
            continue;
        }
...
}

I am very curious with what I have missed and if I still can continue
in this approach. any comment will be welcome and appreciated.



On Aug 8, 11:38 pm, Tom Finegan <tomfine...@google.com> wrote:
> The v2 branch of webmlive can be used as an example of live vp8/vorbis
> encoding, but the final patch for audio support hasn't landed yet (and
> won't in the near future, because I've been working on other projects).
>
> In brief, webmlive works like this:
>
> 1) DirectShow feeds uncompressed audio and video samples to the audio and
> video sink filters.
> 2) The a/v sink filters push data into buffer pool objects.
> 3) The a/v encode thread pulls data from the buffer pools, encodes the data
> using libvpx and libvorbis, and feeds the compressed samples to libwebm.
> 4) (still in the encode thread) When a cluster is completed, the encode
> thread hands the completed cluster off to a data sink interface attached to
> a HTTP uploader that runs in another thread.
>
> webmlive v2 gitweb:http://git.chromium.org/gitweb/?p=webm/webmlive.git;a=shortlog;h=refs...
>
> audio encoding patchset (still under review, not perfect):https://gerrit.chromium.org/gerrit/#/c/22771/
>
> On the other hand, the master branch contains a working live encoding
> example. I wouldn't use that, though. It's a prototype version that relies
> completely on DirectShow, and uses a file reader object to read data
> written to disk after encode/mux/writing to disk takes place within the
> DirectShow filter graph.
>
> On Wed, Aug 8, 2012 at 7:06 AM, Hyoung joon Jun <hi.jo...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I mean more likely muxing vorbis/vpx encoded data dynamically in real time
> > on the buffer. is there any suggested starting point I can approach for
> > that ?
>
> > Thanks & regards
> > joon
>
> > On Wed, Aug 8, 2012 at 9:21 PM, Oleksij Rempel <lexa.fis...@gmail.com>wrote:
>
> >> What do you mean with dynamic muxing?
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "WebM Discussion" group.
> >> To post to this group, send email to webm-disc...@webmproject.org.
> >> To unsubscribe from this group, send email to
> >> webm-discuss+unsubscr...@webmproject.org.
> >> For more options, visit this group at
> >>http://groups.google.com/a/webmproject.org/group/webm-discuss/?hl=en.
>
> > --
> > *Thank you
> > Best regards
> > joon
> > *
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "WebM Discussion" group.
> > To post to this group, send email to webm-disc...@webmproject.org.
> > To unsubscribe from this group, send email to
> > webm-discuss+unsubscr...@webmproject.org.

Tom Finegan

unread,
Aug 23, 2012, 11:24:07 AM8/23/12
to webm-d...@webmproject.org
On Thu, Aug 23, 2012 at 6:26 AM, Joony <hi.j...@gmail.com> wrote:
[...] 


long Segment::DoLoadCluster(
    long long& pos,
    long& len)
{
        ...
#if 0  //we must handle this to support live webm
        if (size == unknown_size)
            return E_FILE_FORMAT_INVALID;  //TODO: allow this
#endif
        ...
        if (id != 0x0F43B675)  //Cluster ID
        {
            if (size == unknown_size)
                return E_FILE_FORMAT_INVALID;  //TODO: liberalize
            m_pos = pos + size;  //consume payload
            continue;
        }
        ...
}

I am very curious with what I have missed and if I still can continue
in this approach. any comment will be welcome and appreciated.


Let's start simple-- the above code is from an old version of libwebm. The first thing you should do is download newer source that has support for handling WebM files contained segments with unknown sizes. I've gone five months back and the code above isn't there. Not sure how old it is, but upgrading to the latest version should improve the situation. What version are you actually using?

Tom
 

Joony

unread,
Aug 24, 2012, 2:58:43 AM8/24/12
to WebM Discussion
The version of libwebm I am using is 1.0.0.5 in "RELEASE.TXT"

1.0.0.5
 * Handled case when no duration
 * Handled empty clusters
 * Handled empty clusters when seeking
 * Implemented check lacing bits


link 1. http://code.google.com/p/webm/source/browse?repo=libwebm
The webm.libwebm I downloaded through git has the same version number,
1.0.0.5 in "RELEASE.TXT". I think I use almost same code or no one
updates RELEASE.TXT. somewhat confusing to me.


link 2. http://git.chromium.org/gitweb/?p=webm/libwebm.git
So. I got another git clone from this link above. "RELEASE.TXT" does
not show up no more than 1.0.0.5 and it shows the following code.


#if 0  //we must handle this to support live webm
        if (size == unknown_size)
            return E_FILE_FORMAT_INVALID;  //TODO: allow this
#endif


Can you give some specific git link for me to download or refer to,
please if you don't mind it ? What is the best way to distinguish
latest libwebm ?


Regards
joon



On Aug 23, 11:24 pm, Tom Finegan <tomfine...@google.com> wrote:

Matthew Heaney

unread,
Aug 24, 2012, 7:16:34 PM8/24/12
to webm-d...@webmproject.org
What version of mkvparser are you using? (Inspect mkvparser::GetVersion()).
> --
> You received this message because you are subscribed to the Google Groups "WebM Discussion" group.
> To post to this group, send email to webm-d...@webmproject.org.
> To unsubscribe from this group, send email to webm-discuss...@webmproject.org.

Joony

unread,
Aug 26, 2012, 12:04:06 PM8/26/12
to WebM Discussion

Hi. Matthew and all

Thanks for your comment. I checked on the version. "mkvparser.cpp"
shows in the following.

void mkvparser::GetVersion(int& major, int& minor, int& build, int&
revision)
{
major = 1;
minor = 0;
build = 0;
revision = 25;
}

It looks that I am using version 25. I confirmed that My VS2010 is
linked with "libwebm" in the correct library directory and
mkvparser.h and others in the correct header file directory.


on first invocation of DoLoadCluster, it does not go into "return
E_FILE_FORMAT_INVALID"
on second invocation of DoLoadCluster,it goes into "return
E_FILE_FORMAT_INVALID"

long Segment::DoLoadCluster(
long long& pos,
long& len)
{
...
for (;;)
{
...
if (id != 0x0F43B675) //Cluster ID
{
if (size == unknown_size)
return E_FILE_FORMAT_INVALID; //TODO:
liberalize // keep falling into

m_pos = pos + size; //consume payload
continue;
}
...
}
...
}

Do I need to set something for parser like
set_mode(mkvmuxer::Segment::Mode::kLive) of mkvmuxer ?

Thanks & regards
joon


On Aug 25, 7:16 am, Matthew Heaney <matthewjhea...@google.com> wrote:
> What version of mkvparser are you using?  (Inspect mkvparser::GetVersion()).
>
>
>
>
>
>
>
> On Thu, Aug 23, 2012 at 11:58 PM, Joony <hi.jo...@gmail.com> wrote:
> > The version of libwebm I am using is 1.0.0.5 in "RELEASE.TXT"
>
> > 1.0.0.5
> >  * Handled case when no duration
> >  * Handled empty clusters
> >  * Handled empty clusters when seeking
> >  * Implemented check lacing bits
>
> > link 1.http://code.google.com/p/webm/source/browse?repo=libwebm
> > The webm.libwebm I downloaded through git has the same version number,
> > 1.0.0.5 in "RELEASE.TXT". I think I use almost same code or no one
> > updates RELEASE.TXT. somewhat confusing to me.
>
> > link 2.http://git.chromium.org/gitweb/?p=webm/libwebm.git
> > To post to this group, send email to webm-disc...@webmproject.org.
> > To unsubscribe from this group, send email to webm-discuss+unsubscr...@webmproject.org.

Joony

unread,
Aug 26, 2012, 10:52:32 PM8/26/12
to WebM Discussion
I just checked on this code.

------------------------------------------------- webmlive
int WebmBufferParser::ParseSegmentHeaders(int32* ptr_element_size) {
...
// Get the segment tracks to obtain its length.
ptr_tracks = const_cast<mkvparser::Tracks*>(segment_->GetTracks());
...
}

this function only returns "m_pTracks".
---------------------------------------------- mkvparser.cpp
const Tracks* Segment::GetTracks() const {
return m_pTracks;
}


"ptr_tracks" looks not correct to me; I tried to step in and out. but,
"ptr_tracks" are initialized with the following values.

ptr_tracks 0x0ce31ef0 {m_pSegment=0x0ce30e68 m_start=123
m_size=4425 ...} mkvparser::Tracks *
m_pSegment 0x0ce30e68 {m_pReader=0x035d8c30 m_element_start=43
m_start=55 ...} mkvparser::Segment * const
m_start
123 const
__int64
m_size
4425 const
__int64
m_element_start
111 const
__int64
m_element_size
4437 const
__int64


so I dive into "Segment::ParseHeaders()"

else if (id == 0x0654AE6B) //Tracks ID
{
if (m_pTracks)
return E_FILE_FORMAT_INVALID;
m_pTracks = new (std::nothrow) Tracks(this, pos, size,
element_start, element_size);
...
}


"m_pTracks" is instantiated with these values above code.
pos 123 __int64
size 4425 const __int64
element_start 111 const __int64
element_size 4437 const __int64


I am not very clear if size(4425) and element_size(4437) value are
correct or not. I think EBML header and all other basic info(segment,
track and so) should be written within around 200~ 240 bytes before
the first cluster starts.

Matthew Heaney

unread,
Aug 27, 2012, 3:58:35 PM8/27/12
to webm-d...@webmproject.org
On Sun, Aug 26, 2012 at 9:04 AM, Joony <hi.j...@gmail.com> wrote:
>
> on first invocation of DoLoadCluster, it does not go into "return
> E_FILE_FORMAT_INVALID"
> on second invocation of DoLoadCluster,it goes into "return
> E_FILE_FORMAT_INVALID"

It looks like you are parsing a webm stream, and you detected an MKV
element whose size is unknown, but the element itself is not a Cluster
element. Hmmmm...

Can you dump the stream to a file (enough of the stream to duplicate
the behavior you are observing) and then send me a link to the file?

Joony

unread,
Aug 27, 2012, 11:18:27 PM8/27/12
to WebM Discussion
Hi. Matthew

I uploaded the two dump files from the buffers for both audio and
video stream. both has recorded for around two minutes to get couple
of clusters although "mkvinfo GUI" does not show number of clusters.
simeple muxer example does not parse well for unknown segment size.

transcodeAudioDump.webm
https://docs.google.com/open?id=0BwdHLqwtsfb2QWNnVlpIRk44TFE

transcodeVideoDump.webm
https://docs.google.com/open?id=0BwdHLqwtsfb2T1BCaktKZTMtTWs

This file is generated by sample_muxer.cpp in libwebm. I changed it
from original sample code for file-based two inputs.
TranscodeVideoDump.webm
https://docs.google.com/open?id=0BwdHLqwtsfb2cjc5NTcwSHlNT0k


My main goal is I am trying to mux both buffer-streams which are
generated by my codes in real-time. I followed webm muxer guideline
( cluster 32.xx seconds limitation, no cue info, unknown segment size
and so )

What I understand is that cluster size info should be written after a
cluster ends and webm writer comes back to the beginning of the
cluster.

if I am not correct, please, correct me any time.

Regards
joon



On Aug 28, 3:58 am, Matthew Heaney <matthewjhea...@google.com> wrote:

Joony

unread,
Aug 28, 2012, 1:07:12 PM8/28/12
to WebM Discussion
I am correcting a link for a muxed file.
TranscodeAVDump_output.webm
https://docs.google.com/open?id=0BwdHLqwtsfb2U3BVQjFweG54djQ


On Aug 28, 11:18 am, Joony <hi.jo...@gmail.com> wrote:
> Hi. Matthew
>
> I uploaded the two dump files from the buffers for both audio and
> video stream. both has recorded for around two minutes to get couple
> of clusters although "mkvinfo GUI" does not show number of clusters.
> simeple muxer example does not parse well for unknown segment size.
>
> transcodeAudioDump.webmhttps://docs.google.com/open?id=0BwdHLqwtsfb2QWNnVlpIRk44TFE
>
> transcodeVideoDump.webmhttps://docs.google.com/open?id=0BwdHLqwtsfb2T1BCaktKZTMtTWs
>
> This file is generated by sample_muxer.cpp in libwebm. I changed it
> from original sample code for file-based two inputs.
> TranscodeVideoDump.webmhttps://docs.google.com/open?id=0BwdHLqwtsfb2cjc5NTcwSHlNT0k

Matthew Heaney

unread,
Aug 28, 2012, 2:38:49 PM8/28/12
to webm-d...@webmproject.org
On Tue, Aug 28, 2012 at 10:07 AM, Joony <hi.j...@gmail.com> wrote:
> I am correcting a link for a muxed file.
> TranscodeAVDump_output.webm
> https://docs.google.com/open?id=0BwdHLqwtsfb2U3BVQjFweG54djQ

All 3 of these files can be parsed using sample.cpp without error.
The A/V file also renders for me in Chrome.

What is the problem that you're having?

Joony

unread,
Aug 28, 2012, 10:40:44 PM8/28/12
to WebM Discussion
The problem is that I can not parse two audio/video buffer stream(not
on files) using webmlive::WebmBufferParser. From some unknown reason,
"Segment::DoLoadCluster" does not detect cluster in above thread. is
there any clue to fix this problem I am facing up?

Thanks & regards
joon



On Aug 29, 2:38 am, Matthew Heaney <matthewjhea...@google.com> wrote:

Tom Finegan

unread,
Aug 29, 2012, 12:35:36 PM8/29/12
to webm-d...@webmproject.org
On Tue, Aug 28, 2012 at 7:40 PM, Joony <hi.j...@gmail.com> wrote:
The problem is that I can not parse two audio/video buffer stream(not
on files) using webmlive::WebmBufferParser. From some unknown reason,
"Segment::DoLoadCluster" does not detect cluster in above thread. is
there any clue to fix this problem I am facing up?


In my initial email I mentioned that the master branch of webmlive is a prototype. WebmBufferParser is a hack that is intended to parse data read from disk after the muxing and file writing are done by the WebM project DirectShow filters. The prototype version of webmlive muxes the data using the dshow filters, and then parses the freshly muxed data after reading it into a buffer-- this is inefficient (and a little silly). 

The v2 branch uses libwebm's muxer directly. The muxer exposes the IMkvWriter interface. IMkvWriter has a method called ElementStartNotify, which can be used to detect when new clusters are created. If your goal is to write a client encoder application that produces WebM clusters for transmission to a server, you should use IMkvWriter::ElementStartNotify to detect when a new cluster is created by the muxer. Whenever a new cluster is created, the previous cluster can be sent to the server.

As I mentioned before, audio support for webmlive v2 is still under review. Another item of note is that webmlive currently builds only with Visual Studio 2008. To obtain the latest version of webmlive's v2 branch, including patches still being reviewed, first checkout the v2 branch, and then merge in the latest patchset:

$ cd path/to/webmlive
$ git checkout v2
$ git fetch https://gerrit.chromium.org/gerrit/webm/webmlive refs/changes/80/23680/4
$ git merge --ff FETCH_HEAD

After git merge finishes you'll have a working example of a client WebM encoder that sends WebM data to an HTTP server via HTTP POST. The classes you're probably most interested in are WebmMuxWriter and LiveWebmMuxer, which are both implemented in webm_mux.cc. The main encode loop is WebmEncoder::EncodeThread, and that's implemented in webm_encoder.cc.

Back on the topic of WebmBufferParser-- this class will eventually be removed from webmlive. It is no longer maintained, and is not necessary for a WebM client encoder application (although it might see some use for testing purposes). 

Hope the above helps,
Tom

Joony

unread,
Aug 29, 2012, 11:30:24 PM8/29/12
to WebM Discussion
Hi. Tom.

I thank for your kind, detailed and long explanation. That is good
enough for me to work on my project along with v2 with fetch code you
mentioned.

However, the link, "git fetch https://gerrit.chromium.org/gerrit/webm/webmliverefs/changes/80/23680/4"
looks not valid or inaccessible with the following error message on my
cygwin

$ git fetch  https://gerrit.chromium.org/gerrit/webm/webmliverefs/
changes/80/23680/4
fatal: remote error: Git repository not found

can you validate it, please, if you don't mind it ?


Regards
Joon

On Aug 30, 12:35 am, Tom Finegan <tomfine...@google.com> wrote:

Tom Finegan

unread,
Aug 30, 2012, 12:32:18 PM8/30/12
to webm-d...@webmproject.org
On Wed, Aug 29, 2012 at 8:30 PM, Joony <hi.j...@gmail.com> wrote:
Hi. Tom.

I thank for your kind, detailed and long explanation. That is good
enough for me to work on my project along with v2 with fetch code you
mentioned.

However, the link, "git fetch https://gerrit.chromium.org/gerrit/webm/webmliverefs/changes/80/23680/4"
looks not valid or inaccessible with the following error message on my
cygwin

$ git fetch  https://gerrit.chromium.org/gerrit/webm/webmliverefs/
changes/80/23680/4

There should be a space between webmlive and refs:

$ git fetch https://gerrit.chromium.org/gerrit/webm/webmlive refs/changes/80/23680/4

In this case, gmail did not wrap a long line-- it really is two separate strings. :)
 
--
You received this message because you are subscribed to the Google Groups "WebM Discussion" group.
To post to this group, send email to webm-d...@webmproject.org.
To unsubscribe from this group, send email to webm-discuss...@webmproject.org.
Reply all
Reply to author
Forward
0 new messages