Opus in WebM

[Brendan Bolles]

Hey everyone, I've added Opus support to my WebM plug-in for Premiere. I'm sure I'm doing some things wrong, not that there is an official way to do them right anyway! I'd love it if some knowledgeable people would take a look:

http://github.com/fnordware/AdobeWebM


Apparently the closest thing to a standards document is here:

http://wiki.xiph.org/MatroskaOpus


There are quite a few things here I can't yet do in libwebm (or can I?) like these SeekPreRoll, CodecDelay, and DiscardPadding fields.

Nor am I doing anything with CuePoints, which I kind of feel libwebm should handle for me transparently.

Would love some feedback!


Brendan

[Tom Finegan]

On Tue, Jan 7, 2014 at 11:34 AM, Brendan Bolles <bre...@fnordware.com> wrote:

Hey everyone, I've added Opus support to my WebM plug-in for Premiere.  I'm sure I'm doing some things wrong, not that there is an official way to do them right anyway!  I'd love it if some knowledgeable people would take a look:

http://github.com/fnordware/AdobeWebM


Apparently the closest thing to a standards document is here:

http://wiki.xiph.org/MatroskaOpus


There are quite a few things here I can't yet do in libwebm (or can I?) like these SeekPreRoll, CodecDelay, and DiscardPadding fields.

Hi Brendan,

Thanks for bringing this up! 

Libwebm HEAD now supports reading/writing:

- Opus/VP9

- SeekPreRoll

- CodecDelay

Reading of DiscardPadding via the parser is also working in HEAD, but the muxer side isn't complete (probably early next week unless something urgent comes up).

 

Nor am I doing anything with CuePoints, which I kind of feel libwebm should handle for me transparently.

Can you elaborate on this point? I'm not sure what you're asking for, feature wise, from libwebm.

[Brendan Bolles]

On Jan 10, 2014, at 4:17 PM, Tom Finegan wrote:

> Thanks for bringing this up!
>
> Libwebm HEAD now supports reading/writing:
> - Opus/VP9
> - SeekPreRoll
> - CodecDelay
>
> Reading of DiscardPadding via the parser is also working in HEAD, but the muxer side isn't complete (probably early next week unless something urgent comes up).

All right, great! I'll pull in your new changes and start using them.

>
>
> Nor am I doing anything with CuePoints, which I kind of feel libwebm should handle for me transparently.
>
>
> Can you elaborate on this point? I'm not sure what you're asking for, feature wise, from libwebm.

Well, I know there are these things called CuePoints in Matroska and libwebm has some functions for creating them and reading them. I've never actually seen a WebM that uses them, but I guess it's supposed to be good practice to do so?

By "transparently" I mean it seems like mkvparser::Track::Seek() should just use CuePoints if they're present. Right now I think it's up to me to use CuePoints properly.

As for writing them, is it as simple as calling mkvmuxer::Segment::AddCuePoint() after I add a frame? Even then, I'd think AddFrame() would just do it for me.


Brendan

[Tom Finegan]

On Sat, Jan 11, 2014 at 11:01 AM, Brendan Bolles <bre...@fnordware.com> wrote:

On Jan 10, 2014, at 4:17 PM, Tom Finegan wrote:

> Thanks for bringing this up!
>
> Libwebm HEAD now supports reading/writing:
> - Opus/VP9
> - SeekPreRoll
> - CodecDelay
>
> Reading of DiscardPadding via the parser is also working in HEAD, but the muxer side isn't complete (probably early next week unless something urgent comes up).

All right, great!  I'll pull in your new changes and start using them.

>
>
> Nor am I doing anything with CuePoints, which I kind of feel libwebm should handle for me transparently.
>
>
> Can you elaborate on this point? I'm not sure what you're asking for, feature wise, from libwebm.

Well, I know there are these things called CuePoints in Matroska and libwebm has some functions for creating them and reading them.  I've never actually seen a WebM that uses them, but I guess it's supposed to be good practice to do so? 

By "transparently" I mean it seems like mkvparser::Track::Seek() should just use CuePoints if they're present.  Right now I think it's up to me to use CuePoints properly.

Yes, you have to walk the cues to find the time you want, and then pass the time to mkvparser::Track::Seek(). Adding a bug to our issue tracker[1] requesting the ability to pass a cue point directly to Track::Seek() seems like a reasonable feature request. Would you mind creating a bug?

[1] https://code.google.com/p/webm/issues/entry?template=Feature%20Request

 

As for writing them, is it as simple as calling mkvmuxer::Segment::AddCuePoint() after I add a frame?  Even then, I'd think AddFrame() would just do it for me.

Libwebm will automatically generate cue points for video keyframes, unless:

- You've set mode to kLive via mkvmuxer::Segment::set_mode().

- Your IMkvWriter object does not support seeking (your implementation of IMkvWriter::Seekable() returns false).

In other words, AddFrame() adds cue points automatically for video key frames, but mkvmuxer::Segment must be in kFile mode. 

[Brendan Bolles]

On Jan 13, 2014, at 6:38 PM, Tom Finegan wrote:

> Yes, you have to walk the cues to find the time you want, and then pass the time to mkvparser::Track::Seek(). Adding a bug to our issue tracker[1] requesting the ability to pass a cue point directly to Track::Seek() seems like a reasonable feature request. Would you mind creating a bug?

Well, so what is the benefit of using the cue point if you can just send the time you want to Track::Seek() directly?

My calls to Cues::Find() always turn up empty because my Cues aren't loaded. What's the proper way to get them loaded? I see Segment::ParseCues(), but that requires I know the position of the cues data, I think.


Brendan

[Johann Koenig]

On Fri, Jan 17, 2014 at 10:58 AM, Brendan Bolles <bre...@fnordware.com> wrote:
> My calls to Cues::Find() always turn up empty because my Cues aren't loaded. What's the proper way to get them loaded? I see Segment::ParseCues(), but that requires I know the position of the cues data, I think.

Yes, there is a bit of extra work involved with cues. It would be nice
if the library handled this directly. See the Android code for one
example:
https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/matroska/MatroskaExtractor.cpp

Line 309: void BlockIterator::seek

[Brendan Bolles]

On Jan 17, 2014, at 11:26 AM, Johann Koenig wrote:

> Yes, there is a bit of extra work involved with cues. It would be nice
> if the library handled this directly. See the Android code for one
> example:
> https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/matroska/MatroskaExtractor.cpp

Ahh, nice. Thanks for the link.

Still wondering what the advantage is over just using Track::Seek(). Seems like either way you end up with the Cluster containing the time you asked for, but Track::Seek() does it without any of this CuePoint stuff. If CuePoints are a more efficient way of seeking, it seems like Track::Seek() would use them itself.


Brendan

[Johann Koenig]

On Fri, Jan 17, 2014 at 3:02 PM, Brendan Bolles <bre...@fnordware.com> wrote:
> Still wondering what the advantage is over just using Track::Seek(). Seems like either way you end up with the Cluster containing the time you asked for, but Track::Seek() does it without any of this CuePoint stuff. If CuePoints are a more efficient way of seeking, it seems like Track::Seek() would use them itself.

For the streaming case: you can jump straight to the location you want
without downloading all the intermediate data.

If you have the entire file you should still see faster seeks (using
cues) especially with larger files.

[Brendan Bolles]

On Jan 17, 2014, at 3:14 PM, Johann Koenig wrote:

> For the streaming case: you can jump straight to the location you want
> without downloading all the intermediate data.
>
> If you have the entire file you should still see faster seeks (using
> cues) especially with larger files.

Ahhh, streaming, of course. Outside my domain. I always have the whole file.

I'm still thinking Track::Seek() should take advantage of cues on its own. Any reason why it shouldn't?


Brendan

[Matthew Heaney]

On Fri, Jan 17, 2014 at 3:02 PM, Brendan Bolles <bre...@fnordware.com> wrote:

Still wondering what the advantage is over just using Track::Seek().  Seems like either way you end up with the Cluster containing the time you asked for, but Track::Seek() does it without any of this CuePoint stuff.  If CuePoints are a more efficient way of seeking, it seems like Track::Seek() would use them itself.

Track::Seek was written before we understood cue points.  Use Cues::Find, or use the iterator functions in Cues, for seeking.

[Matthew Heaney]

Just pretend Track::Seek doesn't exist.

--
You received this message because you are subscribed to the Google Groups "Application Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apps-devel+...@webmproject.org.
To post to this group, send email to apps-...@webmproject.org.
Visit this group at http://groups.google.com/a/webmproject.org/group/apps-devel/.
For more options, visit https://groups.google.com/a/webmproject.org/groups/opt_out.

[Brendan Bolles]

On Jan 17, 2014, at 4:39 PM, Matthew Heaney wrote:

> Just pretend Track::Seek doesn't exist.
>

OK, but shouldn't the contents of that Android BlockIterator::seek() function go into libwebm function somewhere then? And actually, that function should probably be Track::Seek().

Guess I'm the only one who thinks so.


Brendan

[Tom Finegan]

Another update on Opus:

- Libwebm DiscardPadding support on the muxer side just landed.

And, re cue points:

- Parser sample now demonstrates basic cue point walking.

The latter is not entirely relevant; the seeking aspect that brought them into this conversation in the first place is not part of the update, but the lack of any attention whatsoever re cue points in the parser sample was bothering me.

Brendan

[Brendan Bolles]

On Jan 23, 2014, at 3:47 PM, Tom Finegan wrote:

> Another update on Opus:
> - Libwebm DiscardPadding support on the muxer side just landed.
>
> And, re cue points:
> - Parser sample now demonstrates basic cue point walking.
>
> The latter is not entirely relevant; the seeking aspect that brought them into this conversation in the first place is not part of the update, but the lack of any attention whatsoever re cue points in the parser sample was bothering me.

Thanks, Tom. I've pulled in your latest.

Any reason why AddFrameWithDiscardPadding() can't take discard_padding=0 and behave the same as AddFrame() in that case?


Thanks also for adding some cue point code in the sample. In WebM, does every keyframe have a cue point? And does every keyframe start a new cluster?


Brendan

[Tom Finegan]

On Fri, Jan 24, 2014 at 11:54 AM, Brendan Bolles <bre...@fnordware.com> wrote:

On Jan 23, 2014, at 3:47 PM, Tom Finegan wrote:

> Another update on Opus:
> - Libwebm DiscardPadding support on the muxer side just landed.
>
> And, re cue points:
> - Parser sample now demonstrates basic cue point walking.
>
> The latter is not entirely relevant; the seeking aspect that brought them into this conversation in the first place is not part of the update, but the lack of any attention whatsoever re cue points in the parser sample was bothering me.

Thanks, Tom.  I've pulled in your latest.

Any reason why AddFrameWithDiscardPadding() can't take discard_padding=0 and behave the same as AddFrame() in that case?

None in particular excepting that the AddFrameX() variants all do specific things, and fail for unmet conditions. If you were looking for something more general I would use AddGenericFrame(). Granted, AddGenericFrame() currently comes with the penalty of an extra copy, but at least it's a single point of entry for adding "frames" to the output stream.

 

Thanks also for adding some cue point code in the sample.  In WebM, does every keyframe have a cue point?

Cues are optional. Generally speaking you'll have a cue point for the first keyframe in each cluster when a file has cue points. There could be more.

 

 And does every keyframe start a new cluster?

Keyframes should be placed at the beginning of clusters per the WebM container guidelines at http://www.webmproject.org/docs/container/

Note, as implied above, you can have multiple keyframes in a cluster. However, if at all possible, starting a new cluster for each video keyframe is desired.