What formats do we want to support for MediaEncoder?

[Chris Pearce]

I saw that C.J filed bug 883749:

"[MediaEncoder] Implement Vorbis/ MP3 encoding format"
https://bugzilla.mozilla.org/show_bug.cgi?id=883749

I think it would be good to have a discussion as to what formats we're
going to support in MediaEncoder.

In terms of audio formats, Shelly and Randy already been working on an
Ogg/Opus encoder.

MP3 encoding seems like a good idea as MP3 has widespread support. MP3
isn't good at streaming cases, but Opus and Vorbis are. MP3 also isn't
good at low latency.

Vorbis is less clear cut. Opus is technically better than Vorbis, Opus
is better at low latency encoding than Vorbis, and Opus also royalty
free. Vorbis does not have as widespread adoption as MP3. So if we want
to only implement one audio-only royalty free format to discourage
format proliferation, then perhaps we should focus on only supporting Opus?

In terms of video, since we're focusing on mobile/FxOS, it seems to me
that H.264/AAC might give us hardware accelerated encoding, and
widespread support.

It would be good to support a royalty free video format for encoding,
but the only real candidate is VP8/Vorbis/WebM. I've been led to believe
that VP9/Opus/Webm2 isn't really viable yet as VP9 encoding is very slow.

What are people's thoughts?


Cheers,
Chris Pearce

[Chris Double]

On Tue, Jun 18, 2013 at 10:19 AM, Chris Pearce <cpe...@mozilla.com> wrote:
>
> I think it would be good to have a discussion as to what formats we're going
> to support in MediaEncoder.

What platforms are we supporting this on? Assuming it's all of them,
what's the status of MP3 encoding in system libraries? Is it available
on XP/Vista/OS X, etc?

I agree about Vorbis vs Opus but if we want to do WebM files then we'd
need Vorbis until VP9/Opus is more widespreead.

Chris.
--
http://www.bluishcoder.co.nz

[Ralph Giles]

On 13-06-17 3:19 PM, Chris Pearce wrote:

> MP3 encoding seems like a good idea as MP3 has widespread support. MP3
> isn't good at streaming cases, but Opus and Vorbis are. MP3 also isn't
> good at low latency.

MP3 is higher bitrate than Vorbis, but otherwise the formats are about
the same as far as streaming support and latency for technical concerns.

MP3-licensing.com currently collects per-stream royalties, I presume to
encourage migration to AAC streaming. MP3 will be a royalty-free format
in just a couple of years, so we should choose to support mp3 over AAC
for audio files whenever we can.

> Vorbis is less clear cut. Opus is technically better than Vorbis, Opus
> is better at low latency encoding than Vorbis, and Opus also royalty
> free. Vorbis does not have as widespread adoption as MP3. So if we want
> to only implement one audio-only royalty free format to discourage
> format proliferation, then perhaps we should focus on only supporting Opus?

I think we should concentrate on Opus regardless, since it has a better
chance for wide adoption than Vorbis.

> In terms of video, since we're focusing on mobile/FxOS, it seems to me
> that H.264/AAC might give us hardware accelerated encoding, and
> widespread support.

These codecs, with the mp4 container, are something we've committed to
supporting, so for a non-free format for interoperation with Apple and
MS devices this is a good choice.

> It would be good to support a royalty free video format for encoding,
> but the only real candidate is VP8/Vorbis/WebM. I've been led to believe
> that VP9/Opus/Webm2 isn't really viable yet as VP9 encoding is very slow.

If we implement an encoder for VP8/Vorbis/WebM, we might as well support
Vorbis-in-Ogg for audio-only as well. The latter is universally
supported on Android, unlike Opus.

What about VP8/Opus/WebM as a royalty-free video format? These are the
codecs we're using for WebRTC, so recording w/o having to run the
compression engine is possible. It sounds like the WebM project intends
to add both Opus and VP9 to "WebM" without any difference in branding.
So while VP8/Opus files wouldn't play in any older Firefox, Android or
Chrome builds, they would play in new ones going forward. I think this
is our best choice for a royalty-free video format until we have a
better handle on VP9.

-r

[David Richards]

Something that might be informative for vp8 encoding speed is krad cam, a little android app that was recently cobbled together that allows you to create a webm stream. It can be had here: http://websca.ws/krcam.apk

I would be in favor of supporting only Opus for audio-only streams, but I won't cry about other choices.

As for VP9, it is dreadfully slow and unusable at this time, the only good news is that going from supporting vp8 to vp8 and vp9 should just be a few lines.

Thanks,

David

vp8 decoding and encoding benchmark for android: https://play.google.com/store/apps/details?id=ws.websca.benchscaw&hl=en

----- Original Message -----
From: "Chris Pearce" <cpe...@mozilla.com>
To: mozilla-...@lists.mozilla.org
Sent: Monday, June 17, 2013 3:19:04 PM
Subject: What formats do we want to support for MediaEncoder?

I saw that C.J filed bug 883749:

"[MediaEncoder] Implement Vorbis/ MP3 encoding format"
https://bugzilla.mozilla.org/show_bug.cgi?id=883749

I think it would be good to have a discussion as to what formats we're
going to support in MediaEncoder.

In terms of audio formats, Shelly and Randy already been working on an
Ogg/Opus encoder.

MP3 encoding seems like a good idea as MP3 has widespread support. MP3
isn't good at streaming cases, but Opus and Vorbis are. MP3 also isn't
good at low latency.

Vorbis is less clear cut. Opus is technically better than Vorbis, Opus
is better at low latency encoding than Vorbis, and Opus also royalty
free. Vorbis does not have as widespread adoption as MP3. So if we want
to only implement one audio-only royalty free format to discourage
format proliferation, then perhaps we should focus on only supporting Opus?

In terms of video, since we're focusing on mobile/FxOS, it seems to me
that H.264/AAC might give us hardware accelerated encoding, and
widespread support.

It would be good to support a royalty free video format for encoding,
but the only real candidate is VP8/Vorbis/WebM. I've been led to believe
that VP9/Opus/Webm2 isn't really viable yet as VP9 encoding is very slow.

What are people's thoughts?


Cheers,
Chris Pearce

_______________________________________________
dev-media mailing list
dev-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-media

[David Richards]

Oh if anyone tries krad cam, the streams show up here: http://europa.kradradio.com:8008

Thanks,

David

[Chris Pearce]

On 6/18/2013 10:56 AM, Chris Double wrote:
> On Tue, Jun 18, 2013 at 10:19 AM, Chris Pearce <cpe...@mozilla.com> wrote:
>> I think it would be good to have a discussion as to what formats we're going
>> to support in MediaEncoder.
> What platforms are we supporting this on? Assuming it's all of them,
> what's the status of MP3 encoding in system libraries? Is it available
> on XP/Vista/OS X, etc?

Windows Media Foundation's MP3 encoder is only available on Windows 8,
and DirectShow doesn't ship with an MP3 encoder on other Windows
platforms, so we'd need to find another solution on Windows.

And WMF's H.264/AAC/MP4 encoder is only available on Windows 7+, so we'd
need to find another solution for MP4 encoding on earlier versions of
Windows too.


Chris P.

[sl...@mozilla.com]

Hi everyone,

I like the idea of supporting VP9, but with limit hardware resources on mobile devices, performance is a critical concern on FirefoxOS. Does that make H.264/AAC be the next supported format?

-Shelly

[Henri Sivonen]

On Tue, Jun 18, 2013 at 1:19 AM, Chris Pearce <cpe...@mozilla.com> wrote:
> I saw that C.J filed bug 883749:
>
> "[MediaEncoder] Implement Vorbis/ MP3 encoding format"
> https://bugzilla.mozilla.org/show_bug.cgi?id=883749

This bug doesn't say what the use cases are. :-(

Knowing use cases would help inform what sort of compatibility is needed.

> MP3 encoding seems like a good idea as MP3 has widespread support.

AAC has widespread support, too, among MPEG-oriented vendors. Do the
cases where MP3 is supported but AAC not matter? As for royalties,
which is preferable: MP3 expiring sooner (and the free Fluendo decoder
available for desktop Linux) or AAC not having use fees?

--
Henri Sivonen
hsiv...@hsivonen.fi
http://hsivonen.iki.fi/

[c...@mozilla.com]

I had separate bug 883749 into two bugs
https://bugzilla.mozilla.org/show_bug.cgi?id=883749 (Vobis)
https://bugzilla.mozilla.org/show_bug.cgi?id=884162 (MP3)

1. Vobis encoder:
For audio only encoding, I think we do not really need to support Vobis, at least for this moment. Take video encoding into account, we need vobis for "VP8/Vobis/WebM" video format.

2. H.264/AAC/MP4
From performance/ power consumption perspective, "H.264/AAC/MP4" is still the best choice for B2G, for now, since HW codec is wildly supported on most devices.
In video encoder, I think the priority could be
On B2G/ Android:
"H.264/AAC/MP4" > "VP8/Vobis/WebM"/"VP9/Opus/WebM"
On all the other platform
"VP8/Vobis/WebM"/"VP9/Opus/WebM" > "H.264/AAC/MP4"

3. mp3 or m4a, or none of them
With "H.264/AAC/MP4" supported, suppose m4a should take zero implementation effort. Unless we do has real mp3 user story, I think we should put mp3 at lowest priority of audio encoder support. Like Ralph said, we may choose to support mp3 after it become royalty-free format.

[Shelly Lin]

Yeah, I'm curious too, why do we need a mp3 format encoder?
Imaging some use cases of recording on mobile:

1. User records a voice memo, either with native recording app or 3rd party
IM app, and share it through SMS or IM.
In this case, user doesn't care what the file format is, as long as it can be
played.

2. User records a video clip, and very possibly like to share it on Youtube.
In this case, we just need to support a Youtube supported format.

Plus, if we just need a short term solution for FirefoxOS, why can't we use
mozCamera to record videos?


-Shelly

[c...@mozilla.com]

On Wednesday, June 19, 2013 11:37:57 AM UTC+8, Shelly Lin wrote:
> Yeah, I'm curious too, why do we need a mp3 format encoder?
> Imaging some use cases of recording on mobile:
>
> 1. User records a voice memo, either with native recording app or 3rd party
> IM app, and share it through SMS or IM.
> In this case, user doesn't care what the file format is, as long as it can be
> played.

I can't disagree with you.
The only use case comes into my mind is:
If I want to create an audio authoring app, what I will do is
a. Create several audio streams as source
b. Connect these source in AudioContext, and append some filter
c. mix and generate a output media stream, and put it into a media recorder
d. Since I am creating an authoring app, suppose I should be able to select an encoding format that I want.

> 2. User records a video clip, and very possibly like to share it on Youtube.
> In this case, we just need to support a Youtube supported format.

More then that. On mobile device, CPU is not as power as x86 system. To support real time recording at 30FPS, BSP/SIMD/GPU solution is required.

>
>
> Plus, if we just need a short term solution for FirefoxOS, why can't we use
> mozCamera to record videos?

That's what we already did, isn't it?
mozCamera is not connectable with all the other Web specs, such as webaudio/ media stream capture/ media source/ media recorder, I think this is the main reason we should not stick with it in long term.
>
>
>
>
> -Shelly
>

[c...@mozilla.com]

On Tuesday, June 18, 2013 10:45:06 AM UTC+8, Shelly Lin wrote:
> Hi everyone,
> I like the idea of supporting VP9, but with limit hardware resources on mobile devices, performance is a critical concern on FirefoxOS. Does that make H.264/AAC be the next supported format?
>

For B2G, yes. For other products on x86, I think VP9 has higher priority.
>
> -Shelly
>

[Thinker K.F. Li]

c...@mozilla.com writes:

> On Wednesday, June 19, 2013 11:37:57 AM UTC+8, Shelly Lin wrote:
>> Yeah, I'm curious too, why do we need a mp3 format encoder? Imaging
>> some use cases of recording on mobile:
>>
>> 1. User records a voice memo, either with native recording app or 3rd
>> party
>> IM app, and share it through SMS or IM. In this case, user
>> doesn't care what the file format is, as long as it can be played.
> I can't disagree with you. The only use case comes into my mind is:
> If I want to create an audio authoring app, what I will do is
> a. Create several audio streams as source b. Connect these source in
> AudioContext, and append some filter c. mix and generate a output
> media stream, and put it into a media recorder d. Since I am creating
> an authoring app, suppose I should be able to select an encoding
> format that I want.

Is this a valid usecase? For authoring tools, users would like to have
all codecs and containers. I don't think we should take this into the
account. All we need to take care is if our solution having been
covering most key platforms.

--
Sinker
--
天教懶漫帶疏狂

[Thinker K.F. Li]

Chris Pearce <cpe...@mozilla.com> writes:

> I saw that C.J filed bug 883749:
>
> "[MediaEncoder] Implement Vorbis/ MP3 encoding format"
> https://bugzilla.mozilla.org/show_bug.cgi?id=883749
>

I had learnt from 883749 that still qwe are working on Vorbis. I don't
get the point why we insist to implement VP8+Vorbis when we are going to
implement VP9? Even VP9 is slow now for B2G; but it would be improved
expected, and already we are working on h.264+aac solution, it don't
hurt B2G with slow VP9 for temporary.

We support VP9/VP8 for providing and promoting a loyalty free solution.
Since VP9 is our long term solution for loyalty free, not VP8, I don't
think we need to work on VP8+Vorbis that would be dropped certainly
later.

--
Sinker
--
天教懶漫帶疏狂

[c...@mozilla.com]

"I can't disagree with you" basic means I agree with what Shelly said.
The use case I listed is not for persuading to construct mp3 encoder. That means "如果真的要我想 大概只有這種case會用道吧(基本上是不太會有需要)"

[c...@mozilla.com]

On Wednesday, August 14, 2013 12:00:58 PM UTC+8, Thinker K.F. Li wrote:
> I had learnt from 883749 that still qwe are working on Vorbis. I don't
> get the point why we insist to implement VP8+Vorbis when we are going to
> implement VP9? Even VP9 is slow now for B2G; but it would be improved
> expected, and already we are working on h.264+aac solution, it don't
> hurt B2G with slow VP9 for temporary.

It's not relative to B2G. On B2G, we will use platform default encoder in StageFright.

>
> We support VP9/VP8 for providing and promoting a loyalty free solution.
> Since VP9 is our long term solution for loyalty free, not VP8, I don't
> think we need to work on VP8+Vorbis that would be dropped certainly
> later.

That's make sense to me.
>
> --
>
> Sinker
>
> --
>
> 天教懶漫帶疏狂