HTML Media support and format proliferation on Firefox OS, Android and Desktop

[Chris Double]

Mozilla supports various codecs in HTML media across Firefox OS,
Android and Desktop. Currently we support the following (feel free to
add if I've missed any):

H.264 in an MP4 container on Firefox OS, some Android devices and
planned for desktop
AAC in an MP4 container on Firefox OS, some Android devices and
planned for desktop
MP3 on Firefox OS, some Android devices and planned for desktop
WAV on all platforms
Opus in an Ogg container on all platforms
Vorbis in an Ogg and WebM container on all platforms
Theora in an Ogg container on all platforms
VP8 in a WebM container on all platforms

These are the formats that Mozilla has generally decided to support
"on the web". That is, a web developer can expect to provide a video
element with one of these formats as the source and expect it to
play,if not now then at some point in the future.

Recently there has been requests to support other formats,
particularly on Firefox OS. In the mobile world some devices have
cameras that record video in 3GPP format, and audio in other formats.
MMS messages can also contain video in 3GPP format. For Firefox OS we
would want to be able to play these formats if they are produced or
received by the device. While we want the support in Firefox OS we
don't necessarily want these formats to be supported on the wider web.
We want to reduce format proliferation. This makes interoperability
easier and reduces the decisions needed by web developers when they
want a video to play cross platform and cross browser successfully.

To enable Firefox OS to support formats that we don't want to support
in the wider web I think we should restrict these formats such that
they play only via local files or local apps on the device. The
Firefox OS device may be able to play a 3GPP file in the devices video
app for example, but not in a video element on a normal web page. RTSP
support should spawn the a local video player to play it, and not play
RTSP via audio elements on the wider web.

When there is a desire to support a new format we should first
consider if our existing formats cover the required use cases. If
there is a need, due to device requirements or the like, we should
limit the playback such that we don't increase format proliferation
across all operating systems, devices and browsers.

If you have any comments on this approach, feel free to reply here and
start a discussion. In particular I'm interested in if this approach
works in a world with possible partner requirements or other
commercial conflicts with Firefox OS.

--
http://www.bluishcoder.co.nz

[cjco...@gmail.com]

Hi Chris,
Thanks for sharing.
I have a small question: Does this strategy, I mean play unsupported source on external players, apply to B2G as well? On B2G, AFAIK, there is no external player, unless we construct media player in native side first.

[Chris Double]

On Fri, Feb 1, 2013 at 6:30 PM, <cjco...@gmail.com> wrote:
> I have a small question: Does this strategy, I mean play unsupported source on external players, apply to B2G as well? On B2G, AFAIK, there is no external player, unless we construct media player in native side first.

For B2G/Firefox OS we'd only play the formats listed. If a format not
in that list needs to be supported for some reason we'd need to
restrict it on on-device only, not general web content.

--
http://www.bluishcoder.co.nz

[Thinker K.F. Li]

cjco...@gmail.com writes:

> Hi Chris,
> Thanks for sharing.
> I have a small question: Does this strategy, I mean play unsupported
> source on external players, apply to B2G as well? On B2G, AFAIK, there
> is no external player, unless we construct media player in native side
> first.

We need to define a way for interaction and resource management between
external program and B2G if we want it. Once define it, we open a door
let 3rd party vendors write their own native programs. It is definitly
not what we want to encourage.

Another solution is plugin (NSAPI), but it is also another thing that we
don't encourage. Maybe some one can figure out a better solution, but I
don't see at this monent.

For RTSP, if we still want it at last, video tag is still a technically
good solution for now. Other projects, beside B2G, can disable it if
they don't want. If we implement RTSP in a way other than video/audio
tags, it means to create a non-standard way to use it. It may encourage
people to use a non-standard API. I don't see what a non-standard way
is better than video/audio tag.

--
Sinker
--
天教懶漫帶疏狂

[Chris Double]

On Fri, Feb 1, 2013 at 10:29 PM, Thinker K.F. Li <thi...@codemud.net> wrote:
>
> We need to define a way for interaction and resource management between
> external program and B2G if we want it. Once define it, we open a door
> let 3rd party vendors write their own native programs. It is definitly
> not what we want to encourage.

Right, I think there's a misunderstanding. When I said "local apps" I
meant a web app that exists on the device vs a web page on the
internet. I'm not saying using a native app to play these formats.
I'm saying only web applications installed on the device can use them.

--
http://www.bluishcoder.co.nz

[Thinker K.F. Li]

How can we implement RTSP in a web application? Do you have any idea?

--
Sinker
--
嚙諸梧蕭嚙箠嚙踝蕭嚙窮嚙踝蕭嚙篇

[Fabrice Desre]

On 02/01/2013 02:19 AM, Thinker K.F. Li wrote:

>> Right, I think there's a misunderstanding. When I said "local apps" I
>> meant a web app that exists on the device vs a web page on the
>> internet. I'm not saying using a native app to play these formats.
>> I'm saying only web applications installed on the device can use them.
>

> How can we implement RTSP in a web application? Do you have any idea?
>

You don't implement RTSP in the web application itself (or maybe you can
with tcp socket and canvas, I don't know the protocol well enough). You
implement it on the platform side, and we check at run time if the
requesting document is part of a privileged or certified app to allow
the resource to be fetched and played.

Fabrice
--
Fabrice Desré
b2g team
Mozilla Corporation

[Thinker K.F. Li]

I don't think we can make a useful player in the web application itself
if it is possible. We can use websocket and canvas in a web applicaion,
but it also means to implement codecs in JS. It may be workable for
desktop for people having good high-end PC, but it is not for mobile
devices.

For MediaElement approach, it means checking if the requesting app
allowed to play RTSP stream at MediaElement. If we don't go
MediaElement approach, we will do it in a approach that we don't like
others to follow. I afraid that eventually hardware vendors will find
it is a easy way for them to integrate exisiting media player on their
platform. Then, I can foresee a lot of vendors integrate their
platforms in this way.

--
Sinker
--
天教懶漫帶疏狂

[shia...@gmail.com]

Chris Double於 2013年2月1日星期五UTC+8下午5時53分18秒寫道：

> Right, I think there's a misunderstanding. When I said "local apps" I
> meant a web app that exists on the device vs a web page on the
> internet. I'm not saying using a native app to play these formats.
> I'm saying only web applications installed on the device can use them.
>

Hi Chris,

What is the concern for not supporting RTSP in the web page, and play it with external web apps instead? Is it for technical or legal consideration?

Thanks,
Shianyow

[Sotaro Ikeda]

Hi,

I agree that if FirefoxOS do not provide a way to implement RTSP, Venders
are going to implement it by own ways, because it is necessary for
products. And MediaElement approach is suit for RTSP, I think.

When FirefoxOS succeeds as a mobile platform, a lot of venders are going to
implement a lot of existing multimedia capabilities on FirefoxOS. For
example, mobile TV capability needs to be implemented in FirefoxOS. In
Japan, it is one of the very important capabilities. Right now, there are
no guidences about how to implement it. It needs a lot of APIs and
capabilites. Then MediaElement approach do not fit to it, I think.

But if we do not provide a way to implement it. Each venders are going to
implement it by their own ways. I know that it is not so difficult to
implement them, if there are already C/C++ implementation in android. We
should provide these APIs like WebFM if they are important capabilities for
mobile phones. And need to provide media server process capabilities in
gecko in future.

Regards,
Sotaro


2013/2/1 Thinker K.F. Li <thi...@codemud.net>

> I don't think we can make a useful player in the web application itself
> if it is possible. We can use websocket and canvas in a web applicaion,
> but it also means to implement codecs in JS. It may be workable for
> desktop for people having good high-end PC, but it is not for mobile
> devices.
>
> For MediaElement approach, it means checking if the requesting app
> allowed to play RTSP stream at MediaElement. If we don't go
> MediaElement approach, we will do it in a approach that we don't like
> others to follow. I afraid that eventually hardware vendors will find
> it is a easy way for them to integrate exisiting media player on their
> platform. Then, I can foresee a lot of vendors integrate their
> platforms in this way.
>
> --
> Sinker
> --
> 天教懶漫帶疏狂

> _______________________________________________
> dev-media mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-media
>

[Sotaro Ikeda]

Hi,

I agree that if FirefoxOS do not provide a way to implement RTSP, Venders are going to implement it by own ways, because it is necessary for products. And MediaElement approach is suit for RTSP, I think.

When FirefoxOS succeeds as a mobile platform, a lot of venders are going to implement a lot of existing multimedia capabilities on FirefoxOS. For example, mobile TV capability needs to be implemented in FirefoxOS. In Japan, it is one of the very important capabilities. Right now, there are no guidences about how to implement it. It needs a lot of APIs and capabilites. Then MediaElement approach do not fit to it, I think.

But if we do not provide a way to implement it. Each venders are going to implement it by their own ways. I know that it is not so difficult to implement them, if there are already C/C++ implementation in android. We should provide these APIs and implementations like WebFM if they are important capabilities for mobile phones. And need to provide media server process capabilities in gecko in future.

Regards,
Sotaro

[Robert O'Callahan]

On Sat, Feb 2, 2013 at 5:14 PM, Thinker K.F. Li <thi...@codemud.net> wrote:

> For MediaElement approach, it means checking if the requesting app
> allowed to play RTSP stream at MediaElement. If we don't go
> MediaElement approach, we will do it in a approach that we don't like
> others to follow. I afraid that eventually hardware vendors will find
> it is a easy way for them to integrate exisiting media player on their
> platform. Then, I can foresee a lot of vendors integrate their
> platforms in this way.
>

That's a good point. I don't think we disagree here. If we have to have
RTSP, then let's build support for it into HTML media elements, but make
that support available only to B2G apps. (And we may want to limit which
apps can use it, too.)

Rob
--
Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur
Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl
bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat
lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir
— whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb
tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28]

[shia...@gmail.com]

Chris Double於 2013年2月1日星期五UTC+8下午5時53分18秒寫道：

> Right, I think there's a misunderstanding. When I said "local apps" I
> meant a web app that exists on the device vs a web page on the
> internet. I'm not saying using a native app to play these formats.
> I'm saying only web applications installed on the device can use them.
>

[reed...@gmail.com]

Consider this scenario:
1) user takes 3GPP video on FFOS device
2) user plays video on FFOS device (plays fine)
3) user uploads video to file storage site
4) user deletes video to free space on device
5) user accesses video from file storage site; video does not play

[Robert O'Callahan]

On Wed, Mar 20, 2013 at 4:25 AM, <reed...@gmail.com> wrote:

> Consider this scenario:
> 1) user takes 3GPP video on FFOS device
> 2) user plays video on FFOS device (plays fine)
>

Plays fine in the FFOS video player, yes.

3) user uploads video to file storage site
> 4) user deletes video to free space on device
> 5) user accesses video from file storage site; video does not play
>

It wouldn't ever have been playable from the file storage site in the
browser. Different apps supporting different formats isn't good, but it's
already a common issue.

If we were to support 3GPP on FFOS but not desktop Firefox, then there
would be a problem when the user tries to access the video from the file
storage site in desktop Firefox. We have to draw the line somewhere.

This particular scenario is easy to resolve, anyway. We would just need to
handle 3GPP from the Web by downloading it back onto the device and
launching the video player app, like we do for external helper apps on
desktop.

[reed...@gmail.com]

Having to re-download a video just to watch it would be a rather clunky user experience. I'm not sure what fraction of users will tolerate that these days.

Instead of the browser flat refusing to play media in a unblessed format, can the user be offered the opportunity to transcode the video?

[Robert O'Callahan]

During upload you mean? I think so, but transcoding video on the phone
doesn't sound good.

The site itself could usually transcode the video --- Youtube would already
do this, if the "file storage site" was actually Youtube. Of course that's
beyond our control.

Can we make users record in MP4 format instead of 3GPP in the first place?

[Ralph Giles]

On 13-03-20 2:17 PM, Robert O'Callahan wrote:

> Can we make users record in MP4 format instead of 3GPP in the first place?

Or WebM.

-r

[Robert O'Callahan]

On Thu, Mar 21, 2013 at 10:17 AM, Robert O'Callahan <rob...@ocallahan.org>wrote:

> During upload you mean? I think so, but transcoding video on the phone
> doesn't sound good.
>

Actually, assuming the 3GPP already contains H.264 video then transcoding
just the audio (if not AAC already) and reformatting the container is
probably fine from a performance point of view. So yes we could do that.

[Chris Double]

3GP is used to be compatible with sending videos via MMS. IIRC it's
3GP containing H.263/AMR.

--
http://www.bluishcoder.co.nz

[reed...@gmail.com]

Some careful thinking would need to be done about where one could suggest transcoding to the user - as Robert noted, video sites already do transcoding.

One possibility is to have extra, virtual entries in the file selection dialog for media files in an unblessed format. Selecting a virtual entry would read from a transcoding filter rather than the raw file.

My scenario then becomes
...
3a) User taps file input in upload form
3b) User selects virtual entry
3c) User taps "Upload" button in upload form
4) User deletes video to save space on device
5) User accesses transcoded video from file storage site; video plays

Another possibility is, when the user taps on an unblessed media in a file selection dialog, a subordinate dialog pops up to offer a choice between the unblessed format and a blessed format (it could be called a "modern" format to cue users that it's better). The dialog could say something like "Convert to a more compatible format? This will take 2 minutes and use some battery charge."

[Robert O'Callahan]

Where's this video going to come from? If it's something the user recorded,
then it should be H.264 already and transcoding should be (could be) cheap.
Are we talking about receiving a 3GPP with MP4 level 2 video by MMS, and
uploading that? Is it possible/likely to send large videos that way?

Instead of modifying the UI perhaps the best approach would be to transcode
by default and then whitelist a set of sites that we know do their own
transcoding.

[mmma...@gmail.com]

Sorry to be a pain, but I notice FLAC isn't mentioned up there. This is a bit of a showstopper for me, in terms of my entire music collection has been painstakingly (over months) been ripped to FLAC, and I want to play that music in Firefox OS on my ZTE Open. Is FLAC just forgotten from the list or unsupported?

[rahuld...@gmail.com]

Hi,

Where can I get the details of audio supported on Firefox OS? Along with the formats, I would like to know the supported Bitrate and Sampling Rate.

Thanks,
Rahul

[hugo....@gmail.com]

Hi there,
Just a little thing : FLAC is readable in firefox OS but you have to turn the .flac extension into .mp3 or any nativly firefoxos readable format. The metadata are not availble though... I guess flac support wouldn't be very hard to create then as it is only a database problem but I am not a dev...
+++
Hugo