Re: y u no have simple mobile device camera APIs?

[Randell Jesup]

On 4/11/2012 3:23 AM, Jonas Sicking wrote:
> On Tue, Apr 10, 2012 at 11:37 PM, Randell Jesup<rje...@mozilla.com> wrote:
>> I'd like to move this discussion to the dev-media public mailing list,
>> barring objection.

Doing so with this comment; please followup to dev-media

>> On 4/10/2012 10:30 PM, Jonas Sicking wrote:
>>> I'm obviously very late into this thread. I should have payed more
>>> attention.
>>>
>>> I have for a long time been pushing<input type=file> as the way to do
>>> this. It's been hard to get traction behind this idea, but I believe
>>> in part because of it's sorry state of implementation. However
>>> recently things have improved so let me recap where we are:
>>>
>>> In native-Fennec builds the following things work:
>>
>> (uses of<input type=file> snipped)
>>
>>
>>> As for getUserMedia({ picture: true }), we have a patch to display
>>> exactly the same UI as when<input type=file accept="image/*">.click()
>>> is called. However no support for desktop at all.
>>>
>>> That said, getUserMedia({ picture: true }) has advantages:
>>> * It's consistent with other patterns of getUserMedia which will
>>> emerge once we have full WebRTC support.
>>> * We can make it work in workers.
>>>
>>> In the end, maybe we should have both since I know some developers
>>> prefer pure javascript APIs, whereas others like elements even for
>>> things that don't have in-page-rendering such that the DOM is their
>>> one-stop-shop for state/user interaction.
>>>
>>> But I'm also concerned that we get stuck with a mozilla-only API if
>>> other vendors aren't interested in getUserMedia({ picture: true }) and
>>> instead push the<input type=file> route.
>> I'll address this last point, as I've participated in the Media Capture Task
>> Force formed for this purpose. I see no indication that<input> is being
>> pushed by any browser maker and the draft for using it for image capture is
>> abandoned I believe; conversely I see them largely lining up behinds
>> getUserMedia. The Task Force itself was created largely because Microsoft
>> wanted to participate but couldn't due to W3C IPR rules and issues around
>> getting clearance from MS lawyers; in fact a Microsoft person is chairing
>> the Task Force.
>>
>> The biggest issue is the classic one of exposing APIs before the standard is
>> agreed to - they may change. However, I think we'll know the final
>> direction of getUserMedia() for pictures in short order, and also we'll
>> likely prefix getUserMedia for now as Chrome has. (Opera exposed it
>> directly already, and is fighting any API change. It's possible we'll
>> change the name when it's un-prefixed.)
>>
>> The current<input> support will likely remain around due to the normal
>> forces protecting anything that gets sufficient use.
> The draft for "using [<input>] for image capture" was never a very
> well drafted spec. Simply because it was never shown to be necessary.
> I.e. HTML 4 has always had enough of a feature set to implement still
> picture capturing, and HTML 5 a long time ago added the missing pieces
> to get it to feature parity with the current getUserMedia({ picture:
> true }) proposal.
>
> So I wouldn't say that it's demise is meaningful in any way.
>
> I'm ok with exploring getUserMedia, but I strongly believe it would be
> bad for the web to reduce<input> to being a legacy API which we
> should not invest in. It has several nice properties including
> integration with form validation, accessibility, and DOM Events. The
> feedback that I've gotten lately from developers is to have *more*
> APIs done through elements rather than less. For example I've heard
> several requests that the Geolocation API should have been done
> through form integration rather than as a pure JS API, and same
> argument for the up coming joystick API.

An interesting viewpoint; to be honest I'm not up on the issues here,
but a generic note of caution would be that almost any decision will
result in some people saying "couldn't you do it the other way instead?"

> My plan is similarly to add contact support to<input> as a more
> secure version of the current Contacts API.
>
> Also, the claim that browser developers aren't pushing for<input>
> integration but rather getUserMedia doesn't seem to match what I'm
> seeing in Chrome for Android.

We know the WebRTC project in Google has just officially started
supporting Android builds this month, so it appears they are moving
forward with full WebRTC there, at least in Chrome. It's very hard to
say what they'll do in the current default browser; in the longer term
that might become Chrome or they might add WebRTC - and even if they
don't, they may add getUserMedia. Microsoft seems to be pushing this
direction (per my comments), and Opera (if it matters) has already
deployed it un-prefixed. Apple is Apple, and hasn't said anything.

> So I'm ok with exploring getUserMedia. But I still don't understand
> the urgency in pushing the API in ways that doesn't add anything
> beyond the already existing<input> support. Especially when the
> latter works better across browsers and so seems like the better
> choice for authors right now.

If you want to use it in a programmatic way from JS, <input> is annoying
though not necessarily blocking (IIRC).

My expectation is that future uses will move to getUserMedia, especially
once it supports additional controls and constraints. Controlling the
display of live preview, modifying it, allowing the app to control when
to take the picture (modulo security issues), resolution selection,
flash control, etc, etc are all things that will likely come via
getUserMedia/MediaStreams and I don't see coming via <input>, and I
don't see anyone re-opening standardization of these via <input>

You may be right that the simplistic getUserMedia solution won't have
all of these (and in fact would be prefixed). But it would make it
easier to add support for those when they become available. Building
around <input> will likely lock you in to a simple snap-an-image
ability. This may be fine for many uses; if so: great. And I don't
think anyone is saying "don't support <input>".

Randell

[Jonas Sicking]

Absolutely. So we need to be careful who we listen to. It's not
surprising that the Microsoft/Google/Opera people that are
participating in WebRTC is pushing for WebRTC APIs. But that doesn't
mean that other parts of those companies aren't pushing for <input>.
Or that web developers are asking for any particular solution.

>> So I'm ok with exploring getUserMedia. But I still don't understand
>> the urgency in pushing the API in ways that doesn't add anything
>> beyond the already existing<input>  support. Especially when the
>> latter works better across browsers and so seems like the better
>> choice for authors right now.
>
> If you want to use it in a programmatic way from JS, <input> is annoying
> though not necessarily blocking (IIRC).
>
> My expectation is that future uses will move to getUserMedia, especially
> once it supports additional controls and constraints.  Controlling the
> display of live preview, modifying it, allowing the app to control when to
> take the picture (modulo security issues), resolution selection, flash
> control, etc, etc are all things that will likely come via
> getUserMedia/MediaStreams and I don't see coming via <input>, and I don't
> see anyone re-opening standardization of these via <input>
>
> You may be right that the simplistic getUserMedia solution won't have all of
> these (and in fact would be prefixed).  But it would make it easier to add
> support for those when they become available.   Building around <input> will
> likely lock you in to a simple snap-an-image ability.  This may be fine for
> many uses; if so: great.

I totally agree that <input> is basically for the simple snap-an-image
scenario. Though I also think it'd work well for a
record-a-short-video-clip scenario.

Things like streaming audio/video will not work for <input> I don't think.

But likewise I don't think that any page that wants to take a profile
picture should need to include a library to do a complete camera UI,
complete with flash controls, focus etc. That would be annoying for
users since every site would do it differently.

So I think there are strong reasons to have a simple API like the one
requested by the subject line of this thread. Even after we have all
the powerful APIs to do all the advanced stuff. Also keep in mind that
the security model for the two types of APIs will likely be different
which means that they both will have a place in the platform.

> And I don't think anyone is saying "don't support <input>".

Cool.

So the question then is. Is there a reason to add getUserMedia({
picture: true }) *now*, when it's feature set is a strict subset of
what <input> does? (in fact, it's just a different entry point to the
same UI code).

I'm ok with putting an sr stamp on the patch that's sitting in the
bug. But I still think the risk/value ratio isn't terribly great given
the risk it introduces for locking us in for possible future changes
vs. the little value it provides for now.

/ Jonas

[Jonas Sicking]

On Thu, Apr 12, 2012 at 12:59 AM, Ragavan Srinivasan
<rag...@mozilla.com> wrote:
>
>>
>> So the question then is. Is there a reason to add getUserMedia({
>> picture: true }) *now*, when it's feature set is a strict subset of
>> what <input> does? (in fact, it's just a different entry point to the
>> same UI code).
>>
>> I'm ok with putting an sr stamp on the patch that's sitting in the
>> bug. But I still think the risk/value ratio isn't terribly great
>> given
>> the risk it introduces for locking us in for possible future changes
>> vs. the little value it provides for now.
>

> As one of the people really interested in the subject line (which is raising the right set of questions), let me restate the problem statement from a few months ago, at least from an Apps perspective.
>
> I'd like to see apps like Instagram, Paper Camera, Path etc be available in the Mozilla Marketplace, built using web technologies. What API should we ask developers to use? Features like live preview, modifying the image (by applying filters, for example) are important use cases which are not supported by <input>, as I understand it. On the other hand, getUserMedia does appear support the use case that <input> seeks to support.
>
> For that and the other reasons that have surfaced in the thread recently (thanks to Randell), I believe our best bet is with getUserMedia.

getUserMedia will definitely eventually support the features you are mentioning.

But the *current* patch we have for very limited getUserMedia (aka
getUserMedia({picture: true})) supports absolutely no features that
<input> doesn't already support. It is functionality-wise a strict
subset.

So my question is, is there a reason to take *that* patch right now?

/ Jonas

[Jonas Sicking]

Wait. This I disagree with. We should use the right tool for the right
task. I see no reason to push users to getUserMedia when it's not needed.
And I foresee no problem getting authors or other vendors to use/implement
getUseMedia when that's needed. So why spend effort signaling things to the
world?

Anyhow, off to vacation. If we still want I check this in this is fine with
me (modulo review comments in bug)

/ Jonas

On Thursday, April 12, 2012, Ragavan Srinivasan wrote:

> Yes. We are signaling our direction to developers clearly that
> getUserMedia is the future and the fact that the functionality is just a
> subset right now has more to do with deadlines and trains than anything
> else.
>
> Ragavan

[Jonas Sicking]

[Jonas Sicking]

If applications want to switch to using the streaming API, rater than just
get a still picture returned from the platform, then they are going to have
to dramatically rewrite their app no matter what. That's a dramatically
different API than the getUserMedia that we are talking about here.

The only difference between <input> and the getUserMedia functionality we
are talking about here is just a syntactic one. It won't help developers
prepare for the future or anything like that.

The only relation to the streaming API is that the entry point uses the
same getUserMedia function. Everything else is dramatically different. In
functionality, code and security.

/ Jonas

On Thursday, April 12, 2012, Ragavan Srinivasan wrote:

> So what do we tell developers? Rewrite all your apps later?

[Ragavan Srinivasan]

[Ragavan Srinivasan]

>
> So the question then is. Is there a reason to add getUserMedia({
> picture: true }) *now*, when it's feature set is a strict subset of
> what <input> does? (in fact, it's just a different entry point to the
> same UI code).
>
> I'm ok with putting an sr stamp on the patch that's sitting in the
> bug. But I still think the risk/value ratio isn't terribly great
> given
> the risk it introduces for locking us in for possible future changes
> vs. the little value it provides for now.

As one of the people really interested in the subject line (which is raising the right set of questions), let me restate the problem statement from a few months ago, at least from an Apps perspective.

I'd like to see apps like Instagram, Paper Camera, Path etc be available in the Mozilla Marketplace, built using web technologies. What API should we ask developers to use? Features like live preview, modifying the image (by applying filters, for example) are important use cases which are not supported by <input>, as I understand it. On the other hand, getUserMedia does appear support the use case that <input> seeks to support.

For that and the other reasons that have surfaced in the thread recently (thanks to Randell), I believe our best bet is with getUserMedia.

My 2 cents fwiw,
Ragavan

[Ragavan Srinivasan]

Yes. We are signaling our direction to developers clearly that getUserMedia is the future and the fact that the functionality is just a subset right now has more to do with deadlines and trains than anything else.

Ragavan

Jonas Sicking <jo...@sicking.cc> wrote:

On Thu, Apr 12, 2012 at 12:59 AM, Ragavan Srinivasan

[Ragavan Srinivasan]

I should add that it is a very strong message if we can get other browser vendors to also support, which Anant is working on.

Jonas Sicking <jo...@sicking.cc> wrote:

Wait. This I disagree with. We should use the right tool for the right
task. I see no reason to push users to getUserMedia when it's not needed.
And I foresee no problem getting authors or other vendors to use/implement
getUseMedia when that's needed. So why spend effort signaling things to the
world?

Anyhow, off to vacation. If we still want I check this in this is fine with
me (modulo review comments in bug)

/ Jonas

On Thursday, April 12, 2012, Ragavan Srinivasan wrote:

[Anant Narayanan]

On 4/12/2012 1:14 AM, Jonas Sicking wrote:
> getUserMedia will definitely eventually support the features you are mentioning.
>
> But the *current* patch we have for very limited getUserMedia (aka
> getUserMedia({picture: true})) supports absolutely no features that
> <input> doesn't already support. It is functionality-wise a strict
> subset.
>
> So my question is, is there a reason to take *that* patch right now?

The only reason to get the getUserMedia image capture patch in right now
is to get our proverbial foot in the door. The full scale getUserMedia
implementation is likely to take a bit longer so implementing this in
the short term gives us two things:

- Tells the world that we are serious about getUserMedia and that we
think it is the API to use for all image/video capture needs going forward.

- Provide a programmatic *complement* to the declarative <input> API
(<input> isn't going anywhere, as Randell mentioned) and gives us early
developer feedback on the API.

Let's try to stay nimble here, we have nothing to lose by checking in
the simple getUserMedia API (especially if prefixed, and always subject
to change), we can only gain by having developers try it out and let us
know what they think.

I will note that the <input> style API hasn't seen much adoption amongst
developers, most of the time all you hear is that "the web doesn't
support camera access yet", even though that's not true, and all eyes
are on getUserMedia.

-Anant

[Brendan Eich]

Anant Narayanan wrote:
> - Provide a programmatic *complement* to the declarative <input> API
> (<input> isn't going anywhere, as Randell mentioned) and gives us
> early developer feedback on the API.

Sicking may be on vacation but I'll add a +1 here: we don't want no JS
API, only HTML markup, if there's a solid for
programmed-in-JS-not-markup use-case. I believe there is. Anyone think
differently?

/be

[Randell Jesup]

I had a longer response written talking about how it let developers
design code that makes it easier to later add getUserMedia-enabled
features, but I'll just say: I agree with Brendan and Anant here. Get
it in; do not push people to abandon <input> for a partial (prefixed)
API in production code, but start getting some exposure and experience.

Randell