Draft Camera Control API

[Fabrice Desré]

The Gaia Camera Application (https://wiki.mozilla.org/Gaia/Camera) needs
access to camera feature that are beyond just displaying a preview and
taking a picture.

Namely, Gaia needs at least to control the flash and have the
possibility to take full resolution still capture, as well as
starting/stopping video recording. Video recording is different than
previewing since it can be a higher resolution.

Qualcomm has a API for that
(https://developer.qualcomm.com/docs/html5/api/camera.html) that maps
closely to the android HAL layer. It is fully isolated from any other
spec, eg getUserMedia() so we are unlikely to implement it as it is.

Some design questions for this API :
- Where do we hook up this API : my choice would be for MediaStreams
that come from a camera to also implement the camera control interface.

- Use events or callbacks (or DOMRequests) for asynchronous calls.
getUserMedia() will probably use callbacks so it may be more consistent
to also use callbacks here.

- Expose each control as a property or provide a generic
setFeature()/getFeature() access?

Here's a tentative interface description :

interface CameraControl {
// takes a full resolution capture. Passes a DOMFile to successCB.
void takePicture(successCB, [errorCB]);

// asks the hardware to autofocus. successCB is called when the
// autofocus has completed.
void autofocus(successCB, [errorCB]);

// Starts recording. Passes a DOMFile to successCB.
startRecording(successCB);

// stops an ongoing recording. Will cause the successCB
// from startRecording to be called.
stopRecording();

attribute DOMString flash; // "on", "off" or "auto"
attribute DOMString effect; // "none", "mono", "sepia", ...
attribute long brightness;
}

Comments, etc. welcome!

Fabrice
--
Fabrice Desré
b2g Team
Mozilla Corporation

[Anant Narayanan]

Hey Fabrice,

On 05/14/2012 01:40 PM, Fabrice Desré wrote:
> Some design questions for this API :
> - Where do we hook up this API : my choice would be for MediaStreams
> that come from a camera to also implement the camera control interface.

I think that's a great place to start with!

> - Use events or callbacks (or DOMRequests) for asynchronous calls.
> getUserMedia() will probably use callbacks so it may be more consistent
> to also use callbacks here.

I'd prefer callbacks, not only because it's consistent with
getUserMedia, but also that DOMRequests seem more suited to objects that
are actually in the DOM tree (and need to take advantage of multiple
listeners, event bubbling, etc). These are not applicable to Media Streams.

> - Expose each control as a property or provide a generic
> setFeature()/getFeature() access?

Individual properties are better than generic methods, IMHO. Otherwise
we run the risk of overloading too many things into a single function,
like ioctl.

> interface CameraControl {
> // takes a full resolution capture. Passes a DOMFile to successCB.
> void takePicture(successCB, [errorCB]);
>
> // asks the hardware to autofocus. successCB is called when the
> // autofocus has completed.
> void autofocus(successCB, [errorCB]);
>

> attribute DOMString flash; // "on", "off" or "auto"
> attribute DOMString effect; // "none", "mono", "sepia", ...
> attribute long brightness;
> }

All of the above look great! However:

> // Starts recording. Passes a DOMFile to successCB.
> startRecording(successCB);
>
> // stops an ongoing recording. Will cause the successCB
> // from startRecording to be called.
> stopRecording();

We might want to consider moving recording elsewhere. There are few
proposals floating around (Media Source API, Media Recorder in
getUserMedia, and record methods on Media Streams themselves). We can
pass in resolution information via the constraints API for getUserMedia
so that recording can be done at a high resolution, independent of preview.

Cheers,
-Anant

[Jonas Sicking]

On Mon, May 14, 2012 at 2:03 PM, Anant Narayanan <an...@mozilla.com> wrote:
>> - Use events or callbacks (or DOMRequests) for asynchronous calls.
>> getUserMedia() will probably use callbacks so it may be more consistent
>> to also use callbacks here.
>
> I'd prefer callbacks, not only because it's consistent with getUserMedia,
> but also that DOMRequests seem more suited to objects that are actually in
> the DOM tree (and need to take advantage of multiple listeners, event
> bubbling, etc). These are not applicable to Media Streams.

I'm not sure where you got the impression that DOMRequest is best
suited for the DOM tree. We currently don't use them anywhere where we
have DOM trees. Likewise, IDBRequest (which basically is a superclass
of DOMRequest) is used all over IndexedDB which isn't connected to DOM
trees. Similarly XMLHttpRequest and FileReader both use events without
using DOM trees.

Ideally I would like to use promises, but unfortunately there's no
standard for those yet.

The problem that I have with callbacks is they aren't very easy to
expand. For example the geolocation API uses a callback to let the
page know about the user's current location. However if you look at
many mapping applications they show an approximate user location which
is gradually refined until a precise location is known. In order to
add this ability to the geolocation API we'd need to add a third
function argument (of which two would be optional) which makes the
signature pretty messy. Likewise, it's hard to add the ability to
cancel a asynchronous request when using the callback style.

We had the same problem when designing the FileReader API. I initially
proposed using callbacks, but a big reason we switched to using events
was because providing progress notifications and the ability to abort
a started load was hard otherwise.

>> - Expose each control as a property or provide a generic
>> setFeature()/getFeature() access?
>
> Individual properties are better than generic methods, IMHO. Otherwise we
> run the risk of overloading too many things into a single function, like
> ioctl.

I agree.

/ Jonas

[Anant Narayanan]

On 05/14/2012 03:55 PM, Jonas Sicking wrote:
> I'm not sure where you got the impression that DOMRequest is best
> suited for the DOM tree. We currently don't use them anywhere where we
> have DOM trees. Likewise, IDBRequest (which basically is a superclass
> of DOMRequest) is used all over IndexedDB which isn't connected to DOM
> trees. Similarly XMLHttpRequest and FileReader both use events without
> using DOM trees.

The behaviour is quite similar to events on regular DOM objects
(document.onload, form.onreset, etc.), but you're right that it is used
in plenty of other places where there is no DOM tree. I had always
perceived the DOMRequest pattern to be useful only when the caller needs
multiple listeners and event bubbling, preventDefault, etc. But it
sounds like you have other reasons to prefer the DOMRequest model...

> The problem that I have with callbacks is they aren't very easy to
> expand. For example the geolocation API uses a callback to let the
> page know about the user's current location. However if you look at
> many mapping applications they show an approximate user location which
> is gradually refined until a precise location is known. In order to
> add this ability to the geolocation API we'd need to add a third
> function argument (of which two would be optional) which makes the
> signature pretty messy. Likewise, it's hard to add the ability to
> cancel a asynchronous request when using the callback style.

That's a pretty compelling reason, thanks for pointing it out.

We could achieve the same with a hybrid approach, which is what the
current getUserMedia spec is doing. Callbacks are only used for error
and success conditions, and everything else is an event. For instance,
the MediaStream object has events for trackadded, trackremoved, etc. and
we can add more in the future.

In general, that's equivalent to the pattern:

apiCall(options, onsuccess, onerror);
onsuccess(obj) { obj.onevent = function(){} ... }

What do you think of this approach?

Thanks,
-Anant

[Jonas Sicking]

On Mon, May 14, 2012 at 1:40 PM, Fabrice Desré <fab...@mozilla.com> wrote:
> interface CameraControl {
>  // takes a full resolution capture. Passes a DOMFile to successCB.
>  void takePicture(successCB, [errorCB]);

Wouldn't we use the "normal" getUserMedia({picture:true}) API for this?

>  // asks the hardware to autofocus. successCB is called when the
>  // autofocus has completed.
>  void autofocus(successCB, [errorCB]);

Don't we need to be able to say which x/y coordinate to autofocus on?

Many devices today use touch-to-focus which lets the user choose where
on the screen to focus.

Also, do we need to allow setting the "exposure compensation" (the
thing that usually goes from -3EV to +3EV)? The iPhone does this in a
very nifty way where when you click a point on the screen, the camera
both focuses there, and adjusts the exposure compensation such that
that area is neither over exposed or under exposed.

/ Jonas