Draft Camera Control API

[Fabrice Desré]

The Gaia Camera Application (https://wiki.mozilla.org/Gaia/Camera) needs
access to camera feature that are beyond just displaying a preview and
taking a picture.

Namely, Gaia needs at least to control the flash and have the
possibility to take full resolution still capture, as well as
starting/stopping video recording. Video recording is different than
previewing since it can be a higher resolution.

Qualcomm has a API for that
(https://developer.qualcomm.com/docs/html5/api/camera.html) that maps
closely to the android HAL layer. It is fully isolated from any other
spec, eg getUserMedia() so we are unlikely to implement it as it is.

Some design questions for this API :
- Where do we hook up this API : my choice would be for MediaStreams
that come from a camera to also implement the camera control interface.

- Use events or callbacks (or DOMRequests) for asynchronous calls.
getUserMedia() will probably use callbacks so it may be more consistent
to also use callbacks here.

- Expose each control as a property or provide a generic
setFeature()/getFeature() access?

Here's a tentative interface description :

interface CameraControl {
// takes a full resolution capture. Passes a DOMFile to successCB.
void takePicture(successCB, [errorCB]);

// asks the hardware to autofocus. successCB is called when the
// autofocus has completed.
void autofocus(successCB, [errorCB]);

// Starts recording. Passes a DOMFile to successCB.
startRecording(successCB);

// stops an ongoing recording. Will cause the successCB
// from startRecording to be called.
stopRecording();

attribute DOMString flash; // "on", "off" or "auto"
attribute DOMString effect; // "none", "mono", "sepia", ...
attribute long brightness;
}

Comments, etc. welcome!

Fabrice
--
Fabrice Desré
b2g Team
Mozilla Corporation

[Anant Narayanan]

Hey Fabrice,

On 05/14/2012 01:40 PM, Fabrice Desré wrote:
> Some design questions for this API :
> - Where do we hook up this API : my choice would be for MediaStreams
> that come from a camera to also implement the camera control interface.

I think that's a great place to start with!

> - Use events or callbacks (or DOMRequests) for asynchronous calls.
> getUserMedia() will probably use callbacks so it may be more consistent
> to also use callbacks here.

I'd prefer callbacks, not only because it's consistent with
getUserMedia, but also that DOMRequests seem more suited to objects that
are actually in the DOM tree (and need to take advantage of multiple
listeners, event bubbling, etc). These are not applicable to Media Streams.

> - Expose each control as a property or provide a generic
> setFeature()/getFeature() access?

Individual properties are better than generic methods, IMHO. Otherwise
we run the risk of overloading too many things into a single function,
like ioctl.

> interface CameraControl {
> // takes a full resolution capture. Passes a DOMFile to successCB.
> void takePicture(successCB, [errorCB]);
>
> // asks the hardware to autofocus. successCB is called when the
> // autofocus has completed.
> void autofocus(successCB, [errorCB]);
>

> attribute DOMString flash; // "on", "off" or "auto"
> attribute DOMString effect; // "none", "mono", "sepia", ...
> attribute long brightness;
> }

All of the above look great! However:

> // Starts recording. Passes a DOMFile to successCB.
> startRecording(successCB);
>
> // stops an ongoing recording. Will cause the successCB
> // from startRecording to be called.
> stopRecording();

We might want to consider moving recording elsewhere. There are few
proposals floating around (Media Source API, Media Recorder in
getUserMedia, and record methods on Media Streams themselves). We can
pass in resolution information via the constraints API for getUserMedia
so that recording can be done at a high resolution, independent of preview.

Cheers,
-Anant

[Jonas Sicking]

On Mon, May 14, 2012 at 2:03 PM, Anant Narayanan <an...@mozilla.com> wrote:
>> - Use events or callbacks (or DOMRequests) for asynchronous calls.
>> getUserMedia() will probably use callbacks so it may be more consistent
>> to also use callbacks here.
>
> I'd prefer callbacks, not only because it's consistent with getUserMedia,
> but also that DOMRequests seem more suited to objects that are actually in
> the DOM tree (and need to take advantage of multiple listeners, event
> bubbling, etc). These are not applicable to Media Streams.

I'm not sure where you got the impression that DOMRequest is best
suited for the DOM tree. We currently don't use them anywhere where we
have DOM trees. Likewise, IDBRequest (which basically is a superclass
of DOMRequest) is used all over IndexedDB which isn't connected to DOM
trees. Similarly XMLHttpRequest and FileReader both use events without
using DOM trees.

Ideally I would like to use promises, but unfortunately there's no
standard for those yet.

The problem that I have with callbacks is they aren't very easy to
expand. For example the geolocation API uses a callback to let the
page know about the user's current location. However if you look at
many mapping applications they show an approximate user location which
is gradually refined until a precise location is known. In order to
add this ability to the geolocation API we'd need to add a third
function argument (of which two would be optional) which makes the
signature pretty messy. Likewise, it's hard to add the ability to
cancel a asynchronous request when using the callback style.

We had the same problem when designing the FileReader API. I initially
proposed using callbacks, but a big reason we switched to using events
was because providing progress notifications and the ability to abort
a started load was hard otherwise.

>> - Expose each control as a property or provide a generic
>> setFeature()/getFeature() access?
>
> Individual properties are better than generic methods, IMHO. Otherwise we
> run the risk of overloading too many things into a single function, like
> ioctl.

I agree.

/ Jonas

[Anant Narayanan]

On 05/14/2012 03:55 PM, Jonas Sicking wrote:
> I'm not sure where you got the impression that DOMRequest is best
> suited for the DOM tree. We currently don't use them anywhere where we
> have DOM trees. Likewise, IDBRequest (which basically is a superclass
> of DOMRequest) is used all over IndexedDB which isn't connected to DOM
> trees. Similarly XMLHttpRequest and FileReader both use events without
> using DOM trees.

The behaviour is quite similar to events on regular DOM objects
(document.onload, form.onreset, etc.), but you're right that it is used
in plenty of other places where there is no DOM tree. I had always
perceived the DOMRequest pattern to be useful only when the caller needs
multiple listeners and event bubbling, preventDefault, etc. But it
sounds like you have other reasons to prefer the DOMRequest model...

> The problem that I have with callbacks is they aren't very easy to
> expand. For example the geolocation API uses a callback to let the
> page know about the user's current location. However if you look at
> many mapping applications they show an approximate user location which
> is gradually refined until a precise location is known. In order to
> add this ability to the geolocation API we'd need to add a third
> function argument (of which two would be optional) which makes the
> signature pretty messy. Likewise, it's hard to add the ability to
> cancel a asynchronous request when using the callback style.

That's a pretty compelling reason, thanks for pointing it out.

We could achieve the same with a hybrid approach, which is what the
current getUserMedia spec is doing. Callbacks are only used for error
and success conditions, and everything else is an event. For instance,
the MediaStream object has events for trackadded, trackremoved, etc. and
we can add more in the future.

In general, that's equivalent to the pattern:

apiCall(options, onsuccess, onerror);
onsuccess(obj) { obj.onevent = function(){} ... }

What do you think of this approach?

Thanks,
-Anant

[Jonas Sicking]

On Mon, May 14, 2012 at 1:40 PM, Fabrice Desré <fab...@mozilla.com> wrote:
> interface CameraControl {
>  // takes a full resolution capture. Passes a DOMFile to successCB.
>  void takePicture(successCB, [errorCB]);

Wouldn't we use the "normal" getUserMedia({picture:true}) API for this?

>  // asks the hardware to autofocus. successCB is called when the
>  // autofocus has completed.
>  void autofocus(successCB, [errorCB]);

Don't we need to be able to say which x/y coordinate to autofocus on?

Many devices today use touch-to-focus which lets the user choose where
on the screen to focus.

Also, do we need to allow setting the "exposure compensation" (the
thing that usually goes from -3EV to +3EV)? The iPhone does this in a
very nifty way where when you click a point on the screen, the camera
both focuses there, and adjusts the exposure compensation such that
that area is neither over exposed or under exposed.

/ Jonas

[Fabrice Desré]

On Mon 14 May 2012 04:16:56 PM PDT, Jonas Sicking wrote:
> On Mon, May 14, 2012 at 1:40 PM, Fabrice Desré <fab...@mozilla.com> wrote:
>> interface CameraControl {
>> // takes a full resolution capture. Passes a DOMFile to successCB.
>> void takePicture(successCB, [errorCB]);
>
> Wouldn't we use the "normal" getUserMedia({picture:true}) API for this?

At least on b2g, |getUserMedia({picture:true})| will be implemented by
the camera app that uses |getUserMedia({video:true})| and the Camera
Control API.

>> // asks the hardware to autofocus. successCB is called when the
>> // autofocus has completed.
>> void autofocus(successCB, [errorCB]);
>
> Don't we need to be able to say which x/y coordinate to autofocus on?
>
> Many devices today use touch-to-focus which lets the user choose where
> on the screen to focus.

Yes, we can do that.

> Also, do we need to allow setting the "exposure compensation" (the
> thing that usually goes from -3EV to +3EV)? The iPhone does this in a
> very nifty way where when you click a point on the screen, the camera
> both focuses there, and adjusts the exposure compensation such that
> that area is neither over exposed or under exposed.

The android HAL also exposes that (it's called "Metering") and we can
specify metering areas and levels. Would it be better to let the app set
itself the autofocus area and the metering area be the same, or just
provide autofocus and magically ensure the iphone-like behavior?

[Jonas Sicking]

On Mon, May 14, 2012 at 4:44 PM, Fabrice Desré <fab...@mozilla.com> wrote:
> On Mon 14 May 2012 04:16:56 PM PDT, Jonas Sicking wrote:
>>
>> On Mon, May 14, 2012 at 1:40 PM, Fabrice Desré <fab...@mozilla.com>
>> wrote:
>>>
>>> interface CameraControl {
>>>  // takes a full resolution capture. Passes a DOMFile to successCB.
>>>  void takePicture(successCB, [errorCB]);
>>
>>
>> Wouldn't we use the "normal" getUserMedia({picture:true}) API for this?
>
>
> At least on b2g, |getUserMedia({picture:true})| will be implemented by the
> camera app that uses |getUserMedia({video:true})| and the Camera Control
> API.

Ah! Makes sense.

>> Also, do we need to allow setting the "exposure compensation" (the
>> thing that usually goes from -3EV to +3EV)? The iPhone does this in a
>> very nifty way where when you click a point on the screen, the camera
>> both focuses there, and adjusts the exposure compensation such that
>> that area is neither over exposed or under exposed.
>
> The android HAL also exposes that (it's called "Metering") and we can
> specify metering areas and levels. Would it be better to let the app set
> itself the autofocus area and the metering area be the same, or just provide
> autofocus and magically ensure the iphone-like behavior?

I think we should be explicit about what we do. We could either have
an API which lets you choose a "attention point" which makes us
automatically focus and set exposure compensation for that area.

Alternatively we expose separate APIs for setting the focus area and
the exposure compensation area.

When setting the exposure compensation area, is it possible to get the
actual EV value back from the hardware? It would be cool if an app
could display a slider for choosing exposure compensation, while also
allowing the user to click anywhere in the picture to automatically
adjust the slider to the value appropriate for that area.

Actually, that would be cool to be able to to for focus too. I.e. it
would be cool to be able to get the focus distance and be able to
adjust it with a slider.

/ Jonas

[Fabrice Desre]

On 05/15/2012 12:36 AM, Jonas Sicking wrote:
> When setting the exposure compensation area, is it possible to get the
> actual EV value back from the hardware? It would be cool if an app
> could display a slider for choosing exposure compensation, while also
> allowing the user to click anywhere in the picture to automatically
> adjust the slider to the value appropriate for that area.

Yes, we can read and write the EV value.

> Actually, that would be cool to be able to to for focus too. I.e. it
> would be cool to be able to get the focus distance and be able to
> adjust it with a slider.

We can also get the focus distance (and depth of field), but I don't
think we can set it manually. There are preset focus modes (Auto,
Infinity, Macro, Fixed) and continuous autofocus modes (useful when
recording video).

The hardware exposes a lot of knobs we can use (see
http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h).
To me the question is mostly "do we want to expose everything to web apps?"

Fabrice
--
Fabrice Desré
b2g team
Mozilla Corporation

[Jonas Sicking]

On Tue, May 15, 2012 at 9:09 AM, Fabrice Desre <fab...@mozilla.com> wrote:
> On 05/15/2012 12:36 AM, Jonas Sicking wrote:
>>
>> When setting the exposure compensation area, is it possible to get the
>> actual EV value back from the hardware? It would be cool if an app
>> could display a slider for choosing exposure compensation, while also
>> allowing the user to click anywhere in the picture to automatically
>> adjust the slider to the value appropriate for that area.
>
> Yes, we can read and write the EV value.

Very cool!

>> Actually, that would be cool to be able to to for focus too. I.e. it
>> would be cool to be able to get the focus distance and be able to
>> adjust it with a slider.
>
> We can also get the focus distance (and depth of field), but I don't think
> we can set it manually. There are preset focus modes (Auto, Infinity, Macro,
> Fixed) and continuous autofocus modes (useful when recording video).

What does "Fixed" mean?

> The hardware exposes a lot of knobs we can use (see
> http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h).
> To me the question is mostly "do we want to expose everything to web apps?"

Wow, there's lots of candy there. I think I'd prefer to expose a
pretty low-level API which will let authors build a wide variety of
camera apps. There's lots of cool stuff that you can do in software to
improve picture taking.

/ Jonas

[Lucas Adamski]

On 5/15/2012 12:36 AM, Jonas Sicking wrote:
> I think we should be explicit about what we do. We could either have
> an API which lets you choose a "attention point" which makes us
> automatically focus and set exposure compensation for that area.
>
> Alternatively we expose separate APIs for setting the focus area and
> the exposure compensation area.
>

> When setting the exposure compensation area, is it possible to get the
> actual EV value back from the hardware? It would be cool if an app
> could display a slider for choosing exposure compensation, while also
> allowing the user to click anywhere in the picture to automatically
> adjust the slider to the value appropriate for that area.
>

> Actually, that would be cool to be able to to for focus too. I.e. it
> would be cool to be able to get the focus distance and be able to
> adjust it with a slider.

I'd really like to have separate focus and exposure controls. I have an iPhone app that lets you control those
separately and its the only way to capture certain pictures accurately.
Lucas.

[Fabrice Desre]

On 05/15/2012 11:01 AM, Jonas Sicking wrote:
>> We can also get the focus distance (and depth of field), but I don't think
>> we can set it manually. There are preset focus modes (Auto, Infinity, Macro,
>> Fixed) and continuous autofocus modes (useful when recording video).
>
> What does "Fixed" mean?

This is the value used for low-end cameras that can't change their focus.

[Fabrice Desre]

Here's an updated version of the API. I added a "capabilities" interface
since lots of settings have values that can not be simply guessed.

See
http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h
for more details on the different properties.

// camera capabilities
// most values are comma separated tuples
interface CameraCapabilities {
attribute readonly DOMString pictureSizes; // Ex: "2048x1536,1024x768"
attribute readonly DOMString pictureFormats;
attribute readonly DOMString jpegThumbnailSizes;
attribute readonly DOMString whiteBalance;
attribute readonly DOMString effects;
attribute readonly DOMString sceneModes;
attribute readonly DOMString flashModes;
attribute readonly DOMString focusModes;
attribute readonly long maxFocusAreas;
attribute readonly double minExposureCompensation;
attribute readonly double maxExposureCompensation;
attribute readonly double stepExposureCompensation;
attribute readonly long maxMeteringAreas;
attribute readonly DOMString zoomRatios;
attribute readonly boolean zoomSupported;
attribute readonly DOMString videoSizes;
}

interface CameraControl {
attribute readonly CameraCapabilities capabilities;

// takes a full resolution capture. Passes a DOMFile to successCB.
void takePicture(successCB, [errorCB]);

// asks the hardware to autofocus. successCB is called when the
// autofocus has completed.
void autofocus(successCB, [errorCB]);

// Starts recording. Passes a DOMFile to successCB.
startRecording(successCB);

// stops an ongoing recording. Will cause the successCB
// from startRecording to be called.
stopRecording();

// set of properties used for the image saved by takePicture()
attribute DOMString pictureSize; // "1024x768"
attribute DOMString pictureFormat;
attribute DOMString jpegThumbnailSize;
attribute long jpegThumbnailQuality; // 0..100
attribute long jpegQuality; // 0..100
attribute long rotation; // The rotation angle in degrees
relative to the orientation of the camera.
attribute double gpsLongitude;
attribute double gpsLatitude;
attribute double gpsAltitude;
attribute long gpsTimestamp; // UTC in seconds since January 1, 1970

attribute DOMString videoSize;

attribute DOMString whiteBalance; // "auto", "incandescent", ...

attribute DOMString effect; // "none", "mono", "sepia", ...

attribute DOMString sceneMode; // "auto", "night", "fireworks", ...

attribute DOMString flashMode; // "on", "off" or "auto"

attribute DOMString focusMode; // "auto", "macro", "fixed", ...
attribute DOMString focusAreas;

attribute readonly double focalLength;

attribute double exposureCompensation;

attribute DOMString meteringAreas;

attribute double zoom;

attribute DOMString focusDistances;
}

Stuff we could still add:
- exposure and whitebalance locks
- face detection features

[JOSE MANUEL CANTERA FONSECA]

El 16/05/12 08:42, "Fabrice Desre" <fab...@mozilla.com> escribió:

>Here's an updated version of the API. I added a "capabilities" interface
>since lots of settings have values that can not be simply guessed.
>
>See
>http://androidxref.com/source/xref/frameworks/base/include/camera/CameraPa
>rameters.h
>for more details on the different properties.
>
>// camera capabilities
>// most values are comma separated tuples
>interface CameraCapabilities {
> attribute readonly DOMString pictureSizes; // Ex: "2048x1536,1024x768"

I think it would be better something like the above to avoid parsing a
string

dictionary Dimensions {
unsigned long width;
Unsigned long height;
}

attribute readonly Dimensions[] pictureSizes;

Why is this set here and not passed as a parameter to takePicture()? After
setting these parameters and once the picture is taken how do I know if my
request has been honored?

> attribute DOMString pictureSize; // "1024x768"

attribute Dimensions pictureSize;

>_______________________________________________
>dev-webapi mailing list
>dev-w...@lists.mozilla.org
>https://lists.mozilla.org/listinfo/dev-webapi
>



Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[David Flanagan]

On 5/15/12 11:42 PM, Fabrice Desre wrote:
> Here's an updated version of the API. I added a "capabilities"
> interface since lots of settings have values that can not be simply
> guessed.
>
> See
> http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h
> for more details on the different properties.
>
> // camera capabilities
> // most values are comma separated tuples

Comma-separated strings just don't seem right for a JavaScript API. How
hard would it be to make those (lazily-created?) arrays?

> interface CameraCapabilities {
> attribute readonly DOMString pictureSizes; // Ex: "2048x1536,1024x768"

> attribute readonly DOMString pictureFormats;
> attribute readonly DOMString jpegThumbnailSizes;
> attribute readonly DOMString whiteBalance;

Should this be whiteBalancePresets or something similar and plural?

> attribute readonly DOMString effects;
> attribute readonly DOMString sceneModes;
> attribute readonly DOMString flashModes;
> attribute readonly DOMString focusModes;
> attribute readonly long maxFocusAreas;
> attribute readonly double minExposureCompensation;
> attribute readonly double maxExposureCompensation;
> attribute readonly double stepExposureCompensation;
> attribute readonly long maxMeteringAreas;
> attribute readonly DOMString zoomRatios;

Since you've got zoom as a double below, I can't figure out what
zoomRatios is for here.

> attribute readonly boolean zoomSupported;
> attribute readonly DOMString videoSizes;
> }
>
> interface CameraControl {
> attribute readonly CameraCapabilities capabilities;
>
> // takes a full resolution capture. Passes a DOMFile to successCB.
> void takePicture(successCB, [errorCB]);
>
> // asks the hardware to autofocus. successCB is called when the
> // autofocus has completed.
> void autofocus(successCB, [errorCB]);

I'm uncertain of the use case here. If I want the camera to autofocus,
I want it to be continually autofocusing. I want to know when it thinks
it has focus and when it has lost focus, but I don't think the user
would ever want the camera to be in autofocus mode but not be actively
trying to acquire a sharp focus.

So I think the way this autofocus() method would be used would be that
the success and error callbacks would always just call autofocus() again.

I could also imagine listening for device motion events and then calling
autofocus when the camera stopped moving. But that doesn't help if the
subject is moving, and users will still want continual autofocus, I think.

So should this just be a pair of gotFocusCallback and lostFocusCallback
properties?

>
> // Starts recording. Passes a DOMFile to successCB.
> startRecording(successCB);
>
> // stops an ongoing recording. Will cause the successCB
> // from startRecording to be called.
> stopRecording();
>
> // set of properties used for the image saved by takePicture()

> attribute DOMString pictureSize; // "1024x768"

> attribute DOMString pictureFormat;
Is this the file format? Would fileFormat be a better name then?

> attribute DOMString jpegThumbnailSize;
> attribute long jpegThumbnailQuality; // 0..100
> attribute long jpegQuality; // 0..100
> attribute long rotation; // The rotation angle in degrees
> relative to the orientation of the camera.

I don't understand the rotation attribute

> attribute double gpsLongitude;
> attribute double gpsLatitude;
> attribute double gpsAltitude;
> attribute long gpsTimestamp; // UTC in seconds since January 1,
> 1970
>

I assume these will be written into metadata tags for the photo. Do
these properties actually get data from the GPS or are they just there
so camera apps that read the GPS can set them? If the later, why not
have a more general API for setting jpeg metadata tags? And if the
former, should they be readonly so they're not spoofable?

> attribute DOMString videoSize;
>
> attribute DOMString whiteBalance; // "auto", "incandescent", ...
> attribute DOMString effect; // "none", "mono", "sepia", ...
> attribute DOMString sceneMode; // "auto", "night", "fireworks", ...
>
> attribute DOMString flashMode; // "on", "off" or "auto"
>
> attribute DOMString focusMode; // "auto", "macro", "fixed", ...
> attribute DOMString focusAreas;
>
> attribute readonly double focalLength;
>
> attribute double exposureCompensation;
>
> attribute DOMString meteringAreas;
>
> attribute double zoom;
>
> attribute DOMString focusDistances;
> }
>
> Stuff we could still add:
> - exposure and whitebalance locks
> - face detection features
>
> Fabrice

This API supports a long list of capabilities and settings and I worry
about having to update the API every time we want to support a new
camera feature. So maybe a lower-level API would be better. I agree
with Robert Kaiser that we should expose all available features through
this API. And I think that means that it only makes sense to have a
low-level getProperty/setProperty sort of API. I do like the idea of a
capabilities object whose properties exhaustively enumerate everything
the camera can do. Can you write the API without defining a fixed set
of attributes for the capabilities object?

David

[Fabrice Desré]

On 05/16/2012 12:19 AM, JOSE MANUEL CANTERA FONSECA wrote:

> I think it would be better something like the above to avoid parsing a
> string
>
> dictionary Dimensions {
> unsigned long width;
> Unsigned long height;
> }
>
> attribute readonly Dimensions[] pictureSizes;

Yes, and it looks our new webIDL parser supports arrays in attributes.
We'll also need dictionaries for the focus and metering areas.

>> // set of properties used for the image saved by takePicture()
>

> Why is this set here and not passed as a parameter to takePicture()? After
> setting these parameters and once the picture is taken how do I know if my
> request has been honored?

I like the idea of passing them to the takePicture() function. As far as
knowing if the request has been honored, the error callback should be
called if we can't support one of the parameters.

Fabrice
--
Fabrice Desré
b2g Team
Mozilla Corporation

[Fabrice Desré]

On 05/16/2012 12:40 AM, David Flanagan wrote:

>> // most values are comma separated tuples
> Comma-separated strings just don't seem right for a JavaScript API. How
> hard would it be to make those (lazily-created?) arrays?

Yes, I'll change them to arrays.

>> attribute readonly DOMString whiteBalance;
> Should this be whiteBalancePresets or something similar and plural?

Good catch! Would whiteBalanceModes be ok for you, to match flashModes
and focusModes ?

>> attribute readonly DOMString zoomRatios;
> Since you've got zoom as a double below, I can't figure out what
> zoomRatios is for here.

Yep, it's a bit tricky. The interaction between the zoomRatio and the
zoom level is explained there :
http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h#391

Suggestions to provide someting clearer are welcome.

>> // asks the hardware to autofocus. successCB is called when the
>> // autofocus has completed.
>> void autofocus(successCB, [errorCB]);
> I'm uncertain of the use case here. If I want the camera to autofocus, I
> want it to be continually autofocusing. I want to know when it thinks it
> has focus and when it has lost focus, but I don't think the user would
> ever want the camera to be in autofocus mode but not be actively trying
> to acquire a sharp focus.

There is the FOCUS_MODE_CONTINUOUS_PICTURE and
FOCUS_MODE_CONTINUOUS_VIDEO for that (but we can't get the information
of when we lose or get in focus)

> So I think the way this autofocus() method would be used would be that
> the success and error callbacks would always just call autofocus() again.

autofocus() is meant for one-time focus action. Useful eg. when the
focus mode is FOCUS_MODE_MACRO.

>> // Starts recording. Passes a DOMFile to successCB.
>> startRecording(successCB);
>>
>> // stops an ongoing recording. Will cause the successCB
>> // from startRecording to be called.
>> stopRecording();
>>

>> // set of properties used for the image saved by takePicture()

>> attribute DOMString pictureSize; // "1024x768"
>> attribute DOMString pictureFormat;
> Is this the file format? Would fileFormat be a better name then?

This maps to
http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h#598

Actually I think I will just remove it since the only value that makes
sense is jpeg.

>> attribute DOMString jpegThumbnailSize;
>> attribute long jpegThumbnailQuality; // 0..100
>> attribute long jpegQuality; // 0..100
>> attribute long rotation; // The rotation angle in degrees relative to
>> the orientation of the camera.
> I don't understand the rotation attribute

Long story there:
http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h#174

>> attribute double gpsLongitude;
>> attribute double gpsLatitude;
>> attribute double gpsAltitude;
>> attribute long gpsTimestamp; // UTC in seconds since January 1, 1970
>>
> I assume these will be written into metadata tags for the photo. Do
> these properties actually get data from the GPS or are they just there
> so camera apps that read the GPS can set them? If the later, why not
> have a more general API for setting jpeg metadata tags? And if the
> former, should they be readonly so they're not spoofable?

Yes, this is used as exif meta-data. It is the application's duty to get
the data from the geolocation API and set the fields.

Having a generic way to add meta-data would be nice for sure. Let's save
that for v2 ;)

> This API supports a long list of capabilities and settings and I worry
> about having to update the API every time we want to support a new
> camera feature. So maybe a lower-level API would be better. I agree with
> Robert Kaiser that we should expose all available features through this
> API. And I think that means that it only makes sense to have a low-level
> getProperty/setProperty sort of API. I do like the idea of a
> capabilities object whose properties exhaustively enumerate everything
> the camera can do. Can you write the API without defining a fixed set of
> attributes for the capabilities object?

We can write an introspection API that 1) list the available properties
and 2) uses generic getProperty/setProperty calls, but I don't think
that's a good idea. How is an app expected to support new properties and
provide a good UI for them?

That also makes it harder to support different property types, and will
not help interoperability.

[Mike Habicher]

----- Original Message -----
> From: "David Flanagan" <dfla...@mozilla.com>
> To: "Fabrice Desré" <fab...@mozilla.com>
> Cc: dev-w...@lists.mozilla.org, "Mike Habicher" <mhab...@mozilla.com>
> Sent: Wednesday, 16 May, 2012 7:44:20 PM
> Subject: Re: Draft Camera Control API

>
> On 5/16/12 11:38 AM, Fabrice Desré wrote:
>
> > On 05/16/2012 12:40 AM, David Flanagan wrote:
> >
> >> I'm uncertain of the use case here. If I want the camera to
> >> autofocus, I
> >> want it to be continually autofocusing. I want to know when it
> >> thinks it
> >> has focus and when it has lost focus, but I don't think the user
> >> would
> >> ever want the camera to be in autofocus mode but not be actively
> >> trying
> >> to acquire a sharp focus.
> >
> > There is the FOCUS_MODE_CONTINUOUS_PICTURE and
> > FOCUS_MODE_CONTINUOUS_VIDEO for that (but we can't get the
> > information
> > of when we lose or get in focus)
> >
> >> So I think the way this autofocus() method would be used would be that
> >> the success and error callbacks would always just call autofocus() again.
> >
> > autofocus() is meant for one-time focus action. Useful eg. when the
> > focus mode is FOCUS_MODE_MACRO.
>

> So maybe the clever thing to do is this: when the user taps to select an
> autofocus region or after each picture is taken, I put the camera in a
> non-continuous focus mode and call autofocus(). When I get the success
> callback, I show a green box on screen or something to indicate that
> focus is acquired, and then I switch to continuous mode so that the
> camera will attempt to maintain that focus.

This would go against the usage model of most cameras out there, which is to do a one-shot focus and hold before taking a picture (at some arbitrary time later). As an amateur photographer, my expected usage models would be:

1. Open camera
2. Frame subject
3. Press shutter button, have camera autofocus, and then take picture when autofocus is complete (success or failure)

OR:

1. Open camera
2. Frame subject
3. Tap on screen to select autofocus region
(when autofocus is complete, frame region in green on success, red on failure)
4. Press shutter button to take picture immediately with last-determined focus

That said, I like the idea of a continuous focus *option*; keep in mind that the low-level camera API makes no guarantees that continuous mode will hold focus. I'm not saying it won't work or that we couldn't try it; my DSLR has a continuous focus mode, which is active as long as you hold down the shutter button half-way. It doesn't always get the focus right, which is why it's optional.

--Mike.

[Mike Habicher]

One thought about providing the picture-taking options directly to takePicture(): depending on how long it takes a camera library to process any new settings, this could add a delay to the actual picture capture.

The trend in cameras these days is for as short a button-to-shutter lag as possible.

I don't think this will be a problem with the Nexus S hardware, but just something to keep in mind.

--m.

----- Original Message -----
> From: "Fabrice Desré" <fab...@mozilla.com>
> To: "David Flanagan" <dfla...@mozilla.com>
> Cc: "Mike Habicher" <mi...@mozilla.com>, dev-w...@lists.mozilla.org
> Sent: Thursday, 17 May, 2012 5:45:52 PM
> Subject: Re: Draft Camera Control API
>

> Hi all,
>
> Here's an updated webidl definition for this API.
>
> If there are no major objections, we could start implementing that
> and
> refine based on David's feedback using it in Gaia?
>
> /* See
> https://github.com/android/platform_frameworks_base/blob/master/include/camera/CameraParameters.h
> * for details
> */
>
> interface Blob;
> interface DOMError;
>
> interface Dimension {
> attribute long width;
> attribute long height;
> };
>
> interface FocusOrMeteringArea {
> attribute long left;
> attribute long top;
> attribute long right;
> attribute long bottom;
> attribute long weight;
> };
>
> interface CameraCapabilities {
> readonly attribute Dimension[] pictureSizes;
> readonly attribute DOMString[] pictureFormats;
> readonly attribute Dimension[] jpegThumbnailSizes;
> readonly attribute DOMString[] whiteBalanceModes;
> readonly attribute DOMString[] effects;
> readonly attribute DOMString[] flashModes;
> readonly attribute DOMString[] focusModes;
> readonly attribute long maxFocusAreas;
> readonly attribute double minExposureCompensation;
> readonly attribute double maxExposureCompensation;
> readonly attribute double stepExposureCompensation;
> readonly attribute long maxMeteringAreas;
> readonly attribute double[] zoomRatios;
> readonly attribute boolean zoomSupported;
> readonly attribute Dimension[] videoSizes;
> };
>
> interface PictureOptions {
> attribute Dimension pictureSize;
> attribute DOMString pictureFormat;
> attribute Dimension jpegThumbnailSize;
> attribute long jpegThumbnailQuality;
> attribute long jpegQuality;
> attribute long rotation;

> attribute double gpsLongitude;
> attribute double gpsLatitude;
> attribute double gpsAltitude;
> attribute long gpsTimestamp;

> };
>
> interface VideoOptions {
> attribute Dimension videoSize;
> };
>
> callback ErrorCB = void(DOMError error);
>
> callback CaptureSuccessCB = void(Blob image);
>
> callback AutofocusSuccessCB = void();
>
> interface CameraControl {
> readonly attribute CameraCapabilities capabilities;
>
> void takePicture(PictureOptions options, CaptureSuccessCB
> onsuccess,
> ErrorCB? onerror);
>
> void startRecording(VideoOptions options, CaptureSuccessCB
> onsuccess,
> ErrorCB? onerror);
>
> void stopRecording();
>
> void autofocus(AutofocusSuccessCB onsuccess, ErrorCB? onerror);
>
> attribute DOMString effect;
> attribute DOMString whiteBalanceMode;
> attribute DOMString sceneMode;
> attribute DOMString flashMode;
> attribute double exposureCompensation;
> attribute double zoom;
> attribute FocusOrMeteringArea[] meteringAreas;
>
> readonly attribute double focalLength;
>
> attribute FocusOrMeteringArea[] focusAreas;
> readonly attribute double[] focusDistances;
>
> };

[alex.la...@gmail.com]

Hello,
I'm holding a Hack Jam for the Mozilla Code Party in June 23rd.
After a discussion on IRC with benfrancis and daleharvey, we thought a flashlight app would suit the low level of complexity needed for the students at the Hack Jam.
But we are going to need an API to access the flash. Is there any chance for this to be implemented by 23rd June?

Best regards,
Alex

[alex.la...@gmail.com]

[Mike Habicher]

Hi Alex,

We should have the more user-friendly CameraControl API will likely be ready by June 23rd.

In the mean time, once you have the camera stream opened, you can turn the flash on continuously by calling |setParameter( "flash-mode", "torch" )| -- assuming the underlying hardware supports it. You can determine this by calling |getParameter( "flash-mode-values" )|, which will return a comma-delimited string of supported modes.

See camera.js in bug 740997 for examples.

--m.


----- Original Message -----
> From: "alex lakatos qa" <alex.la...@gmail.com>
> To: "mozilla dev webapi" <mozilla.d...@googlegroups.com>
> Cc: dev-w...@lists.mozilla.org
> Sent: Friday, 25 May, 2012 8:54:35 AM
> Subject: Re: Draft Camera Control API
>

[Jonas Sicking]

On Wed, May 16, 2012 at 12:40 AM, David Flanagan <dfla...@mozilla.com> wrote:
> On 5/15/12 11:42 PM, Fabrice Desre wrote:
>>
>> Here's an updated version of the API. I added a "capabilities" interface
>> since lots of settings have values that can not be simply guessed.
>>
>> See
>> http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h
>> for more details on the different properties.
>>
>> // camera capabilities

>> // most values are comma separated tuples
>
> Comma-separated strings just don't seem right for a JavaScript API. How hard
> would it be to make those (lazily-created?) arrays?

I think we should return a simple frozen "JSON objects". I.e. for
pictureSizes something like: [{ w: 2048, h: 1536}, { w: 1024, h: 768}]

>>  // asks the hardware to autofocus. successCB is called when the
>>  // autofocus has completed.
>>  void autofocus(successCB, [errorCB]);
>

> I'm uncertain of the use case here.  If I want the camera to autofocus, I
> want it to be continually autofocusing.  I want to know when it thinks it
> has focus and when it has lost focus, but I don't think the user would ever
> want the camera to be in autofocus mode but not be actively trying to
> acquire a sharp focus.
>

> So I think the way this autofocus() method would be used would be that the
> success and error callbacks would always just call autofocus() again.
>

> I could also imagine listening for device motion events and then calling
> autofocus when the camera stopped moving.  But that doesn't help if the
> subject is moving, and users will still want continual autofocus, I think.
>
> So should this just be a pair of gotFocusCallback and lostFocusCallback
> properties?

I don't think the above is true at all. For example most SLRs have a
feature where you half-way press the take-picture-button to autofocus.
You can then rotate the camera as you want to get picture to emcompass
whatever you want.

Also, when you ask most cameras to find focus they generally try a few
times, but if they can't find it they simply give up. This way at
least the human can step in and manually set focus.

On mobile devices autofocusing often requires adjusting the focus over
and over until the requested area is detected to be in focus. This
isn't something you want to happen unless needed since it's quite
disruptive and can leave the picture unfocused for several seconds.

I do however agree that it would be cool to have a lost-focus
callback. Mobile devices seem to have something like that, or they
simply refocus on a timer.

>>  attribute DOMString videoSize;
>>
>>  attribute DOMString whiteBalance; // "auto", "incandescent", ...
>>  attribute DOMString effect;       // "none", "mono", "sepia", ...
>>  attribute DOMString sceneMode;    // "auto", "night", "fireworks", ...
>>
>>  attribute DOMString flashMode;    // "on", "off" or "auto"
>>
>>  attribute DOMString focusMode;    // "auto", "macro", "fixed", ...
>>  attribute DOMString focusAreas;
>>
>>  attribute readonly double focalLength;
>>
>>  attribute double exposureCompensation;
>>
>>  attribute DOMString meteringAreas;
>>
>>  attribute double zoom;
>>
>>  attribute DOMString focusDistances;
>> }
>>
>> Stuff we could still add:
>> - exposure and whitebalance locks
>> - face detection features
>>
>>    Fabrice
>

> This API supports a long list of capabilities and settings and I worry about
> having to update the API every time we want to support a new camera feature.
>  So maybe a lower-level API would be better. I agree with Robert Kaiser that
> we should expose all available features through this API.  And I think that
> means that it only makes sense to have a low-level getProperty/setProperty
> sort of API.  I do like the idea of a capabilities object whose properties
> exhaustively enumerate everything the camera can do.  Can you write the API
> without defining a fixed set of attributes for the capabilities object?

I don't think that having a generic getProperty/setProperty API really
solves anything. We still have to standardize what strings you can
pass to that API, and what values it returns. So ultimately it's just
a difference in syntax and I prefer the syntax-style above.

/ Jonas

[Jonas Sicking]

On Wed, May 16, 2012 at 11:38 AM, Fabrice Desré <fab...@mozilla.com> wrote:
>>> attribute readonly DOMString zoomRatios;
>>
>> Since you've got zoom as a double below, I can't figure out what
>> zoomRatios is for here.
>
>
> Yep, it's a bit tricky. The interaction between the zoomRatio and the zoom
> level is explained there :
> http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h#391
>
> Suggestions to provide someting clearer are welcome.

I don't really understand the smooth-zooming stuff. The documents says
that more information is available in Camera.h, but I can't find any.

Could we simply expose a "zoomValues" property which is an array of
supported zoom values ([1, 1.5, 2, 3.2, ...]). Then have a separate
"attribute double zoom" property which you can set to the actual zoom
value.

If smooth zooming works by simply having an upper and lower zoom limit
and allows setting any zoom in between, then we could additionally
expose a "zoomSmooth" property which contains a { min: 1, max: 5 }
object and let "zoomValues" be null. The zoom property could then be
set to any value between min and max. But again, I couldn't find any
documentation on how smooth zooming works.

[Jonas Sicking]

On Wed, May 16, 2012 at 4:44 PM, David Flanagan <dfla...@mozilla.com> wrote:
>>> This API supports a long list of capabilities and settings and I worry
>>> about having to update the API every time we want to support a new
>>> camera feature. So maybe a lower-level API would be better. I agree with
>>> Robert Kaiser that we should expose all available features through this
>>> API. And I think that means that it only makes sense to have a low-level
>>> getProperty/setProperty sort of API. I do like the idea of a
>>> capabilities object whose properties exhaustively enumerate everything
>>> the camera can do. Can you write the API without defining a fixed set of
>>> attributes for the capabilities object?
>>
>>

>> We can write an introspection API that 1) list the available properties
>> and 2) uses generic getProperty/setProperty calls, but I don't think that's
>> a good idea. How is an app expected to support new properties and provide a
>> good UI for them?
>>

> I'm assuming that it is harder to change the API in the future than it is to
> update the implementation and the clients that use it.  So, if we
> standardize this API and then phone cameras all get some new 3D recording
> feature, do we add a capabilities.mozSupports3DRecording property and a
> mozEnable3D property?  If you stick to a primitive string key/string value
> API, then its just a matter of publishing the new key name.  Implementations
> and apps can add support if and when they want without waiting for an API
> update.

I think you have some wrong assumptions here. The only thing that is
hard is breaking existing code. It's not meaningfully easier to add an
additional string to a getProperty API than it is to add a new
property to the interface. Likewise, it's just as hard to remove
support for a string from a getProperty API, as it is to remove a
property from an interface.

/ Jonas

[Jonas Sicking]

On Thu, May 17, 2012 at 2:45 PM, Fabrice Desré <fab...@mozilla.com> wrote:
> Hi all,
>
> Here's an updated webidl definition for this API.
>
> If there are no major objections, we could start implementing that and
> refine based on David's feedback using it in Gaia?
>
> /* See
> https://github.com/android/platform_frameworks_base/blob/master/include/camera/CameraParameters.h
>  * for details
>  */
>
> interface Blob;
> interface DOMError;
>
> interface Dimension {
>  attribute long width;
>  attribute long height;
> };
>
> interface FocusOrMeteringArea {
>  attribute long left;
>  attribute long top;
>  attribute long right;
>  attribute long bottom;
>  attribute long weight;
> };

For what it's worth, I don't think that we need to expose these as
real interfaces. Simple "JSON objects" would work fine IMHO.

> interface CameraCapabilities {
>  readonly attribute Dimension[] pictureSizes;
>  readonly attribute DOMString[] pictureFormats;

I agree with David that fileFormats would be a better name here.

>  readonly attribute Dimension[] jpegThumbnailSizes;

I don't understand what this property is about. Are we going to return
the data both as a full-sized picture, and as a thumbnail? I think
simply returning the fullsized picture and allowing the app to create
a thumbnail is a simpler and more flexible solution.

Or is the thumbnail somehow encoded into the jpeg file? Still not
convinced that this isn't something we can leave up to software.

>  readonly attribute DOMString[] whiteBalanceModes;

The android API can also support ability to lock the auto
whitebalance. But I'm fine with leaving that out for now. I don't
think messing around with white balance manually is terribly common.

>  readonly attribute DOMString[] effects;

I'm not entirely convinced that we need to expose this. It seems like
this can be better done in software. But I could be wrong about that
so I'm fine with keeping this.

>  readonly attribute long        maxFocusAreas;

I'm not really sure I understand how focus areas work. Do applications
*set* the list of focus areas that they want, and then tell the API to
auto-focus, at which point the device uses the list of auto focus
areas

> interface PictureOptions {
>  attribute Dimension pictureSize;
>  attribute DOMString pictureFormat;
>  attribute Dimension jpegThumbnailSize;
>  attribute long      jpegThumbnailQuality;
>  attribute long      jpegQuality;

Again, I'd just remove the thumbnail stuff here.

>  attribute long      rotation;

We need to document the unit and range here. Same goes for a lot of
other variables.

>  attribute double    gpsLongitude;
>  attribute double    gpsLatitude;
>  attribute double    gpsAltitude;
>  attribute long      gpsTimestamp;

I think I'd prefer to remove the 'gps' prefix here since there isn't
anything gps related here really. The data might come from other
sources.

> interface CameraControl {
>  readonly attribute CameraCapabilities capabilities;
>
>  void takePicture(PictureOptions options, CaptureSuccessCB onsuccess,
> ErrorCB? onerror);
>
>  void startRecording(VideoOptions options, CaptureSuccessCB onsuccess,
> ErrorCB? onerror);
>
>  void stopRecording();

Is the idea that the page would use startRecording/stopRecording to
get a preview, and then takePicture to actually take the high quality
still picture? And use just start/stopRecording to record a video of
course.

I'm a little surprised that there's no difference between a "preview
stream" and a "high quality recordable stream". But I'm happy if
that's not the case since it makes the API simpler :)

>  attribute DOMString             effect;
>  attribute DOMString             whiteBalanceMode;
>  attribute DOMString             sceneMode;
>  attribute DOMString             flashMode;
>  attribute double                exposureCompensation;

I think we should add support for "auto" exposure compensation as
well. This is supported in the android API and seems like a very
useful mode. Though it's unclear from the docs if you can set exposure
compensation to auto and then read the exposureCompensation, below
I've assumed that this is possible.

I can't really think of any great way construct such an API though.
Either we could have:

attribute double exposureCompensation;
attribute bool exposureCompensationAuto;

where setting the latter attribute to "true" allows the page to read
the current compensation through the first attribute. Setting the
first attribute would also set the second to "true" which is a bit
ugly.

Another alternative is:

void setExposureCompensation(double? value);
readonly attribute bool exposureCompensation;

Calling setExposureCompensation(null) enables auto mode and the
effective value can always be read through the attribute.

>  attribute double                zoom;
>  attribute FocusOrMeteringArea[] meteringAreas;
>
>  readonly attribute double       focalLength;
>
>  attribute FocusOrMeteringArea[] focusAreas;
>  readonly attribute double[]     focusDistances;

We should expose focusDistances as three separate properties:
focalLengthNear, focalLengthOptimum, focalLengthFar. I wish we could
fire an event whenever these were updated, but I can't see any way in
the android API to get that information short of polling. In general,
the focus part of the android API seems like it could do with a lot of
improvement :(

Either way, we definitely need a focusMode attribute. Right now I
can't see a way to set fixed/auto/macro/etc.

/ Jonas

[Mounir Lamouri]

Why is this API using success/error callbacks instead of DOMRequest
objects as most APIs we've built so far use?

--
Mounir

On 05/16/2012 08:42 AM, Fabrice Desre wrote:
> Here's an updated version of the API. I added a "capabilities" interface
> since lots of settings have values that can not be simply guessed.
>
> See
> http://androidxref.com/source/xref/frameworks/base/include/camera/CameraParameters.h
> for more details on the different properties.
>
> // camera capabilities
> // most values are comma separated tuples

> interface CameraCapabilities {
> attribute readonly DOMString pictureSizes; // Ex: "2048x1536,1024x768"
> attribute readonly DOMString pictureFormats;
> attribute readonly DOMString jpegThumbnailSizes;
> attribute readonly DOMString whiteBalance;

> attribute readonly DOMString effects;
> attribute readonly DOMString sceneModes;
> attribute readonly DOMString flashModes;
> attribute readonly DOMString focusModes;
> attribute readonly long maxFocusAreas;
> attribute readonly double minExposureCompensation;
> attribute readonly double maxExposureCompensation;
> attribute readonly double stepExposureCompensation;
> attribute readonly long maxMeteringAreas;
> attribute readonly DOMString zoomRatios;

> attribute readonly boolean zoomSupported;

> attribute readonly DOMString videoSizes;
> }
>
> interface CameraControl {
> attribute readonly CameraCapabilities capabilities;
>

> // takes a full resolution capture. Passes a DOMFile to successCB.
> void takePicture(successCB, [errorCB]);

>
> // asks the hardware to autofocus. successCB is called when the
> // autofocus has completed.
> void autofocus(successCB, [errorCB]);
>

> // Starts recording. Passes a DOMFile to successCB.
> startRecording(successCB);
>
> // stops an ongoing recording. Will cause the successCB
> // from startRecording to be called.
> stopRecording();
>
> // set of properties used for the image saved by takePicture()
> attribute DOMString pictureSize; // "1024x768"
> attribute DOMString pictureFormat;

> attribute DOMString jpegThumbnailSize;
> attribute long jpegThumbnailQuality; // 0..100
> attribute long jpegQuality; // 0..100
> attribute long rotation; // The rotation angle in degrees
> relative to the orientation of the camera.

> attribute double gpsLongitude;
> attribute double gpsLatitude;
> attribute double gpsAltitude;

> attribute long gpsTimestamp; // UTC in seconds since January 1, 1970
>

[Fabrice Desré]

On 05/29/2012 09:51 AM, Mounir Lamouri wrote:
> Why is this API using success/error callbacks instead of DOMRequest
> objects as most APIs we've built so far use?

Because it is expected to be used with the webRTC apis that will use
callbacks.

Fabrice

[Fabrice Desré]

Thanks for the comments Jonas. Here are a few comments, and I'll update
the IDL.

On 05/29/2012 02:29 AM, Jonas Sicking wrote:

>>
>> interface FocusOrMeteringArea {
>> attribute long left;
>> attribute long top;
>> attribute long right;
>> attribute long bottom;
>> attribute long weight;
>> };
>
> For what it's worth, I don't think that we need to expose these as
> real interfaces. Simple "JSON objects" would work fine IMHO.

ok.

>> readonly attribute Dimension[] jpegThumbnailSizes;
>
> I don't understand what this property is about. Are we going to return
> the data both as a full-sized picture, and as a thumbnail? I think
> simply returning the fullsized picture and allowing the app to create
> a thumbnail is a simpler and more flexible solution.

The jpeg images that the hardware produces includes a thumbnail that
jpeg libs expose. But since we can't access it on the web for now, it
would only be useful when you tranfer images to another device that can
use that. And if we don't expose this, we'll get a thumbnail with the
default size anyway - this is probably good enough

>> readonly attribute long maxFocusAreas;
>
> I'm not really sure I understand how focus areas work. Do applications
> *set* the list of focus areas that they want, and then tell the API to
> auto-focus, at which point the device uses the list of auto focus
> areas

Yes, that's the behavior.

>> interface CameraControl {
>> readonly attribute CameraCapabilities capabilities;
>>
>> void takePicture(PictureOptions options, CaptureSuccessCB onsuccess,
>> ErrorCB? onerror);
>>
>> void startRecording(VideoOptions options, CaptureSuccessCB onsuccess,
>> ErrorCB? onerror);
>>
>> void stopRecording();
>
> Is the idea that the page would use startRecording/stopRecording to
> get a preview, and then takePicture to actually take the high quality
> still picture? And use just start/stopRecording to record a video of
> course.
>
> I'm a little surprised that there's no difference between a "preview
> stream" and a "high quality recordable stream". But I'm happy if
> that's not the case since it makes the API simpler :)

You get the preview stream directly from the getUserMedia() webRTC call.
Here, when you startRecording(), we encode frames (preferably using a hw
encoder), and we produce a DOMFile passed to onsuccess after
stopRecording().

>
> Another alternative is:
>
> void setExposureCompensation(double? value);
> readonly attribute bool exposureCompensation;
>
> Calling setExposureCompensation(null) enables auto mode and the
> effective value can always be read through the attribute.

Let's try the second option!

> We should expose focusDistances as three separate properties:
> focalLengthNear, focalLengthOptimum, focalLengthFar. I wish we could
> fire an event whenever these were updated, but I can't see any way in
> the android API to get that information short of polling. In general,
> the focus part of the android API seems like it could do with a lot of
> improvement :(

Yes, it's not

> Either way, we definitely need a focusMode attribute. Right now I
> can't see a way to set fixed/auto/macro/etc.

Good catch, we have a focusModes attribute in the capabilities, and I
forgot the focusMode here.

[David Flanagan]

On 5/29/12 3:51 PM, Fabrice Desré wrote:
> Thanks for the comments Jonas. Here are a few comments, and I'll
> update the IDL.
>
> On 05/29/2012 02:29 AM, Jonas Sicking wrote:
>
>>> readonly attribute Dimension[] jpegThumbnailSizes;
>>
>> I don't understand what this property is about. Are we going to return
>> the data both as a full-sized picture, and as a thumbnail? I think
>> simply returning the fullsized picture and allowing the app to create
>> a thumbnail is a simpler and more flexible solution.
>
> The jpeg images that the hardware produces includes a thumbnail that
> jpeg libs expose. But since we can't access it on the web for now, it
> would only be useful when you tranfer images to another device that
> can use that. And if we don't expose this, we'll get a thumbnail with
> the default size anyway - this is probably good enough

Why do you say we can't access the thumbnail on the web?

I'm just starting work on code to extract the metadata from jpeg files
(not just files from the camera: any jpeg image that ends up displayed
in the Gallery app). The first time I encounter a new file, I'm going to
scan it to figure out (at least) its dimension and when it was taken. I
don't really understand how the thumbnail is embedded within the jpeg
file, but if its there, I can presumably extract it and store it in my
database for future use. I would guess that doing that would be faster
than copying the jpeg to an offscreen canvas and resizing it.

In the short run, it will probably be easier to create all my thumbnails
the brute-force way. But longer term, it seems quite useful to have
them in the image itself.

David

[Jonas Sicking]

On Tue, May 29, 2012 at 3:51 PM, Fabrice Desré <fab...@mozilla.com> wrote:
> Thanks for the comments Jonas. Here are a few comments, and I'll update the
> IDL.
>
>
> On 05/29/2012 02:29 AM, Jonas Sicking wrote:
>
>>>

>>> interface FocusOrMeteringArea {
>>>  attribute long left;
>>>  attribute long top;
>>>  attribute long right;
>>>  attribute long bottom;
>>>  attribute long weight;
>>> };
>>
>>
>> For what it's worth, I don't think that we need to expose these as
>> real interfaces. Simple "JSON objects" would work fine IMHO.
>
>
> ok.
>
>

>>>  readonly attribute Dimension[] jpegThumbnailSizes;
>>
>>
>> I don't understand what this property is about. Are we going to return
>> the data both as a full-sized picture, and as a thumbnail? I think
>> simply returning the fullsized picture and allowing the app to create
>> a thumbnail is a simpler and more flexible solution.
>
>
> The jpeg images that the hardware produces includes a thumbnail that jpeg
> libs expose. But since we can't access it on the web for now, it would only
> be useful when you tranfer images to another device that can use that. And
> if we don't expose this, we'll get a thumbnail with the default size anyway
> - this is probably good enough

Sounds good to me.

>>> interface CameraControl {
>>>  readonly attribute CameraCapabilities capabilities;
>>>
>>>  void takePicture(PictureOptions options, CaptureSuccessCB onsuccess,
>>> ErrorCB? onerror);
>>>
>>>  void startRecording(VideoOptions options, CaptureSuccessCB onsuccess,
>>> ErrorCB? onerror);
>>>
>>>  void stopRecording();
>>
>>
>> Is the idea that the page would use startRecording/stopRecording to
>> get a preview, and then takePicture to actually take the high quality
>> still picture? And use just start/stopRecording to record a video of
>> course.
>>
>> I'm a little surprised that there's no difference between a "preview
>> stream" and a "high quality recordable stream". But I'm happy if
>> that's not the case since it makes the API simpler :)
>
>
> You get the preview stream directly from the getUserMedia() webRTC call.
> Here, when you startRecording(), we encode frames (preferably using a hw
> encoder), and we produce a DOMFile passed to onsuccess after
> stopRecording().

Hmm... that's a bit odd. I would have expected getUserMedia to return
a recordable stream. Also, wasn't part of the idea of the Camera
Control API that getUserMedia would be implemented using intents, and
that the app implementing the intent would use the Camera Control API?

>> Another alternative is:
>>
>> void setExposureCompensation(double? value);
>> readonly attribute bool exposureCompensation;
>>
>> Calling setExposureCompensation(null) enables auto mode and the
>> effective value can always be read through the attribute.
>
>
> Let's try the second option!

Sounds good.

/ Jonas

[Jonas Sicking]

On Tue, May 29, 2012 at 9:25 PM, David Flanagan <dfla...@mozilla.com> wrote:
> On 5/29/12 3:51 PM, Fabrice Desré wrote:
>>
>> Thanks for the comments Jonas. Here are a few comments, and I'll update
>> the IDL.
>>
>> On 05/29/2012 02:29 AM, Jonas Sicking wrote:
>>
>>>>  readonly attribute Dimension[] jpegThumbnailSizes;
>>>
>>>
>>> I don't understand what this property is about. Are we going to return
>>> the data both as a full-sized picture, and as a thumbnail? I think
>>> simply returning the fullsized picture and allowing the app to create
>>> a thumbnail is a simpler and more flexible solution.
>>
>>
>> The jpeg images that the hardware produces includes a thumbnail that jpeg
>> libs expose. But since we can't access it on the web for now, it would only
>> be useful when you tranfer images to another device that can use that. And
>> if we don't expose this, we'll get a thumbnail with the default size anyway
>> - this is probably good enough
>

> Why do you say we can't access the thumbnail on the web?
>
> I'm just starting work on code to extract the metadata from jpeg files (not
> just files from the camera: any jpeg image that ends up displayed in the
> Gallery app). The first time I encounter a new file, I'm going to scan it to
> figure out (at least) its dimension and when it was taken. I don't really
> understand how the thumbnail is embedded within the jpeg file, but if its
> there, I can presumably extract it and store it in my database for future
> use.  I would guess that doing that would be faster than copying the jpeg to
> an offscreen canvas and resizing it.
>
> In the short run, it will probably be easier to create all my thumbnails the
> brute-force way.  But longer term, it seems quite useful to have them in the
> image itself.

You're right, you can definitely extract the thumbnail picture by
reading it out of the file yourself. You might even be able to use
Blob.slice() to get a valid jpeg Blob which you can pass to an <img>
element.

However likewise you can also edit the Blob returned from the API to
modify whatever thumbnail is there. There's definitely no *need* to
have the hardware do that (though that might be more convenient and
use less batteries, but unlikely meaningfully so).

/ Jonas

[Fabrice Desré]

On 05/30/2012 12:56 AM, Jonas Sicking wrote:

>>
>> You get the preview stream directly from the getUserMedia() webRTC call.
>> Here, when you startRecording(), we encode frames (preferably using a hw
>> encoder), and we produce a DOMFile passed to onsuccess after
>> stopRecording().
>
> Hmm... that's a bit odd. I would have expected getUserMedia to return
> a recordable stream. Also, wasn't part of the idea of the Camera
> Control API that getUserMedia would be implemented using intents, and
> that the app implementing the intent would use the Camera Control API?

So, we'll get a mediastream recorder API later (I can't find our
proposal on that, but I'm sure Roc wrote one). We could probably use
instead of this, but we can't really wait for b2g.

For the activities part:
- getUserMedia({picture: true}) and <input accept=image/*> will be
implemented by an activity in the camera app (that will send back a blob).
- this activity will use getUserMedia({video: true})

[Jonas Sicking]

This feels very confusing. In some cases getUserMedia invokes intents
and thus isn't affected by the Camera Control API, but in other cases
getUserMedia doesn't use intents and thus is affected by the Camera
Control API.

Wouldn't it make more sense that we have an API on the Camera Control
API which returns a preview stream? This way we are staying out of the
way for getUserMedia and it can evolve freely.

/ Jonas

[Fabrice Desre]

On 06/04/2012 02:04 AM, Jonas Sicking wrote:

> Wouldn't it make more sense that we have an API on the Camera Control
> API which returns a preview stream? This way we are staying out of the
> way for getUserMedia and it can evolve freely.

We could, but isn't it just replicating here getUserMedia({ video: true
}) ? One upside is that it would make the security model clearer, by
limiting the access to this low-level API to trusted apps, and letting
any page use getUserMedia();

Fabrice
--
Fabrice Desr�
b2g team
Mozilla Corporation

[Mike Habicher]

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Fabrice Desré" <fab...@mozilla.com>
> Cc: dev-w...@lists.mozilla.org

> Sent: Monday, 4 June, 2012 5:04:18 AM
> Subject: Re: Draft Camera Control API
>

> On Wed, May 30, 2012 at 11:53 AM, Fabrice Desré <fab...@mozilla.com>
> wrote:
> > On 05/30/2012 12:56 AM, Jonas Sicking wrote:
> >
> >>>
> >>> You get the preview stream directly from the getUserMedia()
> >>> webRTC call.
> >>> Here, when you startRecording(), we encode frames (preferably
> >>> using a hw
> >>> encoder), and we produce a DOMFile passed to onsuccess after
> >>> stopRecording().
> >>
> >>
> >> Hmm... that's a bit odd. I would have expected getUserMedia to
> >> return a recordable stream. Also, wasn't part of the idea of the Camera
> >> Control API that getUserMedia would be implemented using intents,
> >> and that the app implementing the intent would use the Camera Control
> >> API?

Some technical background: on the Nexus S, the preview stream is only guaranteed a frame rate within the range of 15 to 30 frames per second. I'm not sure why it does this, possibly to same battery power, or to keep the viewfinder from hogging all of the CPU. I haven't played with the video recorder yet, but I assume it makes a better effort to keep the frame rate up.

> > So, we'll get a mediastream recorder API later (I can't find our
> > proposal on
> > that, but I'm sure Roc wrote one). We could probably use instead of
> > this,
> > but we can't really wait for b2g.
> >
> > For the activities part:
> > - getUserMedia({picture: true}) and <input accept=image/*> will be
> > implemented by an activity in the camera app (that will send back a
> > blob).
> > - this activity will use getUserMedia({video: true})
>
> This feels very confusing. In some cases getUserMedia invokes intents
> and thus isn't affected by the Camera Control API, but in other cases
> getUserMedia doesn't use intents and thus is affected by the Camera
> Control API.
>

> Wouldn't it make more sense that we have an API on the Camera Control
> API which returns a preview stream? This way we are staying out of
> the way for getUserMedia and it can evolve freely.

I agree with this idea. It makes sense to me to have the the preview stream available as just another "control" on the camera API that the app can choose to use or not. (I can't speak as to why an app may want to use the camera without a view finder, but maybe someone will have a reason, so why force them into a less flexible usage model?)

e.g.,

var whichOne = 'front';
navigator.cameras.getCameraControl( whichOne, /* onSuccess */ function( cameraControl ) {
cameraControl.flashMode( 'on' );
cameraControl.focusMode( 'macro' );
if( previewRequired ) {
cameraControl.getPreviewStream( previewOptions, /* onSuccess */ function( stream ) {
viewfinder.src = stream;
},
/* onError */ function( error ) {
console.log( "failed to get preview stream: " + error );
});
} else {
cameraControl.takePicture( pictureOptions, /* onSuccess */ function( blob ) {
doSomethingWithImageBlob( blob );
},
/* onError */ function( error ) {
console.log( "failed to take picture: " + error );
});
}
},
/* onError */ function( error ) {
console.log( "failed to get " + whichOne + " camera: " + error );
});

Something like the above code could be wrapped in some chrome JS and exported to untrusted pages as getUserMedia().

One potential issue: is the stream returned by getUserMedia() intended to be used as-is (either for recording or streaming)? If so, is the best-effort frame-rate stream going to be a problem?

--m.

[Jonas Sicking]

On Mon, Jun 4, 2012 at 9:47 AM, Fabrice Desre <fab...@mozilla.com> wrote:
> On 06/04/2012 02:04 AM, Jonas Sicking wrote:
>

>> Wouldn't it make more sense that we have an API on the Camera Control
>> API which returns a preview stream? This way we are staying out of the
>> way for getUserMedia and it can evolve freely.
>
>

> We could, but isn't it just replicating here getUserMedia({ video: true }) ?
> One upside is that it would make the security model clearer, by limiting the
> access to this low-level API to trusted apps, and letting any page use
> getUserMedia();

It does seem to me that the model is simpler and clearer if the Camera
Control API and the getUserMedia APIs are both separate and self
sufficient. Even if that means that for now we'd have two equivalent
APIs.

Does getUserMedia({ video: true }) have a plan for things like focus
handling, exposure compensation etc?

/ Jonas

[Mike Habicher]

Hi everyone,

Here is the XPIDL for the CameraControl API. It has been updated to incorporate the changes discussed on this thread. Please look it over and let me know if you have any concerns or second thoughts or if I've missed anything.

Thanks!

--Mike.


---------- 8< ----------
#include "domstubs.idl"

#include "nsIDOMMediaStream.idl"
#include "nsIDOMDOMRequest.idl"
#include "nsIDOMFile.idl"

[scriptable, uuid(64196840-0d03-4b65-a955-790f43a4b810)]
interface nsICameraCapabilities : nsISupports
{
/* an array of objects with 'height' and 'width' properties
supported for the preview stream */
readonly attribute jsval previewSizes;

/* an array of objects with 'height' and 'width' properties
supported for picture taking */
readonly attribute jsval pictureSizes;

/* an array of strings, e.g. [ "jpeg", "rgb565" ] */
readonly attribute jsval fileFormats;

/* an array of strings, e.g. [ "auto", "fluorescent", etc. ] */
readonly attribute jsval whiteBalanceModes;

/* an array of strings, e.g. [ "auto", "night", "beach", etc. ] */
readonly attribute jsval sceneModes;

/* an array of strings, e.g. [ "normal", "sepia", "mono", etc. ] */
readonly attribute jsval effects;

/* an array of strings, e.g. [ "auto", "off", "on", etc. ] */
readonly attribute jsval flashModes;

/* an array of strings, e.g. [ "auto", "fixed", "macro", etc. ] */
readonly attribute jsval focusModes;

/* the maximum number of focus areas supported by the camera */
readonly attribute long maxFocusAreas;

/* the minimum supported exposure compensation value */
readonly attribute double minExposureCompensation;

/* the maximum supported exposure compensation value */
readonly attribute double maxExposureCompensation;

/* exposure compensation minimum step-size */
readonly attribute double stepExposureCompensation;

/* the maximum number of metering areas supported by the camera */
readonly attribute long maxMeteringAreas;

/* an array of doubles, e.g. [ 1.0, 1.2, 1.5, 2.0, 3.0, etc. ],
or null if zooming is not supported */
readonly attribute jsval zoomRatios;

/* an array of objects with 'height' and 'width' properties
supported for video recording */
readonly attribute jsval videoSizes;
};

/*
These properties only affect the captured image;
invalid property settings are ignored.
*/
[scriptable, uuid(e127cbfc-bc60-4621-97b3-cb78d2549237)]
interface nsICameraPictureOptions : nsISupports
{
/* an object with a combination of 'height' and 'width' properties
chosen from nsICameraCapabilities.pictureSizes */
attribute jsval pictureSize;

/* one of the file formats chosen from
nsICameraCapabilities.fileFormats */
attribute DOMString fileFormat;

/* the rotation of the image in degrees, from 0 to 270 in
steps of 90; this doesn't affect the image, only the
rotation recorded in the image header.*/
attribute long rotation;

/* an object containing any or all of 'latitude', 'longitude',
'elevation', and 'timestamp', used to record when and where
the image was taken. e.g.
{
latitude: 43.647118,
longitude: -79.3943,
elevation: 500
// timestamp not specified, in this case
}

can be null in the case where position information isn't
available/desired.

'elevation' is in metres; 'timestamp' is UTC, in seconds from
January 1, 1970.
*/
attribute jsval position;
};

[scriptable, function, uuid(0444a687-4bc9-462c-8246-5423f0fe46a4)]
interface nsICameraPreviewStreamCallback : nsISupports
{
void handleEvent(in nsIDOMMediaStream stream);
};

[scriptable, function, uuid(6baa4ac7-9c25-4c48-9bb0-5193b38b9b0a)]
interface nsICameraAutoFocusCallback : nsISupports
{
void handleEvent();
};

[scriptable, function, uuid(17af779e-cb6f-4ca5-890c-06468ff82e4f)]
interface nsICameraTakePictureCallback : nsISupports
{
void handleEvent(in nsIDOMBlob picture);
};

[scriptable, function, uuid(ac43f123-529c-48d3-84dd-ad206b7aca9b)]
interface nsICameraStartRecordingCallback : nsISupports
{
void handleEvent(); /* need to determine parameters */
};

[scriptable, function, uuid(fb80db71-e315-42f0-9ea9-dd3dd312ed70)]
interface nsICameraShutterCallback : nsISupports
{
void handleEvent();
}

[scriptable, function, uuid(a302c6c9-3776-4d1d-a395-f4105d47c3d3)]
interface nsICameraErrorCallback : nsISupports
{
void handleEvent(in nsIDOMError error);
};

/*
attributes here affect the preview, any pictures taken, and/or
any video recorded by the camera.
*/
[scriptable, uuid(3066c884-d2c3-4477-847d-08ea1c2d188a)]
interface nsICameraControl : nsISupports
{
readonly attribute nsICameraCapabilities capabilities;

/* one of the vales chosen from capabilities.effects;
default is "none" */
attribute DOMString effect;

/* one of the values chosen from capabilities.whiteBalanceModes;
default is "auto" */
attribute DOMString whiteBalanceMode;

/* one of the valus chosen from capabilities.sceneModes;
default is "auto" */
attribute DOMString sceneMode;

/* one of the values chosen from capabilities.flashModes;
default is "auto" */
attribute DOMString flashMode;

/* one of the values chosen from capabilities.focusModes;
default is "auto", if supported, or "fixed" */
attribute DOMString focusMode;

/* one of the values chosen from capabilities.zoomRatios; other
values will be rounded to the nearest supported value;
default is 1.0 */
attribute double zoom;

/* an array of one or more objects that define where the
camera will perform light metering, each defining the properties:
{
top: -1000,
left: -1000,
bottom: 1000,
right: 1000,
weight: 1000
}

'top', 'left', 'bottom', and 'right' all range from -1000 at
the top-/leftmost of the sensor to 1000 at the bottom-/rightmost
of the sensor.

objects missing one or more of these properties will be ignored;
if the array contains more than capabilities.maxMeteringAreas,
extra areas will be ignored.

this attribute can be set to null to allow the camera to determine
where to perform light metering. */
attribute jsval meteringAreas;

/* an array of one or more objects that define where the camera will
perform auto-focusing, with the same definition as meteringAreas.

if the array contains more than capabilities.maxFocusAreas, extra
areas will be ignored.

this attribute can be set to null to allow the camera to determine
where to focus. */
attribute jsval focusAreas;

/* focal length in millimetres */
readonly attribute double focalLength;

/* the distances in metres to where the image subject appears to be
in focus. 'focusDistanceOptimum' is where the subject will appear
sharpest; the difference between 'focusDistanceFar' and
'focusDistanceNear' is the image's depth of field.

'focusDistanceFar' may be infinity. */
readonly attribute double focusDistanceNear;
readonly attribute double focusDistanceOptimum;
readonly attribute double focusDistanceFar;

/* 'compensation' is optional, and if missing, will
set the camera to use automatic exposure compensation.

acceptable values must range from minExposureCompensation
to maxExposureCompensation in steps of stepExposureCompensation;
invalid values will be rounded to the nearest valid value. */
void setExposureCompensation([optional] in double compensation);
readonly attribute double exposureCompensation;

/* the function to call on the camera's shutter event, to trigger
a shutter sound and/or a visual shutter indicator. */
attribute nsIShutterCallback onShutter;

/* tell the camera to attempt to focus the image */
void autoFocus(in nsICameraAutoFocusCallback onSuccess, [optional] in nsICameraErrorCallback onError);

/* capture an image and return it as a blob to the 'onSuccess' callback;
if the camera supports it, this may be invoked while the camera is
already recording video.

invoking this function will stop the preview stream, which must be
manually restarted (e.g. by calling .play() on it). */
void takePicture(in nsICameraTakePictureCallback onSuccess, [optional] in nsICameraErrorCallback onError);

/* start recording video; 'aOptions' define the frame size of to
capture, chosen from capabilities.videoSizes, e.g.:
{
width: 640,
height: 480
}
*/
void startRecording(in jsval aOptions, in nsICameraStartRecordingCallback onSuccess, [optional] in nsICameraErrorCallback onError);

/* stop precording video. */
void stopRecording();

/* get a media stream to be used as a camera viewfinder; the options
define the desired frame size of the preview, chosen from
capabilities.previewSizes, e.g.:
{
height: 640,
width: 480,
}
*/
[implicit_jscontext]
void getPreviewStream(in jsval aOptions, in nsICameraPreviewStreamCallback onSuccess, [optional] in nsICameraErrorCallback onError);
};

[scriptable, uuid(671ee624-0336-441a-a24e-26b5319f14ff)]
interface nsICameraManager : nsISupports
{
/* get a camera instance; options will be used to specify which
camera to get from the list returned by getListOfCameras(), e.g.:
{
camera: front
}
*/
[implicit_jscontext]
nsICameraControl getCamera([optional] in jsval aOptions);

/* return a JSON array of camera identifiers, e.g.
[ "front", "back" ]
*/
[implicit_jscontext]
jsval getListOfCameras();
};

> > Wouldn't it make more sense that we have an API on the Camera
> > Control
> > API which returns a preview stream? This way we are staying out of
> > the way for getUserMedia and it can evolve freely.
>