Intent to Implement: Shape Detection API

Skip to first unread message

Miguel Casas-Sanchez

Sep 12, 2016, 3:31:49 PM9/12/16
to blink-dev, Photos and images constitute the largest chunk of the Web, and many include recognisable features, such as human faces or QR codes. Hardware manufacturers have long been providing support for such operations given their (high) computational complexity. The Shape Detection API allows accessing these hardware-accelerated detectors where available. (Note that it does _not_ provide any software fallback)
Detection of features in images is done in JS libraries which are inherently slow due to high computational complexity. This API surfaces hardware capabilities where available.
Firefox: Public support Edge: No public signals Safari: No public signals Web developers: Positive
The API closely mimics the underlying platforms which are provided by stable, widespread mobile platforms, so not much change is expected in this front and, consequently, the (future potential) interoperability risk is low.

Websites use JS libraries which might provide slightly different recognition data events or characteristics, but the underlying functionality is clearly understood now.
Only a few platforms implement this feature, e.g. Android and AVFoundation (both in iOS and Mac).
Rationale: It will only be available where there is Hw/OS support for it, namely: certain Android, AVFoundation (Mac, iOS).

Stephen Chenney

Sep 12, 2016, 3:49:39 PM9/12/16
to Miguel Casas-Sanchez, blink-dev
This feature requires a design doc. It is not clear to me how you will implement it with decent performance, given many images and certainly video never leave the compositor.


Alex Russell

Sep 22, 2016, 1:32:53 PM9/22/16
to Stephen Chenney, Miguel Casas-Sanchez, blink-dev
Many of the use-cases are scoped to video which comes from getUserMedia and which may have hardware support for the detection (without pulling the frames back into the document's main thread).

I'm hopeful that an explainer document will be added to the proposal repo to help clarify which problems are being solved and which are out of scope.


You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to

Rick Byers

Sep 24, 2016, 5:16:43 PM9/24/16
to Alex Russell, Stephen Chenney, Miguel Casas-Sanchez, blink-dev
Exposing the capabilities from Android and iOS here sounds great to me!

Have you started the process of figuring out at the various vendors or in the various related working groups might have the interest and expertise to collaborate on this?  Ultimately we'll probably want some support from at least one other browser before shipping, so trying to drum up some other browser interest once you have a good high-level explainer is probably a good idea.


Miguel Casas-Sanchez

Nov 18, 2016, 4:54:14 PM11/18/16
to blink-dev
DD document is being written in
Reply all
Reply to author
0 new messages