Intent to Ship: Barcode Detection API

434 views
Skip to first unread message

Reilly Grant

unread,
Jul 26, 2019, 2:27:28 PM7/26/19
to blink-dev

Contact emails

rei...@chromium.org, mca...@chromium.org 


Explainer

https://github.com/WICG/shape-detection-api/blob/master/README.md


Spec

https://wicg.github.io/shape-detection-api/


TAG review: https://github.com/w3ctag/design-reviews/issues/176


Summary

The BarcodeDetector class gives web content access to the platform’s native support for recognizing barcodes in images. For example, the BarcodeDetector provided on Android devices by the Google Play services library or the Core Image and Vision frameworks provided by macOS. This can provide faster results than solutions built with JavaScript or WebAssembly while also not requiring developers to ship additional code to their users to support a feature their device already provides.


Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/d/msg/blink-dev/JkdoxpINjxQ/CUWOBwc0AgAJ


Origin Trial feedback summary

The primary feedback during the Origin Trial was around the limited availability of barcode detection support across Blink platforms. In response to this feedback we added a getSupportedFormats() method to more easily allow developers to detect when the current platform supports the desired formats and when a polyfill is necessary.


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

This feature is supported on macOS, Android and Android WebView. The reason for this is that it depends on the platform providing built-in support for detecting barcodes in images. Sites are expected to provide a polyfill for this capability when running in a browser or on a platform that does not support it.


Demo link

https://qrsnapper.com/


Risks

Interoperability and Compatibility

This API has been designed to support robust feature detection because barcode detection capabilities are expected to vary across both browsers and platforms. This lessens the interoperability risk as sites are already incentivized to provide a backup polyfill library for the detection capability. If this feature were removed from the platform sites would lose the performance advantage of a native (potentially hardware-accelerated) detection capability but are expected to continue to function.


Edge: No signals

Firefox: Public support

Safari: No signals

Web / Framework developers: Positive


Ergonomics

This API is frequently used with the getUserMedia() API to perform detection on a live video stream. The API supports multiple types of HTML elements as image sources.


Activation

Given the lack of consistent both cross-vendor and cross-platform support this API benefits strongly from polyfill libraries such as the Web Perception Toolkit. This is by design, as platform support is considered an optimization of a capability that sites can already implement using their own script.


Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

The interface exposed by this API is fully tested by a Web Platform Tests suite: https://github.com/web-platform-tests/wpt/tree/master/shape-detection


These tests do not fully pass on wpt.fyi (https://wpt.fyi/results/shape-detection) because they rely on running in a content_shell build that provides interfaces for injecting test data. web-platform-tests/results-collection#81 has been filed to change the options passed to Chrome when running tests to enable these interfaces.


Entry on the feature dashboard

https://chromestatus.com/feature/4757990523535360


Note to jmedley@: This entry is for the Shape Detection API in general. Since I intend to ship each component of the Shape Detection API individually should I repurpose this entry for this intent and create new entries for FaceDetector and TextDetector?

Boris Zbarsky

unread,
Jul 26, 2019, 2:50:00 PM7/26/19
to Reilly Grant, blink-dev
On 7/26/19 2:27 PM, Reilly Grant wrote:
> Firefox: Public support
> <https://discourse.wicg.io/t/rfc-proposal-for-face-detection-api/1642/3>

Maybe I am missing something, but that is a comment in a 2-year-old
thread about a different API (Face detection API) that has a different
purpose, somewhat different API shape, different fingerprinting
characteristics, etc. I don't see how that comment implies anything
about Mozilla's position on this, different, API.

I'd suggest filing an issue at
https://github.com/mozilla/standards-positions/issues/new to figure out
what Mozilla's position here is.

-Boris

Reilly Grant

unread,
Jul 26, 2019, 4:36:50 PM7/26/19
to Boris Zbarsky, blink-dev
On Fri, Jul 26, 2019 at 11:49 AM Boris Zbarsky <bzba...@mit.edu> wrote:
On 7/26/19 2:27 PM, Reilly Grant wrote:
> Firefox: Public support
> <https://discourse.wicg.io/t/rfc-proposal-for-face-detection-api/1642/3>

Maybe I am missing something, but that is a comment in a 2-year-old
thread about a different API (Face detection API) that has a different
purpose, somewhat different API shape, different fingerprinting
characteristics, etc.  I don't see how that comment implies anything
about Mozilla's position on this, different, API.

The WICG Discourse thread started with the particular proposal of a face detection API but included room for detection of additional shapes in images which is why the draft specification is called "Shape Detection API." Marcos was in favor of the more general framing of detectable shapes. This intent proposes shipping one particular subset of detectable shapes, barcodes, for which the API has been sufficiently tested by developers and reviewed by the TAG. Face detection is coming next but there are still some details to work out. Am I wrong to assume that if Mozilla is in favor of the broader proposal for shape detection it would not be in favor of a particular well-developed subset? 

I'd suggest filing an issue at
https://github.com/mozilla/standards-positions/issues/new to figure out
what Mozilla's position here is.

It looks like Marcos did file an RFP for this API: mozilla/standards-positions#21 The answer there was "defer" with the suggestion to re-examine after the Origin Trial (which has been completed). I've pinged the issue and will update the thread here if there is a definitive response. For the time being let's consider Mozilla's position on this to be neutral.

-Boris

Boris Zbarsky

unread,
Jul 26, 2019, 4:44:11 PM7/26/19
to Reilly Grant, blink-dev
On 7/26/19 4:36 PM, Reilly Grant wrote:
> Am I wrong to assume that if Mozilla is in favor of the broader proposal for shape detection
> it would not be in favor of a particular well-developed subset?

It really depends on the specific APIs.

Put another way, Mozilla can be in favor of the general idea of a
barcode detection API while not being in favor of a specific API proposal.

> It looks like Marcos did file an RFP for this API:
> mozilla/standards-positions#21

Great, thank you for finding that. That's the right place to look for a
Mozilla position.

-Boris

Joe Medley

unread,
Jul 29, 2019, 11:41:16 AM7/29/19
to Reilly Grant, blink-dev
What version are you planning to ship in?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEmk%3DMbki5G4JjbWxYM-hf-1%3DxPjf44PCHLki%2BrbHNq8jtSKAg%40mail.gmail.com.

Daniel Bratell

unread,
Jul 29, 2019, 1:20:37 PM7/29/19
to blink-dev, Reilly Grant
Not being cross-platform, and even missing the largest desktop platform (Windows) seems like a bummer. It looks like Windows does have some kind of barcode scanner API ( https://docs.microsoft.com/en-us/windows/uwp/devices-sensors/pos-camerabarcode-get-started ). Could that be used to give this better platform support?

/Daniel
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEmk%3DMbki5G4JjbWxYM-hf-1%3DxPjf44PCHLki%2BrbHNq8jtSKAg%40mail.gmail.com.



--
/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

Reilly Grant

unread,
Jul 29, 2019, 2:07:01 PM7/29/19
to Joe Medley, blink-dev
On Mon, Jul 29, 2019 at 8:41 AM Joe Medley <jme...@google.com> wrote:
What version are you planning to ship in?

M-78

Reilly Grant

unread,
Jul 29, 2019, 2:10:08 PM7/29/19
to Daniel Bratell, blink-dev
On Mon, Jul 29, 2019 at 10:20 AM Daniel Bratell <bra...@opera.com> wrote:
Not being cross-platform, and even missing the largest desktop platform (Windows) seems like a bummer. It looks like Windows does have some kind of barcode scanner API ( https://docs.microsoft.com/en-us/windows/uwp/devices-sensors/pos-camerabarcode-get-started ). Could that be used to give this better platform support?

The Windows barcode scanner API is not appropriate for implementing this API because it tightly integrates with a camera to create a virtual barcode scanner rather than allowing an application to detect a barcode on an existing bitmap. I've been reaching out to the Edge team to see if this it is possible to get this functionality exposed from Windows at the right level to implement this API.

Arvind Murching

unread,
Jul 30, 2019, 12:07:05 AM7/30/19
to blink-dev, bra...@opera.com
Reilly is correct, there isn't a Windows API today that takes a bitmap and detects a barcode.  I've passed along feedback to the OS team.

Arvind
Microsoft Edge/Web Platform

rhal...@google.com

unread,
Aug 1, 2019, 6:08:33 AM8/1/19
to blink-dev
I have a privacy question. The design doc says the raw image data nor the detection results are stored or transmitted by the browser.
  • Does it mean that the webapp is supposed to get permission to access camera and take the photo, then pass it to this API for detection?
  • Do we have any guarantee that the underlying mechanisms do not keep a local history of the images? If not, can't it result in local storage of the image without user's consent?
  • Is there a launch bug for this feature?

Joshua Bell

unread,
Aug 1, 2019, 12:51:10 PM8/1/19
to Ramin Halavati, blink-dev
On Thu, Aug 1, 2019 at 3:08 AM rhalavati via blink-dev <blin...@chromium.org> wrote:
I have a privacy question. The design doc says the raw image data nor the detection results are stored or transmitted by the browser.
  • Does it mean that the webapp is supposed to get permission to access camera and take the photo, then pass it to this API for detection?
Correct. A local image could also be used (e.g. via file upload) or any other means of acquiring an image (e.g. via some online feed) 
  • Do we have any guarantee that the underlying mechanisms do not keep a local history of the images? If not, can't it result in local storage of the image without user's consent?
Neither storage access nor network access are gated by a permission, so it is the case that once a web app has camera access the image can be persisted locally or transmitted to a remote server.

Agreed that this could be clarified that this new API does not expand or restrict those capabilities in any way, but they are already present in the web platform (e.g. once camera access is granted or the user has selected a local image file) 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Reilly Grant

unread,
Aug 1, 2019, 1:43:23 PM8/1/19
to Ramin Halavati, blink-dev
On Thu, Aug 1, 2019 at 3:08 AM rhalavati via blink-dev <blin...@chromium.org> wrote:
I have a privacy question. The design doc says the raw image data nor the detection results are stored or transmitted by the browser.
  • Does it mean that the webapp is supposed to get permission to access camera and take the photo, then pass it to this API for detection?
  • Do we have any guarantee that the underlying mechanisms do not keep a local history of the images? If not, can't it result in local storage of the image without user's consent?
  • Is there a launch bug for this feature?
The launch issue is https://crbug.com/728474.
 
--

Alex Russell

unread,
Aug 1, 2019, 3:28:19 PM8/1/19
to blink-dev, rhal...@google.com
LGTM1
To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

Chris Harrelson

unread,
Aug 1, 2019, 3:36:55 PM8/1/19
to Alex Russell, blink-dev, Ramin Halavati
One question regarding barcode formats: it seems like a pretty big list of current and legacy formats. Is there any concern about implicitly depending on these side-specs in a web-exposed API?

Second question is regarding origin trial feedback: is there any summary of how useful this feature was from the origin trial?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0f15dadf-3100-44ca-8870-8a3286ad24b6%40chromium.org.

Reilly Grant

unread,
Aug 2, 2019, 1:51:25 PM8/2/19
to Chris Harrelson, Alex Russell, blink-dev, Ramin Halavati
On Thu, Aug 1, 2019 at 12:36 PM Chris Harrelson <chri...@chromium.org> wrote:
One question regarding barcode formats: it seems like a pretty big list of current and legacy formats. Is there any concern about implicitly depending on these side-specs in a web-exposed API?

The format specifications themselves seem reasonably well-defined and web-exposed APIs depend on plenty of other side-specifications through other means, for example encryption algorithms by way of HTTPS and TLS. My primary concern is that we may not be referring to them specifically enough. As an example, what if encoding FOO as implemented by Chrome on Android only really decodes some variant FOO_A. Would changes to the specification be needed if another platform gained support for FOO but only variant FOO_B?

Second question is regarding origin trial feedback: is there any summary of how useful this feature was from the origin trial?

Feedback on the Origin Trial was overwhelmingly complaints about the inconsistency in support across different platforms and how that was communicated in a confusing way, which is why I have been focusing on improving the ability to feature detect this capability.

Outside of the Origin Trial I've received feedback from developers in the commercial and industrial space who are interested in the performance advantage of this API over JS/WASM solutions. The API also gets a mention in eBay's recent blog post about implementing a barcode scanner in their application.
 

Chris Harrelson

unread,
Aug 2, 2019, 1:55:18 PM8/2/19
to Reilly Grant, Alex Russell, blink-dev, Ramin Halavati
On Fri, Aug 2, 2019 at 10:51 AM Reilly Grant <rei...@chromium.org> wrote:
On Thu, Aug 1, 2019 at 12:36 PM Chris Harrelson <chri...@chromium.org> wrote:
One question regarding barcode formats: it seems like a pretty big list of current and legacy formats. Is there any concern about implicitly depending on these side-specs in a web-exposed API?

The format specifications themselves seem reasonably well-defined and web-exposed APIs depend on plenty of other side-specifications through other means, for example encryption algorithms by way of HTTPS and TLS. My primary concern is that we may not be referring to them specifically enough. As an example, what if encoding FOO as implemented by Chrome on Android only really decodes some variant FOO_A. Would changes to the specification be needed if another platform gained support for FOO but only variant FOO_B?

That's a good question. Do you think this needs more discussion before shipping?
 

Second question is regarding origin trial feedback: is there any summary of how useful this feature was from the origin trial?

(referring to your response below) This is excellent feedback! Sounds like the origin trial was quite useful.
 

Feedback on the Origin Trial was overwhelmingly complaints about the inconsistency in support across different platforms and how that was communicated in a confusing way, which is why I have been focusing on improving the ability to feature detect this capability.

Feature detecting whether a particular barcode format is supported on a particular platform, you mean?
 

Boris Zbarsky

unread,
Aug 2, 2019, 1:59:58 PM8/2/19
to Reilly Grant, Chris Harrelson, Alex Russell, blink-dev, Ramin Halavati
On 8/2/19 1:51 PM, Reilly Grant wrote:
> which is why I have been focusing on
> improving the ability to feature detect this capability.

I also asked this in
<https://github.com/mozilla/standards-positions/issues/21#issuecomment-515594606>,
but has there been thought on the fingerprinting impact of adding this
detection? Ideally the proposed spec would address that aspect of things.

-Boris

Reilly Grant

unread,
Aug 7, 2019, 2:03:10 PM8/7/19
to Chris Harrelson, Alex Russell, blink-dev, Ramin Halavati
On Fri, Aug 2, 2019 at 10:55 AM Chris Harrelson <chri...@chromium.org> wrote:


On Fri, Aug 2, 2019 at 10:51 AM Reilly Grant <rei...@chromium.org> wrote:
On Thu, Aug 1, 2019 at 12:36 PM Chris Harrelson <chri...@chromium.org> wrote:
One question regarding barcode formats: it seems like a pretty big list of current and legacy formats. Is there any concern about implicitly depending on these side-specs in a web-exposed API?

The format specifications themselves seem reasonably well-defined and web-exposed APIs depend on plenty of other side-specifications through other means, for example encryption algorithms by way of HTTPS and TLS. My primary concern is that we may not be referring to them specifically enough. As an example, what if encoding FOO as implemented by Chrome on Android only really decodes some variant FOO_A. Would changes to the specification be needed if another platform gained support for FOO but only variant FOO_B?

That's a good question. Do you think this needs more discussion before shipping?

I think we've done all the investigation we can on this. I just wanted to mention that it as an inevitable concern when creating an enumeration like this.
 
Second question is regarding origin trial feedback: is there any summary of how useful this feature was from the origin trial?

(referring to your response below) This is excellent feedback! Sounds like the origin trial was quite useful.
 

Feedback on the Origin Trial was overwhelmingly complaints about the inconsistency in support across different platforms and how that was communicated in a confusing way, which is why I have been focusing on improving the ability to feature detect this capability.

Feature detecting whether a particular barcode format is supported on a particular platform, you mean?

Yes.

Chris Harrelson

unread,
Aug 8, 2019, 3:20:42 PM8/8/19
to Reilly Grant, Alex Russell, blink-dev, Ramin Halavati

Chris Harrelson

unread,
Aug 8, 2019, 3:37:36 PM8/8/19
to Reilly Grant, Alex Russell, blink-dev, Ramin Halavati
Make that LGTM2, since Alex already LGTM1'ed.

Yoav Weiss

unread,
Aug 8, 2019, 6:13:10 PM8/8/19
to Chris Harrelson, Reilly Grant, Alex Russell, blink-dev, Ramin Halavati
Regarding the fingerprinting concerns raised in this thread and after talking to Reilly, my understanding is that the formats exposed clearly map to data already exposed by the browser (i.e. the OS part of the UA string).

Given that, LGTM3

At the same time, it would be good to add that to the spec's security and privacy section.
 

Boris Zbarsky

unread,
Aug 8, 2019, 6:49:35 PM8/8/19
to Yoav Weiss, Chris Harrelson, Reilly Grant, Alex Russell, blink-dev, Ramin Halavati
On 8/8/19 6:12 PM, Yoav Weiss wrote:
> Regarding the fingerprinting concerns raised in this thread and after
> talking to Reilly, my understanding is that the formats exposed clearly
> map to data already exposed by the browser (i.e. the OS part of the UA
> string).

Is that still true if the browser purposefully lies about the OS part of
the UA string (e.g. to claim a more widespread OS than the one it's
actually running on)? In that situation would this spec require it to
expose the "correct" format set or the set that corresponds to the OS
it's claiming? And in the latter case, are there spec affordances for
always failing to detect formats that were claimed to be supported, and
is that behavior something consumers will expect?

-Boris

Reilly Grant

unread,
Aug 9, 2019, 6:05:24 PM8/9/19
to Boris Zbarsky, Yoav Weiss, Chris Harrelson, Alex Russell, blink-dev, Ramin Halavati
An implementation that is purposefully lying about the OS part of the UA string could either (1) provide its own implementation of the capabilities normally available on that platform, (2) include the unsupported formats but fail to detect them or (3) return an empty set from getSupportedFormats() to indicate a lack of support for this feature entirely.

(1) is the most compatible option but hardest to implement. (2) will likely break applications that rely on the feature when the UA claims it is available but this may be an acceptable compromise for the user. (3) is compatible and easy to implement. Both (2) and (3) could be detected by a page to identify to the site that the UA is lying about the OS if it is known that the OS should support a particular format.

-Boris

Reilly Grant

unread,
Feb 18, 2020, 2:50:56 PM2/18/20
to Yoav Weiss, Chris Harrelson, Alex Russell, blink-dev, Ramin Halavati
An update, since this was delayed by last-minute polish work that took way too long to find time for: This will be shipping in Chrome 82.
Reilly Grant | Software Engineer | rei...@chromium.org | Google Chrome

Reilly Grant

unread,
Nov 29, 2021, 3:03:13 PM11/29/21
to blink-dev, Chris Harrelson, sligh...@google.com, blink-dev, Ramin Halavati, yo...@yoav.ws
As of Chrome 98 the Barcode Detection API will be available on Windows and Linux as well, making this API available on all supported Chrome platforms (including Chrome OS, which shipped support awhile ago without an announcement).

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Alex Russell

unread,
Nov 30, 2021, 3:07:45 PM11/30/21
to Reilly Grant, blink-dev, Chris Harrelson, sligh...@google.com, Ramin Halavati, yo...@yoav.ws
This is great. Thanks for letting us know!

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e4110d58-a2b3-4ae2-bf43-dea8fd6d8537n%40chromium.org.

Reilly Grant

unread,
Dec 9, 2021, 2:00:42 PM12/9/21
to Alex Russell, blink-dev, Chris Harrelson, sligh...@google.com, Ramin Halavati, yo...@yoav.ws
Apologies, I spoke too soon. We'll be holding off shipping this on Windows and Linux for the time being. Android, macOS and Chrome OS continue to support it.

Reilly Grant | Software Engineer | rei...@chromium.org | Google Chrome

Reply all
Reply to author
Forward
0 new messages