Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item

[Anupam Snigdha]

Contact emails

sni...@microsoft.com, m...@chromium.org

Explainer

https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata

Specification

https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata

Summary

Add promise support to `ClipboardItem` object. This helps the web author to call async clipboard write method without having to provide the Blob data synchronously. Authors can choose to resolve the promise later when the Blob data is available.

Blink component

Blink>DataTransfer

TAG review

N/A. The spec is in Working Draft state and has already been shipped by Apple.

TAG review status

Not applicable

Risks

None. Currently the API takes in a Blob type and the promise to a Blob would resolve implicitly which wouldn’t require any changes nor would it break any existing sites.

Interoperability and Compatibility

Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)

WebKit: Shipped

Web developers: Positive (https://bugs.chromium.org/p/chromium/issues/detail?id=1014310). This is also a highly requested feature by MS Office products.

Debuggability

The async clipboard APIs have basic tooling support as described in this doc.

Is this feature fully tested by web-platform-tests?

Yes

Flag name

None.

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1014310

Estimated milestones

96

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5733949474078720

This intent message was generated by Chrome Platform Status.

 

[Yoav Weiss]

On Wed, Sep 29, 2021 at 3:02 AM 'Anupam Snigdha' via blink-dev <blin...@chromium.org> wrote:

Contact emails

sni...@microsoft.com, m...@chromium.org

Explainer

https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata

That's not really an explainer. Can you expand on what this method does, how would the new method look like and how would developers use it?

 

Specification

https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata

Summary

Add promise support to `ClipboardItem` object. This helps the web author to call async clipboard write method without having to provide the Blob data synchronously. Authors can choose to resolve the promise later when the Blob data is available.

Blink component

Blink>DataTransfer

TAG review

N/A. The spec is in Working Draft state and has already been shipped by Apple.

Was this change discussed in a standards venue?

TAG review status

Not applicable

Risks

None. Currently the API takes in a Blob type and the promise to a Blob would resolve implicitly which wouldn’t require any changes nor would it break any existing sites.

Is there compat risk here? Are developers already using the non-Promise method? What is it returning today?

What should adoption patterns  to avoid risks in non-supporting browsers?

Interoperability and Compatibility

Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)

It's not clear what this issue actually implements. Did they already implement and ship the Promise based API?

If not, it'd be good to ask for official signals: https://bit.ly/blink-signals

 

WebKit: Shipped

Web developers: Positive (https://bugs.chromium.org/p/chromium/issues/detail?id=1014310). This is also a highly requested feature by MS Office products.

Debuggability

The async clipboard APIs have basic tooling support as described in this doc.

Is this feature fully tested by web-platform-tests?

Yes

Flag name

None.

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1014310

Estimated milestones

96

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5733949474078720

This intent message was generated by Chrome Platform Status.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SN6PR00MB0397C14B0F073635A96B475ACFA99%40SN6PR00MB0397.namprd00.prod.outlook.com.

[Mike Taylor]

On 9/29/21 7:53 AM, Yoav Weiss wrote:

On Wed, Sep 29, 2021 at 3:02 AM 'Anupam Snigdha' via blink-dev <blin...@chromium.org> wrote:

Interoperability and Compatibility

Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)

It's not clear what this issue actually implements. Did they already implement and ship the Promise based API?

If not, it'd be good to ask for official signals: https://bit.ly/blink-signals

https://bugzilla.mozilla.org/show_bug.cgi?id=1712122 was probably the intended bug, but agree a signal would be useful as it seems the patch from that bug was abandoned a few months back (https://phabricator.services.mozilla.com/D115768#3889959), and the bug is currently unowned.

[Anupam Snigdha]

That's not really an explainer. Can you expand on what this method does, how would the new method look like and how would developers use it?

 

Sorry for not being clear in my initial I2S. The change is fairly trivial from a developer’s perspective and has no compat risks so I wasn’t sure an i2S was required at all. This is one of the reasons I didn’t prepare an Explainer and other documentations as the changes have no impact to existing users of async clipboard API.

 

The ClipboardItem has a record of string that represents a MIME type and ClipboardItemData that represents Promises to Blobs corresponding to the MIME types.

The current implementation of Clipboarditem’s constructor in Chromium takes Blobs and not Promises to Blobs which is not how it’s defined in the specification. ClipboardItem is only used in async clipboard API for reading/writing data to the clipboard. This API has been standardized and implemented in Chromium & Safari. This doesn’t affect existing sites that use DataTransfer items for copy/paste scenarios.

 

Definition of existing ClipboardItem constructor in Chromium:

interface ClipboardItem {

  [RaisesException] constructor(record<DOMString, Blob> items,

              optional ClipboardItemOptions options = {});

  readonly attribute FrozenArray<DOMString> types;

 

  [

    CallWith=ScriptState

  ] Promise<Blob> getType(DOMString type);

};

Proposed definition of ClipboardItem constructor:

interface ClipboardItem {

  [RaisesException] constructor(record<DOMString, Promise<Blob>> items,

              optional ClipboardItemOptions options = {});

  readonly attribute FrozenArray<DOMString> types;

 

  [

    CallWith=ScriptState

  ] Promise<Blob> getType(DOMString type);

};

 

Currently in Chromium, developers call the async write using the Blob type as shown below:

const html_text = new Blob(

                ['<html><body><div>hello</div></body></html>'], {type: 'text/html'});

const clipboard_item = new ClipboardItem({

                 'text/html': html_text

               });

navigator.clipboard.write([clipboard_item]);

 

With the proposed implementation, they can pass promises to Blobs as shown below:

navigator.clipboard.write([

                   new ClipboardItem({

                       "text/html": Promise.resolve(new Blob(['<p style=\'color: red; font-style: oblique;\'>This text was copied using </p>'], {type: 'text/html'})),}),]);

 

Was this change discussed in a standards venue?

This is an existing spec and have been approved by the Editing WG.

 

Is there compat risk here? Are developers already using the non-Promise method? What is it returning today?

What should adoption patterns  to avoid risks in non-supporting browsers?

No compat risks as shown in the above examples. Developers can keep using the Blobs to ClipboardItem constructor. The promises would be resolved implicitly with my change.

Non supporting browsers are not affected by this change as the ClipboardItem object can only be used in async clipboard APIs.

 

If not, it'd be good to ask for official signals: https://bit.ly/blink-signals

I’m not sure if it requires any signals as the async clipboard API is already in development. Browsers have to implement async clipboard APIs first before making any changes to Clipboarditem object. When they do implement this API, they should ideally follow the spec and implement a promise based ClipboardItem and not what is currently implemented in Chromium.

[Yoav Weiss]

OK, so let me recap my understanding: this is an interop bug fix where we're currently not accepting Promises in the ClipboardItem constructor, but Safari is (and we should).

There's no compat risk with existing content, as non-Promise calls get converted into Promises. Future-compat risk exists with browsers that'd implement Async Clipboard without this, but this intent tries to minimize that risk.

Are there specific WPT tests that cover the desired behavior? Looking at https://wpt.fyi/results/clipboard-apis?label=experimental&label=master&aligned, there's a lot of yellow...

Thanks for writing that down, that greatly clarifies things! :)

[Thomas Steiner]

Also see #c5 https://bugs.webkit.org/show_bug.cgi?id=222262#c5 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfX1OBfHzKxfC49qXbw67NiFx6JkO-Af1oEM_%2B7E9CmUKA%40mail.gmail.com.

--

Thomas Steiner, PhD—Developer Advocate (https://blog.tomayac.com, https://twitter.com/tomayac)

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.23 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom.hTtPs://xKcd.cOm/1181/
-----END PGP SIGNATURE-----

[Anne van Kesteren]

On Wed, Sep 29, 2021 at 7:31 PM 'Anupam Snigdha' via blink-dev
<blin...@chromium.org> wrote:
> The ClipboardItem [...] This API has been standardized and implemented in Chromium & Safari.

I wouldn't call it standardized. Yes, the specification contains some
IDL, but nowhere is it defined what the associated behavior is. See
https://github.com/w3c/clipboard-apis/issues/135 for instance.

[Anupam Snigdha]

Yes, your understanding about interop fix and compat risk is correct. But I would just add one thing -- In the future, browsers shouldn’t implement it without promises to Blobs and deviate from the spec. See Thomas’s comment on this bug where the authors have to do this ugly workaround – The user activation issue is different from promises to Blobs during clipboard.write.

 

Are there specific WPT tests that cover the desired behavior?

I couldn’t find any, but I’m planning to add some tests in crrev.com/c/3169593.

[Yoav Weiss]

Thanks Anne and Thomas for the cross-browser context.

Anupam - looking at the issue Anne posted, it seems Firefox explicitly did not implement this.

I think it'd be interesting to get their opinions as to why, and whether we should align with the current WebKit behavior or the current Chromium one.

Anne - do y'all want to chime in here, or would a standards position issue be the best venue?

[Anne van Kesteren]

On Fri, Oct 1, 2021 at 12:35 PM Yoav Weiss <yoav...@chromium.org> wrote:
> Thanks Anne and Thomas for the cross-browser context.
>
> Anupam - looking at the issue Anne posted, it seems Firefox explicitly did not implement this.
> I think it'd be interesting to get their opinions as to why, and whether we should align with the current WebKit behavior or the current Chromium one.
>
> Anne - do y'all want to chime in here, or would a standards position issue be the best venue?

There is an existing issue:
https://github.com/mozilla/standards-positions/issues/89. The problem
we are having in evaluating all this is that there isn't really any
specification to speak of, as I tried to point out. One can take
guesses as to what the expected behavior is likely to be, but that is
very far from ideal and normally not even close to acceptable, right?

[Yoav Weiss]

That's fair. 

From my perspective, the ideal outcome here would be:

1) Getting agreement on this specific issue that's currently causing developer pain, and moving forward in that direction.

2) Properly specifying the correct behavior for the rest of the broader API, in ways that would enable interoperable implementations (and reduce developer pain in the future).

It seems like (2) requires Someone™ to take on the work of gathering the different unspecified behaviors, opening an issue to discuss each one, reaching agreement on them and pushing to align existing implementations. 

Maybe we can decouple (1) and (2), but I'd like to see we have a concrete plan for (2) before we do that.

Anupam - who would be best positioned to take on that work?

[Anupam Snigdha]

For #1: I don’t think we would want to diverge from the spec. There is a reason why we have promises to Blobs and not just Blobs in the ClipboardItem because, well, not having promises defeats the purpose of having an async API. Also waiting for the Blob data synchronously without triggering the clipboard write operation leads to problems like this and performance issues in sites like Excel Online where the copy payload is in MBs.

 

For #2: This might sound more of a rant so apologies in advance.

 

I agree with Anne that the spec is not really clear at all on the specifics of the async clipboard API and some of the terminologies used in the algorithms.

However, making changes to the spec or even clarifying the language after an API has been shipped, is really hard, as we need to get consensus from all browser vendors.

I tried to clarify what “sanitization” means just for the HTML format and Apple opposed to this change. Perhaps I can add a non-normative note which would at least give some clarity on the sanitization process, but that would probably require UA specific non normative notes which defeats the purpose of standardization.

I also tried to make changes to address Mozilla’s concern about mandatory data types supported by async clipboard APIs: https://github.com/w3c/clipboard-apis/pull/155, but this PR has been sitting for almost a month now and I’m not able to make any progress.

 

In order to make progress on spec changes, we decided to have a discussion with the Editing WG members and submit changes to the spec if no one objects to it. Currently the Editing WG has representatives from Apple and MS who regularly attend the monthly status meetings. So my question is, if we get an approval from the WG, then does that meet the minimum bar to make spec changes?

 

Anyways, I’m working on updating the spec to at least define the Clipboard interface IDL, but since Apple and Chromium browsers have already shipped this API, I don’t think it’s possible to make any significant changes to the  APIs without breaking at least one of the browsers.

This change addresses the discrepancy between the ClipboardItem IDL as defined in the spec(also implemented by Apple) and what is implemented in Chromium. This is not a breaking change so I think the risk is minimal here.

 

From: Yoav Weiss <yoav...@chromium.org>
Sent: Friday, October 1, 2021 4:15 AM
To: Anne van Kesteren <ann...@annevk.nl>
Cc: Anupam Snigdha <sni...@microsoft.com>; blink-dev <blin...@chromium.org>; Marijn Kruisselbrink <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>
Subject: Re: [EXTERNAL] Re: [blink-dev] Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item

 

 

 

On Fri, Oct 1, 2021 at 12:46 PM Anne van Kesteren <ann...@annevk.nl> wrote:

[Anupam Snigdha]

Here is a WIP PR to address the spec issue: https://github.com/w3c/clipboard-apis/pull/158.

[Yoav Weiss]

On Tue, Oct 5, 2021 at 4:02 AM Anupam Snigdha <sni...@microsoft.com> wrote:

Here is a WIP PR to address the spec issue: https://github.com/w3c/clipboard-apis/pull/158.

Can you address feedback from Anne on the PR?

 

 

From: Anupam Snigdha
Sent: Monday, October 4, 2021 10:45 AM
To: 'Yoav Weiss' <yoav...@chromium.org>; Anne van Kesteren <ann...@annevk.nl>
Cc: blink-dev <blin...@chromium.org>; Marijn Kruisselbrink <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>
Subject: RE: [EXTERNAL] Re: [blink-dev] Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item

 

For #1: I don’t think we would want to diverge from the spec. There is a reason why we have promises to Blobs and not just Blobs in the ClipboardItem because, well, not having promises defeats the purpose of having an async API. Also waiting for the Blob data synchronously without triggering the clipboard write operation leads to problems like this and performance issues in sites like Excel Online where the copy payload is in MBs.

That makes sense.

 

 

For #2: This might sound more of a rant so apologies in advance.

 

I agree with Anne that the spec is not really clear at all on the specifics of the async clipboard API and some of the terminologies used in the algorithms.

However, making changes to the spec or even clarifying the language after an API has been shipped, is really hard, as we need to get consensus from all browser vendors.

I tried to clarify what “sanitization” means just for the HTML format and Apple opposed to this change.

That's unfortunate :/

 

Perhaps I can add a non-normative note which would at least give some clarity on the sanitization process, but that would probably require UA specific non normative notes which defeats the purpose of standardization.

I also tried to make changes to address Mozilla’s concern about mandatory data types supported by async clipboard APIs: https://github.com/w3c/clipboard-apis/pull/155, but this PR has been sitting for almost a month now and I’m not able to make any progress.

FWIW, it seemed to have made some progress initially and then stalled. Pinging it may make sense.

 

 

In order to make progress on spec changes, we decided to have a discussion with the Editing WG members and submit changes to the spec if no one objects to it. Currently the Editing WG has representatives from Apple and MS who regularly attend the monthly status meetings. So my question is, if we get an approval from the WG, then does that meet the minimum bar to make spec changes?

That's for the Editing WG to decide. Looking at its charter, it seems the chairs may be able to help move things along (e.g. by bringing decisions to a vote, if no consensus is reached).

[Anupam Snigdha]

Yep, I’ll address the feedback from Anne and mbrodesser (from Mozilla).

Thanks for all your help Anne and Yoav!

 

From: Yoav Weiss <yoav...@chromium.org>
Sent: Thursday, October 7, 2021 12:03 AM
To: Anupam Snigdha <sni...@microsoft.com>

[Anupam Snigdha]

Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers), and Apple has already shipped this feature. Please let me know in case of any concerns.

 

-Anupam

[Yoav Weiss]

LGTM1 to ship conditional that y'all continue to work on PR #158 specifically, and clarifying the spec's processing model in general.

On Thursday, October 21, 2021 at 2:04:53 AM UTC+2 snianu wrote:

Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers)

FWIW, I got curious RE why that *should* work, and did some digging.

It seems like the bindings methods that accept a `Promise<T>` input value call `NativeValueTraits<IDLPromise>` on that value, which casts the value foo into a `Promise.resolve(foo)`, if it wasn't a Promise already.

The same seems to work in WebKit as well. Do you know if this bindings behavior is specified?

Also, can you add tests for both input cases as part of your CLs for this?

[Ashley Gullen]

I'm not sure why this API needs to exist.

It used to be the case that you could not write to the clipboard after await in Chrome. That's because after an await it was no longer in a synchronous user input trigger, and so the call was blocked.

User Activation v2 fixed that. Now you can write to the clipboard after await in Chrome, because the User Activation v2 rules mean you get a 1 second timeout during which you can write to the clipboard. So long as the async code runs quickly (and it almost always will if you're just doing something like reading a Blob), then everything works.

Safari and Firefox don't support User Activation v2, so still have the original problem. It appears this proposal aims to solve the same problem that User Activation v2 does. IMO it would be better if Safari and Firefox just support User Activation v2 as well.

Assuming a browser supports UAv2, the only benefit this API would appear to bring, is the ability to write to the clipboard beyond the 1 second deadline. If the deadline is unlimited, then this effectively grants the web page permission to write to the clipboard *at any time* after the first copy operation.

Is the intent of this API really just to allow an unlimited deadline? Aren't there ways that could be abused?

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7a237c30-9d53-4181-9c5d-1954d2bf6a0cn%40chromium.org.

[Domenic Denicola]

On Thu, Oct 21, 2021 at 5:21 AM Yoav Weiss <yoav...@chromium.org> wrote:

LGTM1 to ship conditional that y'all continue to work on PR #158 specifically, and clarifying the spec's processing model in general.

On Thursday, October 21, 2021 at 2:04:53 AM UTC+2 snianu wrote:

Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers)

FWIW, I got curious RE why that *should* work, and did some digging.

It seems like the bindings methods that accept a `Promise<T>` input value call `NativeValueTraits<IDLPromise>` on that value, which casts the value foo into a `Promise.resolve(foo)`, if it wasn't a Promise already.

The same seems to work in WebKit as well. Do you know if this bindings behavior is specified?

It is: https://webidl.spec.whatwg.org/#es-promise

 

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7a237c30-9d53-4181-9c5d-1954d2bf6a0cn%40chromium.org.

[Anupam Snigdha]

LGTM1 to ship conditional that y'all continue to work on PR #158 specifically, and clarifying the spec's processing model in general

Yes, we’re committed to work on improving the clipboard API spec. That is sort of a prerequisite for Pickling API 😊

 

Also, can you add tests for both input cases as part of your CLs for this?

Yep, already added: https://chromium-review.googlesource.com/c/chromium/src/+/3169593

 

@Ashley Gullen user activation is orthogonal to this change. The intent of this change is to align the Clipboarditem API with the spec and Safari. This is needed to support promises to Blobs so web authors don’t have to create Blobs synchronously. This may not matter much to sites that have small payloads, but sites like Excel Online where the workbook could contain charts, images etc wouldn’t be able to synchronously create Blobs just in time to satisfy the user activation requirement of async clipboard write method.

[Daniel Bratell]

Inspired by the recent talk about user interaction, I feel like there is one thing I want to understand.

So with a Promise you move the execution to a later time. Is it possible here for a malicious page to delay an action to much, much later and then do that clipboard operation on data that was not available at the time of the clipboard operation the user initiated?

If so, could that have security implications?

Could there even be more than one ongoing clipboard operation at a time?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra-e-Y3%3DSn_LQ7qCLKahrg8WaOfoi4LR1TGMN4%3D5-Dn7kQ%40mail.gmail.com.

[Ashley Gullen]

I know that this change aims to align with the specification and Safari. I am questioning why the specification needs this in the first place, given the user activation rules already specified. It looks like if other browsers support the specified user activation rules, then this part of the specification allowing promises for the Clipboard API becomes redundant.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/4c7d0610-b7de-b0d0-91d1-4943f1e4a6c2%40gmail.com.

[Ashley Gullen]

Just to be clear: you can already asynchronously create content to copy to the clipboard in Chrome, because its user activation rules give you 1 second to prepare the data. Assuming sites like Excel Online can prepare their data within 1 second, then it's already possible to do this within user activation rules, and they don't need this API. If the data takes longer than 1 second to prepare, then this API could be useful purely for extending the deadline.

But then as I noted an unlimited deadline seems like it could be abused, as it essentially allows permission to write to the clipboard at any time, rather than close in time to the user initiating a copy.

[Anupam Snigdha]

Thanks Daniel for raising this concern. I have opened an issue to discuss with the Editing WG: https://github.com/w3c/clipboard-apis/issues/161. In the current implementation, I think verifying whether the Document is active or not after the promises have been resolved should mitigate some of the concerns.

[Mike West]

We discussed this in the API owners' meeting, and there's some confusion both about what's specified, and what Chromium's implementation does. I read https://github.com/w3c/clipboard-apis/issues/161 as suggesting that the `ClipboardItem` has a promise which, when resolved, provides access to the current contents of the clipboard (and a response suggests that there's a 1 second limit on that). Step 2.3 of https://www.w3.org/TR/clipboard-apis/#dom-clipboard-read suggests that read operations return "a copy of the system clipboard data", which seems inconsistent.

Can y'all help me understand what our model is here? And what the model ought to be, if it diverges from what we've currently implemented?

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SN6PR00MB03977ABE1FD1C66C98131F60CF859%40SN6PR00MB0397.namprd00.prod.outlook.com.

[Anupam Snigdha]

Hi Mike,

Let me try to clarify few things. Currently there is no user activation requirements in async clipboard read/write in Chromium browsers. e.g. https://clumsy-garnet-meeting.glitch.me/. Just load this page and refresh it without clicking anywhere on the page, you will see the content will get written to the clipboard without any gesture.

Safari, due to security concerns, implemented a gesture requirement to access clipboard via async clipboard APIs. The read()/write() method can only be called inside a trusted user gesture event handler, but the promises to Blobs can be resolved later which gives the web authors the flexibility to not block the thread to populate the payload.

In Pickling API, which adds capability to the async clipboard API to read/write unsanitized content, initially we added a transient user activation requirement because the API lets web authors read/write unsanitized content using a custom clipboard format. Both Chrome security team(see "User Gesture Requirement" section in https://github.com/w3c/editing/issues/315) and TAG(https://github.com/w3ctag/design-reviews/issues/636#issuecomment-857829725) raised the concern that transient user activation is NOT sufficient to give clipboard access to web authors.

This change is required to not only support the Pickling API's user gesture event handling requirements, but also improve the security by adding an even stronger signal of the user intent to copy. We also talked about this in the Editing WG call with Apple and they are opposed to transient user activation and I believe Firefox raised concerns as well(https://github.com/w3c/clipboard-apis/issues/52#issuecomment-599443986).

Re: clipboard spec: I believe the spec needs to be updated so I've started this PR: https://github.com/w3c/clipboard-apis/pull/158.

Please let me know if you have any other quesions/concerns.

-Anupam

---

From: Mike West <mk...@chromium.org>
Sent: Thursday, October 28, 2021 12:17 PM
To: Anupam Snigdha <sni...@microsoft.com>
Cc: brat...@gmail.com <brat...@gmail.com>; dom...@chromium.org <dom...@chromium.org>; yoav...@chromium.org <yoav...@chromium.org>; blin...@chromium.org <blin...@chromium.org>; ann...@annevk.nl <ann...@annevk.nl>; m...@chromium.org <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>

[Mike West]

On Fri, Oct 29, 2021 at 3:56 AM Anupam Snigdha <sni...@microsoft.com> wrote:

Let me try to clarify few things. Currently there is no user activation requirements in async clipboard read/write in Chromium browsers. e.g. https://clumsy-garnet-meeting.glitch.me/. Just load this page and refresh it without clicking anywhere on the page, you will see the content will get written to the clipboard without any gesture.

Writing to the clipboard is quite distinct from reading from the clipboard, isn't it? https://gifted-stingy-ketchup.glitch.me/ actually surfaces a permission prompt when executing `navigator.clipboard.read()`, which is more in line with my expectations.

 

Safari, due to security concerns, implemented a gesture requirement to access clipboard via async clipboard APIs. The read()/write() method can only be called inside a trusted user gesture event handler, but the promises to Blobs can be resolved later which gives the web authors the flexibility to not block the thread to populate the payload.

The question in my mind relates to the content of that promise's resolution. Is the resulting blob created at the point at which the `read()` method is called (as a "copy of the system clipboard data")? Or is the resolution "active" in some sense, returning a blob representing whatever happens to be in the clipboard at the time the promise is resolved?

 

In Pickling API, which adds capability to the async clipboard API to read/write unsanitized content, initially we added a transient user activation requirement because the API lets web authors read/write unsanitized content using a custom clipboard format. Both Chrome security team(see "User Gesture Requirement" section in https://github.com/w3c/editing/issues/315) and TAG(https://github.com/w3ctag/design-reviews/issues/636#issuecomment-857829725) raised the concern that transient user activation is NOT sufficient to give clipboard access to web authors.

That sounds right to me, and points again to the question above: does a click give you access to the clipboard _now_, or at some arbitrary point in the future?

[Anupam Snigdha]

Sorry for the late response (I was out on vacation).

Writing to the clipboard is quite distinct from reading from the clipboard, isn't it? https://gifted-stingy-ketchup.glitch.me/ actually surfaces a permission prompt when executing `navigator.clipboard.read()`, which is more in line with my expectations.

 

Yes, it is, but this feature really affects writing to the clipboard as the promises to the Blobs are provided by the web authors. Thus, browser is not (at least for now) in control of the timing of when the promises are resolved. For reading, we are in complete control of the promises. In Chromium we read all the supported clipboard formats during the read() call and after user grants permission to access the clipboard, and resolve the promises with the Blob data. UAs could choose to not do this and may read the payload and resolve the Blob data when it’s actually queried via getType method.

 

The question in my mind relates to the content of that promise's resolution. Is the resulting blob created at the point at which the `read()` method is called (as a "copy of the system clipboard data")? Or is the resolution "active" in some sense, returning a blob representing whatever happens to be in the clipboard at the time the promise is resolved?

At least in chromium, we read the payload from the clipboard corresponding to the formats during the read() method call (asynchronously so we don’t block the UI thread): https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/clipboard/clipboard_promise.cc;drc=8cc4402c875055fe7989c15f7dfdd04a46c753fc;l=256

 

But like I said, write() affects how web authors process the ClipboardItem object which in turn affects when the content is written to the clipboard. I don’t think there is any issue with the read() API, and this proposal doesn’t affect how Chromium reads formats from the clipboard and when the promises to the Blobs are resolved.

 

does a click give you access to the clipboard _now_, or at some arbitrary point in the future?

Yes, click gives you access to the clipboard, but promises to the blobs don’t have to be resolved within the event handler.

e.g.

Here instead of resolving the promises, you could resolve it later, but the call to the write() method needs to happen inside the click event handler.

<button id="copy-html">Copy text and markup</button>

<div>Then paste in the box below:</div>

<div contenteditable spellcheck="false" style="width: 200px; height: 100px; overflow: hidden; border: 1px solid black;"></div>

<script>

document.getElementById("copy-html").addEventListener("click", event => {

    navigator.clipboard.write([

        new ClipboardItem({

            "text/plain": Promise.resolve("This text was copied using `Clipboard.prototype.write`."),

            "text/html": Promise.resolve("<p style='color: red; font-style: oblique;'>This text was copied using <code>Clipboard.prototype.write</code>.</p>"),

        }),

    ]);

});

</script>

[Joshua Bell]

Drive-by comment on the thread:

There was a question up-thread about whether the 1s user activation was sufficient time for real-world use cases. Unfortunately it is not - some apps that do data transcoding - e.g. image editors that need to composite multiple layers then do an image format encoding pass - can take longer than 1s, especially on lower end hardware.

(I've seen this sort of delay in native apps too, with sufficiently large datasets being copied.)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/DM5PR00MB0391A4E6B8DB02F48B43ED57CF8A9%40DM5PR00MB0391.namprd00.prod.outlook.com.

[Thomas Steiner]

One problem with async blob creation I ran into is dynamically created blobs of unknown type:

    navigator.clipboard.write([
      new ClipboardItem({
        'foo/bar': new Promise(async (resolve) => {
          // Obtain blob data somehow, the type `foo/bar` will be dynamic.
          resolve(new Blob([data], { type: 'foo/bar' }));
        }),
      }),
    ]);

You need to know `foo/bar` for passing it to `ClipboardItem`, but it may only be available after the data has been created. Since Safari and Chrome expire user gestures differently (see https://bugs.webkit.org/show_bug.cgi?id=222262#c5), this renders some copy use cases currently impossible.

[Anupam Snigdha]

This is a good scenario, but I'm not sure if this can be solved using the current design. You probably want a promise to the ClipboardItem? Also, I think the MIME type in the Clipboarditem is only useful when we are specifying DOMString instead of Blobs, which is currently not supported in Chromium. If we create a Blob after the data has been populated, then we can probably just use the MIME type from the Blob during the clipboard write operation. This feature can be implemented just like the promises to blobs without breaking the sites that use ClipboardItem instead of promises to ClipboardItem, so we should definitely consider adding this to the async clipboard API in the future.

The promises to Blobs feature unblocks some of the key scenarios in Excel online as well as scenarios where a promise to the Blob helps in creating the Blob asynchronously without losing the transient user activation delay, so I think this feature will still be useful even if we implement promise to ClipboardItem.

-Anupam

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALgRrLk728Y92SKX%2BEw0mPC_OhTkGqU9geyC%3D70gn4%2BGPPskkA%40mail.gmail.com.

[Thomas Steiner]

On Wed, Nov 3, 2021 at 7:13 PM Anupam Snigdha <snianu.m...@gmail.com> wrote:

This is a good scenario, but I'm not sure if this can be solved using the current design. You probably want a promise to the ClipboardItem? Also, I think the MIME type in the Clipboarditem is only useful when we are specifying DOMString instead of Blobs, which is currently not supported in Chromium. If we create a Blob after the data has been populated, then we can probably just use the MIME type from the Blob during the clipboard write operation. This feature can be implemented just like the promises to blobs without breaking the sites that use ClipboardItem instead of promises to ClipboardItem, so we should definitely consider adding this to the async clipboard API in the future.

The promises to Blobs feature unblocks some of the key scenarios in Excel online as well as scenarios where a promise to the Blob helps in creating the Blob asynchronously without losing the transient user activation delay, so I think this feature will still be useful even if we implement promise to ClipboardItem.

Taking one step back, in my ideal world, Safari would just let me do the following without expiring the user gesture:

    // Works in Chromium, throws in Safari…
    const { data, type } = await getDataOfUnknownType();    

    const blob = new Blob([data], { type });
    navigator.clipboard.write([
      new ClipboardItem({
        [blob.type]: blob,
      }),
    ]);

[Anupam Snigdha]

Gentle ping..

[Mike West]

LGTM2.

After reading through things again, I'm comfortable that this intent doesn't make any substantive changes to the risk of the clipboard API, insofar as the contents of a write to the clipboard are already opaque to the user and under the site's control. The timing of that write is less critical than the timing of a read, which this intent does not affect.

Regarding the spec, I share Yoav's desire for this to be more clearly specified. I'm happy to see that there's been some movement on https://github.com/w3c/clipboard-apis/pull/158 in the meantime, and look forward to more clarification landing in the future.

-mike

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2Bm%3DdJpVtqT753tNoQPkvEZKQrmiDOXj_QuoeESoR-%3DtiH%3DrPw%40mail.gmail.com.

[Daniel Bratell]

LGTM3

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3DeHev9kELPJ5885ZH3Spvv0iuEAPa1UPM%3DZWb636WWisQ%40mail.gmail.com.