Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item
Anupam Snigdha
Contact emails
sni...@microsoft.com, m...@chromium.org
Explainer
https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata
Specification
https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata
Summary
Add promise support to `ClipboardItem` object. This helps the web author to call async clipboard write method without having to provide the Blob data synchronously. Authors can choose to resolve the promise
later when the Blob data is available.
Blink component
TAG review
N/A. The spec is in Working Draft state and has already been shipped by Apple.
TAG review status
Not applicable
Risks
None. Currently the API takes in a Blob type and the promise to a Blob would resolve implicitly which wouldn’t require any changes nor would it break any
existing sites.
Interoperability and Compatibility
Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)
WebKit: Shipped
Web developers: Positive (https://bugs.chromium.org/p/chromium/issues/detail?id=1014310). This is also a highly requested feature by MS Office products.
Debuggability
The async clipboard APIs have basic tooling support as described in this doc.
Is this feature fully tested by web-platform-tests?
Yes
Flag name
None.
Requires code in //chrome?
False
Tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1014310
Estimated milestones
96
Link to entry on the Chrome Platform Status
https://www.chromestatus.com/feature/5733949474078720
This intent message was generated by Chrome Platform Status.
Yoav Weiss
Contact emails
sni...@microsoft.com, m...@chromium.org
Explainer
https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata
Specification
https://www.w3.org/TR/clipboard-apis/#typedefdef-clipboarditemdata
Summary
Add promise support to `ClipboardItem` object. This helps the web author to call async clipboard write method without having to provide the Blob data synchronously. Authors can choose to resolve the promise later when the Blob data is available.
Blink component
TAG review
N/A. The spec is in Working Draft state and has already been shipped by Apple.
TAG review status
Not applicable
Risks
None. Currently the API takes in a Blob type and the promise to a Blob would resolve implicitly which wouldn’t require any changes nor would it break any existing sites.
Interoperability and Compatibility
Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)
WebKit: Shipped
Web developers: Positive (https://bugs.chromium.org/p/chromium/issues/detail?id=1014310). This is also a highly requested feature by MS Office products.
Debuggability
The async clipboard APIs have basic tooling support as described in this doc.
Is this feature fully tested by web-platform-tests?
Yes
Flag name
None.
Requires code in //chrome?
False
Tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1014310
Estimated milestones
96
Link to entry on the Chrome Platform Status
https://www.chromestatus.com/feature/5733949474078720
This intent message was generated by Chrome Platform Status.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SN6PR00MB0397C14B0F073635A96B475ACFA99%40SN6PR00MB0397.namprd00.prod.outlook.com.
Mike Taylor
On Wed, Sep 29, 2021 at 3:02 AM 'Anupam Snigdha' via blink-dev <blin...@chromium.org> wrote:
Interoperability and Compatibility
Gecko: In Development(https://bugzilla.mozilla.org/show_bug.cgi?id=1619947)
It's not clear what this issue actually implements. Did they already implement and ship the Promise based API?If not, it'd be good to ask for official signals: https://bit.ly/blink-signals
https://bugzilla.mozilla.org/show_bug.cgi?id=1712122
was probably the intended bug, but agree a signal would be useful
as it seems the patch from that bug was abandoned a few months
back (https://phabricator.services.mozilla.com/D115768#3889959),
and the bug is currently unowned.
Anupam Snigdha
That's not really an explainer. Can you expand on what this method does, how would the new method look like and how would developers use it?
Sorry for not being clear in my initial I2S. The change is fairly trivial from a developer’s perspective and has no compat risks so I wasn’t sure an i2S was required at all. This is one of the reasons I didn’t prepare an Explainer and other documentations as the changes have no impact to existing users of async clipboard API.
The ClipboardItem has a record of string that represents a MIME type and ClipboardItemData that represents Promises to Blobs corresponding to the MIME types.
The current implementation of Clipboarditem’s constructor in Chromium takes Blobs and not Promises to Blobs which is not how it’s defined in the specification. ClipboardItem is only used in async clipboard API for reading/writing data to the clipboard. This API has been standardized and implemented in Chromium & Safari. This doesn’t affect existing sites that use DataTransfer items for copy/paste scenarios.
Definition of existing ClipboardItem constructor in Chromium:
interface ClipboardItem {
[RaisesException] constructor(record<DOMString, Blob> items,
optional ClipboardItemOptions options = {});
readonly attribute FrozenArray<DOMString> types;
[
CallWith=ScriptState
] Promise<Blob> getType(DOMString type);
};
Proposed definition of ClipboardItem constructor:
interface ClipboardItem {
[RaisesException] constructor(record<DOMString, Promise<Blob>> items,
optional ClipboardItemOptions options = {});
readonly attribute FrozenArray<DOMString> types;
[
CallWith=ScriptState
] Promise<Blob> getType(DOMString type);
};
Currently in Chromium, developers call the async write using the Blob type as shown below:
const html_text = new Blob(
['<html><body><div>hello</div></body></html>'], {type: 'text/html'});
const clipboard_item = new ClipboardItem({
'text/html': html_text
});
navigator.clipboard.write([clipboard_item]);
With the proposed implementation, they can pass promises to Blobs as shown below:
navigator.clipboard.write([
new ClipboardItem({
"text/html": Promise.resolve(new Blob(['<p style=\'color: red; font-style: oblique;\'>This text was copied using </p>'], {type: 'text/html'})),}),]);
Was this change discussed in a standards venue?
This is an existing spec and have been approved by the Editing WG.
Is there compat risk here? Are developers already using the non-Promise method? What is it returning today?
What should adoption patterns to avoid risks in non-supporting browsers?
No compat risks as shown in the above examples. Developers can keep using the Blobs to ClipboardItem constructor. The promises would be resolved implicitly with my change.
Non supporting browsers are not affected by this change as the ClipboardItem object can only be used in async clipboard APIs.
If not, it'd be good to ask for official signals: https://bit.ly/blink-signals
I’m not sure if it requires any signals as the async clipboard API is already in development. Browsers have to implement async clipboard APIs first before making any changes to Clipboarditem object. When they do implement this API, they should ideally follow the spec and implement a promise based ClipboardItem and not what is currently implemented in Chromium.
Yoav Weiss
Thomas Steiner
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfX1OBfHzKxfC49qXbw67NiFx6JkO-Af1oEM_%2B7E9CmUKA%40mail.gmail.com.
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.23 (GNU/Linux)
iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom.hTtPs://xKcd.cOm/1181/
-----END PGP SIGNATURE-----
Anne van Kesteren
<blin...@chromium.org> wrote:
> The ClipboardItem [...] This API has been standardized and implemented in Chromium & Safari.
I wouldn't call it standardized. Yes, the specification contains some
IDL, but nowhere is it defined what the associated behavior is. See
https://github.com/w3c/clipboard-apis/issues/135 for instance.
Anupam Snigdha
Yes, your understanding about interop fix and compat risk is correct. But I would just add one thing -- In the future, browsers shouldn’t implement it without promises to Blobs and deviate from the spec. See Thomas’s comment on this bug where the authors have to do this ugly workaround – The user activation issue is different from promises to Blobs during clipboard.write.
Are there specific WPT tests that cover the desired behavior?
I couldn’t find any, but I’m planning to add some tests in crrev.com/c/3169593.
Yoav Weiss
Anne van Kesteren
> Thanks Anne and Thomas for the cross-browser context.
>
> Anupam - looking at the issue Anne posted, it seems Firefox explicitly did not implement this.
> I think it'd be interesting to get their opinions as to why, and whether we should align with the current WebKit behavior or the current Chromium one.
>
> Anne - do y'all want to chime in here, or would a standards position issue be the best venue?
https://github.com/mozilla/standards-positions/issues/89. The problem
we are having in evaluating all this is that there isn't really any
specification to speak of, as I tried to point out. One can take
guesses as to what the expected behavior is likely to be, but that is
very far from ideal and normally not even close to acceptable, right?
Yoav Weiss
Anupam Snigdha
For #1: I don’t think we would want to diverge from the spec. There is a reason why we have promises to Blobs and not just Blobs in the ClipboardItem because, well, not having promises defeats the purpose of having an async API. Also waiting for the Blob data synchronously without triggering the clipboard write operation leads to problems like this and performance issues in sites like Excel Online where the copy payload is in MBs.
For #2: This might sound more of a rant so apologies in advance.
I agree with Anne that the spec is not really clear at all on the specifics of the async clipboard API and some of the terminologies used in the algorithms.
However, making changes to the spec or even clarifying the language after an API has been shipped, is really hard, as we need to get consensus from all browser vendors.
I tried to clarify what “sanitization” means just for the HTML format and Apple opposed to this change. Perhaps I can add a non-normative note which would at least give some clarity on the sanitization process, but that would probably require UA specific non normative notes which defeats the purpose of standardization.
I also tried to make changes to address Mozilla’s concern about mandatory data types supported by async clipboard APIs: https://github.com/w3c/clipboard-apis/pull/155, but this PR has been sitting for almost a month now and I’m not able to make any progress.
In order to make progress on spec changes, we decided to have a discussion with the Editing WG members and submit changes to the spec if no one objects to it. Currently the Editing WG has representatives from Apple and MS who regularly attend the monthly status meetings. So my question is, if we get an approval from the WG, then does that meet the minimum bar to make spec changes?
Anyways, I’m working on updating the spec to at least define the Clipboard interface IDL, but since Apple and Chromium browsers have already shipped this API, I don’t think it’s possible to make any significant changes to the APIs without breaking at least one of the browsers.
This change addresses the discrepancy between the ClipboardItem IDL as defined in the spec(also implemented by Apple) and what is implemented in Chromium. This is not a breaking change so I think the risk is minimal here.
From: Yoav Weiss <yoav...@chromium.org>
Sent: Friday, October 1, 2021 4:15 AM
To: Anne van Kesteren <ann...@annevk.nl>
Cc: Anupam Snigdha <sni...@microsoft.com>; blink-dev <blin...@chromium.org>; Marijn Kruisselbrink <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>
Subject: Re: [EXTERNAL] Re: [blink-dev] Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item
Anupam Snigdha
Here is a WIP PR to address the spec issue: https://github.com/w3c/clipboard-apis/pull/158.
Yoav Weiss
Here is a WIP PR to address the spec issue: https://github.com/w3c/clipboard-apis/pull/158.
From: Anupam Snigdha
Sent: Monday, October 4, 2021 10:45 AM
To: 'Yoav Weiss' <yoav...@chromium.org>; Anne van Kesteren <ann...@annevk.nl>
Cc: blink-dev <blin...@chromium.org>; Marijn Kruisselbrink <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>
Subject: RE: [EXTERNAL] Re: [blink-dev] Intent to Implement and Ship: Add support for Promise to Blobs in clipboard item
For #1: I don’t think we would want to diverge from the spec. There is a reason why we have promises to Blobs and not just Blobs in the ClipboardItem because, well, not having promises defeats the purpose of having an async API. Also waiting for the Blob data synchronously without triggering the clipboard write operation leads to problems like this and performance issues in sites like Excel Online where the copy payload is in MBs.
For #2: This might sound more of a rant so apologies in advance.
I agree with Anne that the spec is not really clear at all on the specifics of the async clipboard API and some of the terminologies used in the algorithms.
However, making changes to the spec or even clarifying the language after an API has been shipped, is really hard, as we need to get consensus from all browser vendors.
I tried to clarify what “sanitization” means just for the HTML format and Apple opposed to this change.
Perhaps I can add a non-normative note which would at least give some clarity on the sanitization process, but that would probably require UA specific non normative notes which defeats the purpose of standardization.
I also tried to make changes to address Mozilla’s concern about mandatory data types supported by async clipboard APIs: https://github.com/w3c/clipboard-apis/pull/155, but this PR has been sitting for almost a month now and I’m not able to make any progress.
In order to make progress on spec changes, we decided to have a discussion with the Editing WG members and submit changes to the spec if no one objects to it. Currently the Editing WG has representatives from Apple and MS who regularly attend the monthly status meetings. So my question is, if we get an approval from the WG, then does that meet the minimum bar to make spec changes?
Anupam Snigdha
Yep, I’ll address the feedback from Anne and mbrodesser (from Mozilla).
Thanks for all your help Anne and Yoav!
From: Yoav Weiss <yoav...@chromium.org>
Sent: Thursday, October 7, 2021 12:03 AM
To: Anupam Snigdha <sni...@microsoft.com>
Anupam Snigdha
Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers), and Apple has already shipped this feature. Please let me know in case of any concerns.
-Anupam
Yoav Weiss
Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers)
Ashley Gullen
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7a237c30-9d53-4181-9c5d-1954d2bf6a0cn%40chromium.org.
Domenic Denicola
LGTM1 to ship conditional that y'all continue to work on PR #158 specifically, and clarifying the spec's processing model in general.On Thursday, October 21, 2021 at 2:04:53 AM UTC+2 snianu wrote:Gentle ping as the branch cutoff date for 97 is pretty close. While I agree that the issues related to clipboard API spec need to be addressed, I don’t think this feature needs to be blocked on that. It’s not a breaking change i.e. sites can continue to use Blobs if they want to(although I don’t think any developer would want to have different codepaths for Apple and Chromium browsers)
FWIW, I got curious RE why that *should* work, and did some digging.It seems like the bindings methods that accept a `Promise<T>` input value call `NativeValueTraits<IDLPromise>` on that value, which casts the value foo into a `Promise.resolve(foo)`, if it wasn't a Promise already.The same seems to work in WebKit as well. Do you know if this bindings behavior is specified?
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7a237c30-9d53-4181-9c5d-1954d2bf6a0cn%40chromium.org.
Anupam Snigdha
LGTM1 to ship conditional that y'all continue to work on PR #158 specifically, and clarifying the spec's processing model in general
Yes, we’re committed to work on improving the clipboard API spec. That is sort of a prerequisite for Pickling API 😊
Also, can you add tests for both input cases as part of your CLs for this?
Yep, already added: https://chromium-review.googlesource.com/c/chromium/src/+/3169593
@Ashley Gullen user activation is orthogonal to this change. The intent of this change is to align the Clipboarditem API with the spec and Safari. This is needed to support promises to Blobs so web authors don’t have to create Blobs synchronously. This may not matter much to sites that have small payloads, but sites like Excel Online where the workbook could contain charts, images etc wouldn’t be able to synchronously create Blobs just in time to satisfy the user activation requirement of async clipboard write method.
Daniel Bratell
Inspired by the recent talk about user interaction, I feel like
there is one thing I want to understand.
So with a Promise you move the execution to a later time. Is it possible here for a malicious page to delay an action to much, much later and then do that clipboard operation on data that was not available at the time of the clipboard operation the user initiated?
If so, could that have security implications?
Could there even be more than one ongoing clipboard operation at
a time?
/Daniel
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra-e-Y3%3DSn_LQ7qCLKahrg8WaOfoi4LR1TGMN4%3D5-Dn7kQ%40mail.gmail.com.
Ashley Gullen
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/4c7d0610-b7de-b0d0-91d1-4943f1e4a6c2%40gmail.com.
Ashley Gullen
Anupam Snigdha
Thanks Daniel for raising this concern. I have opened an issue to discuss with the Editing WG: https://github.com/w3c/clipboard-apis/issues/161. In the current implementation, I think verifying whether the Document is active or not after the promises have been resolved should mitigate some of the concerns.
Mike West
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SN6PR00MB03977ABE1FD1C66C98131F60CF859%40SN6PR00MB0397.namprd00.prod.outlook.com.
Anupam Snigdha
Sent: Thursday, October 28, 2021 12:17 PM
To: Anupam Snigdha <sni...@microsoft.com>
Cc: brat...@gmail.com <brat...@gmail.com>; dom...@chromium.org <dom...@chromium.org>; yoav...@chromium.org <yoav...@chromium.org>; blin...@chromium.org <blin...@chromium.org>; ann...@annevk.nl <ann...@annevk.nl>; m...@chromium.org <m...@chromium.org>; Bo Cupp <pc...@microsoft.com>
Mike West
Let me try to clarify few things. Currently there is no user activation requirements in async clipboard read/write in Chromium browsers. e.g. https://clumsy-garnet-meeting.glitch.me/. Just load this page and refresh it without clicking anywhere on the page, you will see the content will get written to the clipboard without any gesture.
Safari, due to security concerns, implemented a gesture requirement to access clipboard via async clipboard APIs. The read()/write() method can only be called inside a trusted user gesture event handler, but the promises to Blobs can be resolved later which gives the web authors the flexibility to not block the thread to populate the payload.
In Pickling API, which adds capability to the async clipboard API to read/write unsanitized content, initially we added a transient user activation requirement because the API lets web authors read/write unsanitized content using a custom clipboard format. Both Chrome security team(see "User Gesture Requirement" section in https://github.com/w3c/editing/issues/315) and TAG(https://github.com/w3ctag/design-reviews/issues/636#issuecomment-857829725) raised the concern that transient user activation is NOT sufficient to give clipboard access to web authors.
Anupam Snigdha
Sorry for the late response (I was out on vacation).
Writing to the clipboard is quite distinct from reading from the clipboard, isn't it? https://gifted-stingy-ketchup.glitch.me/ actually surfaces a permission prompt when executing `navigator.clipboard.read()`, which is more in line with my expectations.
Yes, it is, but this feature really affects writing to the clipboard as the promises to the Blobs are provided by the web authors. Thus, browser is not (at least for now) in control of the timing of when the promises are resolved. For reading, we are in complete control of the promises. In Chromium we read all the supported clipboard formats during the read() call and after user grants permission to access the clipboard, and resolve the promises with the Blob data. UAs could choose to not do this and may read the payload and resolve the Blob data when it’s actually queried via getType method.
The question in my mind relates to the content of that promise's resolution. Is the resulting blob created at the point at which the `read()` method is called (as a "copy of the system clipboard data")? Or is the resolution "active" in some sense, returning a blob representing whatever happens to be in the clipboard at the time the promise is resolved?
At least in chromium, we read the payload from the clipboard corresponding to the formats during the read() method call (asynchronously so we don’t block the UI thread): https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/clipboard/clipboard_promise.cc;drc=8cc4402c875055fe7989c15f7dfdd04a46c753fc;l=256
But like I said, write() affects how web authors process the ClipboardItem object which in turn affects when the content is written to the clipboard. I don’t think there is any issue with the read() API, and this proposal doesn’t affect how Chromium reads formats from the clipboard and when the promises to the Blobs are resolved.
does a click give you access to the clipboard _now_, or at some arbitrary point in the future?
Yes, click gives you access to the clipboard, but promises to the blobs don’t have to be resolved within the event handler.
e.g.
Here instead of resolving the promises, you could resolve it later, but the call to the write() method needs to happen inside the click event handler.
<button id="copy-html">Copy text and markup</button>
<div>Then paste in the box below:</div>
<div contenteditable spellcheck="false" style="width: 200px; height: 100px; overflow: hidden; border: 1px solid black;"></div>
<script>
document.getElementById("copy-html").addEventListener("click", event => {
navigator.clipboard.write([
new ClipboardItem({
"text/plain": Promise.resolve("This text was copied using `Clipboard.prototype.write`."),
"text/html": Promise.resolve("<p style='color: red; font-style: oblique;'>This text was copied using <code>Clipboard.prototype.write</code>.</p>"),
}),
]);
});
</script>
