Intent to Implement: Picture-in-Picture for arbitrary content
364 views
Skip to first unread message
Becca Hughes
Apr 26, 2019, 5:50:43 PM4/26/19
to blink-dev, Francois Beaufort, Mounir Lamouri
becca...@chromium.org
mlam...@chromium.org
https://github.com/WICG/picture-in-picture/blob/v2/v2_explainer.md
https://docs.google.com/document/d/1zZwiNkLn24SvTMmnXj6AgGK88jZhkaxexUiZr-hOfsU/edit
Full spec not yet available
Waiting on full spec
We are adding support for requesting a Picture-in-Picture window that can contain arbitrary HTML content instead of a video layer. This content can be either interactive or not.
The original Picture-in-Picture API is limited to HTMLVideoElement. It takes the video layer and moves it into a window that follows display rules specific to Picture-in-Picture.
Various partners expressed interest for a more flexible API. The feature requests fell into two categories: customising the look & feel and customising the user experience. The former can technically be done using canvas elements but the latter requires a different model that would require user interaction and therefore no longer be compatible with Android, iOS and macOS APIs.
Firefox: No public signals
Edge: Public support (https://github.com/WICG/picture-in-picture/issues/113#issuecomment-460512662)
Safari: Opposed
Apple expressed very clearly that they would not implement such an API, regardless of the form it takes. However, they were supportive with us taking this direction.
Web developers: Positive
For the interactive use case we will show the origin on hover. Therefore, if there is Chrome UI overlapping the Picture-in-Picture window we may use the CSS env() safe area variables to provide this information to sites.
Developers will need to do some work to make their sites compatible with the new API. We will do some outreach around best practices for developers.
The UI for the interactive use case will show the origin and a border to make it distinct from the rest of the browser.
All platforms except WebView. However, Android will only support non-interactive content.
https://chromestatus.com/features/4844605453369344
Mike West
Apr 27, 2019, 4:51:35 PM4/27/19
to Becca Hughes, Emily Stark, blink-dev, Francois Beaufort, Mounir Lamouri
On Fri, Apr 26, 2019 at 11:50 PM Becca Hughes <becca...@chromium.org> wrote:
becca...@chromium.orgmlam...@chromium.org https://github.com/WICG/picture-in-picture/blob/v2/v2_explainer.md https://docs.google.com/document/d/1zZwiNkLn24SvTMmnXj6AgGK88jZhkaxexUiZr-hOfsU/editFull spec not yet available Waiting on full spec We are adding support for requesting a Picture-in-Picture window that can contain arbitrary HTML content instead of a video layer. This content can be either interactive or not. The original Picture-in-Picture API is limited to HTMLVideoElement. It takes the video layer and moves it into a window that follows display rules specific to Picture-in-Picture. Various partners expressed interest for a more flexible API. The feature requests fell into two categories: customising the look & feel and customising the user experience. The former can technically be done using canvas elements but the latter requires a different model that would require user interaction and therefore no longer be compatible with Android, iOS and macOS APIs.Firefox: No public signals Edge: Public support (https://github.com/WICG/picture-in-picture/issues/113#issuecomment-460512662) Safari: Opposed Apple expressed very clearly that they would not implement such an API, regardless of the form it takes. However, they were supportive with us taking this direction.
That sounds... not great. :) Any chance that communication was public? It would be nice to link to the conversation here.
Web developers: Positive For the interactive use case we will show the origin on hover. Therefore, if there is Chrome UI overlapping the Picture-in-Picture window we may use the CSS env() safe area variables to provide this information to sites. Developers will need to do some work to make their sites compatible with the new API. We will do some outreach around best practices for developers. The UI for the interactive use case will show the origin and a border to make it distinct from the rest of the browser.
Please make sure to chat with folks like +Emily Stark and her team when working through the UX considerations here.
-mike
All platforms except WebView. However, Android will only support non-interactive content. https://chromestatus.com/features/4844605453369344
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFeLsEKt3KBfBSZUo8N3FOTonvdmvKDsQaCgz4Z%2BQPqUHKOtNw%40mail.gmail.com.
Emily Stark
Apr 29, 2019, 12:25:23 AM4/29/19
to Mike West, Becca Hughes, Emily Stark, blink-dev, Francois Beaufort, Mounir Lamouri
On Sat, Apr 27, 2019 at 1:51 PM Mike West <mk...@chromium.org> wrote:
On Fri, Apr 26, 2019 at 11:50 PM Becca Hughes <becca...@chromium.org> wrote:becca...@chromium.orgmlam...@chromium.org https://github.com/WICG/picture-in-picture/blob/v2/v2_explainer.md https://docs.google.com/document/d/1zZwiNkLn24SvTMmnXj6AgGK88jZhkaxexUiZr-hOfsU/editFull spec not yet available Waiting on full spec We are adding support for requesting a Picture-in-Picture window that can contain arbitrary HTML content instead of a video layer. This content can be either interactive or not. The original Picture-in-Picture API is limited to HTMLVideoElement. It takes the video layer and moves it into a window that follows display rules specific to Picture-in-Picture. Various partners expressed interest for a more flexible API. The feature requests fell into two categories: customising the look & feel and customising the user experience. The former can technically be done using canvas elements but the latter requires a different model that would require user interaction and therefore no longer be compatible with Android, iOS and macOS APIs.Firefox: No public signals Edge: Public support (https://github.com/WICG/picture-in-picture/issues/113#issuecomment-460512662) Safari: Opposed Apple expressed very clearly that they would not implement such an API, regardless of the form it takes. However, they were supportive with us taking this direction.That sounds... not great. :) Any chance that communication was public? It would be nice to link to the conversation here.Web developers: Positive For the interactive use case we will show the origin on hover. Therefore, if there is Chrome UI overlapping the Picture-in-Picture window we may use the CSS env() safe area variables to provide this information to sites. Developers will need to do some work to make their sites compatible with the new API. We will do some outreach around best practices for developers. The UI for the interactive use case will show the origin and a border to make it distinct from the rest of the browser.Please make sure to chat with folks like +Emily Stark and her team when working through the UX considerations here.
We've been chatting already, and have enumerated several risks, some of which are already present in the existing video PIP API:
* If the website can entice the user to drag the PIP window over browser UI, the PIP content could spoof browser UI, like fake a URL in the omnibox (covering the real omnibox). This is partially mitigated by having a drop shadow border around the PIP window.
* Allowing arbitrary user interaction with the PIP window could definitely be a spoofing/phishing risk, but it would probably be feasible to limit interactions to clicks to reduce the risk.
* There is also a reverse-clickjacking risk: the PIP window could float over a cross-origin window and entice the user to click, and then disappear at the right moment so that the click passes through to the page underneath. We are not sure how to mitigate this. It would be difficult to pull off, like the first attack in this list (spoofing browser UI), because the attacker would have to get the user to drag the window into the right place, and the window doesn't know its position (just its size).
* There is a general risk of confusion/annoyance about where the content in the PIP window comes from. There should be a way to show the origin and window controls (like to close the window) in some ideally non-spoofable, always-accessible way.
Mounir Lamouri
Apr 29, 2019, 12:57:17 PM4/29/19
to Mike West, Becca Hughes, Emily Stark, blink-dev, Francois Beaufort
On Sat, 27 Apr 2019 at 13:51, Mike West <mk...@chromium.org> wrote:
On Fri, Apr 26, 2019 at 11:50 PM Becca Hughes <becca...@chromium.org> wrote:becca...@chromium.orgmlam...@chromium.org https://github.com/WICG/picture-in-picture/blob/v2/v2_explainer.md https://docs.google.com/document/d/1zZwiNkLn24SvTMmnXj6AgGK88jZhkaxexUiZr-hOfsU/editFull spec not yet available Waiting on full spec We are adding support for requesting a Picture-in-Picture window that can contain arbitrary HTML content instead of a video layer. This content can be either interactive or not. The original Picture-in-Picture API is limited to HTMLVideoElement. It takes the video layer and moves it into a window that follows display rules specific to Picture-in-Picture. Various partners expressed interest for a more flexible API. The feature requests fell into two categories: customising the look & feel and customising the user experience. The former can technically be done using canvas elements but the latter requires a different model that would require user interaction and therefore no longer be compatible with Android, iOS and macOS APIs.Firefox: No public signals Edge: Public support (https://github.com/WICG/picture-in-picture/issues/113#issuecomment-460512662) Safari: Opposed Apple expressed very clearly that they would not implement such an API, regardless of the form it takes. However, they were supportive with us taking this direction.That sounds... not great. :) Any chance that communication was public? It would be nice to link to the conversation here.
To give some background, Apple made a product decision to only allow videos to be in Picture-in-Picture on iOS and macOS. Because of that, they have no plan to implement this API. The concern isn't technical AFAIUI.
-- Mounir
Daniel Bratell
Apr 30, 2019, 10:28:57 AM4/30/19
to Mike West, Emily Stark, Becca Hughes, blink-dev, Francois Beaufort, Mounir Lamouri
On Mon, 29 Apr 2019 06:25:02 +0200, Emily Stark <est...@chromium.org> wrote:
On Sat, Apr 27, 2019 at 1:51 PM Mike West <mk...@chromium.org> wrote:On Fri, Apr 26, 2019 at 11:50 PM Becca Hughes <becca...@chromium.org> wrote:becca...@chromium.orgmlam...@chromium.org https://github.com/WICG/picture-in-picture/blob/v2/v2_explainer.md https://docs.google.com/document/d/1zZwiNkLn24SvTMmnXj6AgGK88jZhkaxexUiZr-hOfsU/editFull spec not yet available Waiting on full spec We are adding support for requesting a Picture-in-Picture window that can contain arbitrary HTML content instead of a video layer. This content can be either interactive or not. The original Picture-in-Picture API is limited to HTMLVideoElement. It takes the video layer and moves it into a window that follows display rules specific to Picture-in-Picture. Various partners expressed interest for a more flexible API. The feature requests fell into two categories: customising the look & feel and customising the user experience. The former can technically be done using canvas elements but the latter requires a different model that would require user interaction and therefore no longer be compatible with Android, iOS and macOS APIs.Firefox: No public signals Edge: Public support (https://github.com/WICG/picture-in-picture/issues/113#issuecomment-460512662) Safari: Opposed Apple expressed very clearly that they would not implement such an API, regardless of the form it takes. However, they were supportive with us taking this direction.That sounds... not great. :) Any chance that communication was public? It would be nice to link to the conversation here.Web developers: Positive For the interactive use case we will show the origin on hover. Therefore, if there is Chrome UI overlapping the Picture-in-Picture window we may use the CSS env() safe area variables to provide this information to sites. Developers will need to do some work to make their sites compatible with the new API. We will do some outreach around best practices for developers. The UI for the interactive use case will show the origin and a border to make it distinct from the rest of the browser.Please make sure to chat with folks like +Emily Stark and her team when working through the UX considerations here.We've been chatting already, and have enumerated several risks, some of which are already present in the existing video PIP API:* If the website can entice the user to drag the PIP window over browser UI, the PIP content could spoof browser UI, like fake a URL in the omnibox (covering the real omnibox). This is partially mitigated by having a drop shadow border around the PIP window.* Allowing arbitrary user interaction with the PIP window could definitely be a spoofing/phishing risk, but it would probably be feasible to limit interactions to clicks to reduce the risk.* There is also a reverse-clickjacking risk: the PIP window could float over a cross-origin window and entice the user to click, and then disappear at the right moment so that the click passes through to the page underneath. We are not sure how to mitigate this. It would be difficult to pull off, like the first attack in this list (spoofing browser UI), because the attacker would have to get the user to drag the window into the right place, and the window doesn't know its position (just its size).* There is a general risk of confusion/annoyance about where the content in the PIP window comes from. There should be a way to show the origin and window controls (like to close the window) in some ideally non-spoofable, always-accessible way.
Some of these would overlap with similar concerns for normal popup windows, especially the ones without any UI.
Is there a difference in the difficulty for a malicous site in opening a popup compared to these?
/Daniel
--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */
Becca Hughes
Apr 30, 2019, 12:09:32 PM4/30/19
to Daniel Bratell, Mike West, Emily Stark, blink-dev, Francois Beaufort, Mounir Lamouri
For popup windows a site can open as many as they wish. For Picture-in-Picture there is only one window at any one time across the whole browser.
The API will also require user activation.
Emily Stark
May 1, 2019, 10:27:12 PM5/1/19
to Becca Hughes, Daniel Bratell, Mike West, Emily Stark, blink-dev, Francois Beaufort, Mounir Lamouri
Popup windows are not always-on-top like a PIP window is. Also I think (?) most browsers including Chrome no longer allow you to open popups without browser UI.
Reply all
Reply to author
Forward
0 new messages