Intent to Ship: Document picture-in-picture

[Tommy Steimel]

Contact emails

ste...@chromium.org, libe...@chromium.org, y...@chromium.org

Explainer

https://github.com/WICG/document-picture-in-picture/blob/main/README.md

Specification

https://wicg.github.io/document-picture-in-picture

Summary

Document PiP adds a new API to open an always-on-top window that can be populated with arbitrary HTMLElements. This is an expansion upon the existing HTMLVideoElement API that only allows for an HTMLVideoElement to be put into a PiP window. This allows web developers to provide a better PiP experience to users.

Blink component

Blink>Media>PictureInPicture

TAG review

https://github.com/w3ctag/design-reviews/issues/798

TAG review status

Issues open

Risks

Interoperability and Compatibility

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/670)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/41) no official position, but have some concerns

Web developers: Positive (https://discourse.wicg.io/t/proposal-document-picture-in-picture/5736/2?u=steimel) In addition to the linked comment, we have some signals from partners that this would be valuable, and are checking to see if they are OK making that public

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

N/A since we are not enabling this API on Android

Debuggability

The new methods and properties proposed in this spec will show up in autocomplete functionality (e.g. window.documentPictureInPicture). The enter event will support event listener breakpoints in the "Picture-in-Picture" category.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

At least initially, we're not supporting Android since the Android APIs don't easily support always-on-top windows with arbitrary interaction

Is this feature fully tested by web-platform-tests?

Yes

Flag name

blink::features::kDocumentPictureInPictureAPI

Requires code in //chrome?

True

Tracking bug

https://crbug.com/1315351

Launch bug

https://crbug.com/1269059

Measurement

We have UseCounters for the documentPictureInPicture APIs

Availability expectation

Feature is available only in Chromium browsers for the foreseeable future

Adoption expectation

Feature is used by specific partner(s) to provide functionality within 12 months of launch in Chrome

Adoption plan

Feature is already being used in the OT by partners who plan to continue using the feature in stable

Non-OSS dependencies

Does the feature depend on any code or APIs outside the Chromium open source repository and its open-source dependencies to function?

N/A

Estimated milestones

Shipping on desktop	116
OriginTrial desktop last	115
OriginTrial desktop first	111

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

No future backwards-incompatible spec changes anticipated

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5755179560337408

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGyVZ8L-SmBFbBMvvbm0x3TwZ66JQ1Fm7_zb3nSiBvYhXavAuA%40mail.gmail.com Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/Tz1gUh92dXs

This intent message was generated by Chrome Platform Status.

[Domenic Denicola]

I was the spec mentor for Tommy's work on this feature, and am chiming in to provide the requested spec maturity summary.

I'm very happy with the spec work for this feature. Tommy worked hard to incorporate all my feedback about how to ensure everything was rigorously defined, and integrated well with existing concepts like opening new windows (navigables). The monkey patches are small, focused, and easy to understand.

There is some TAG feedback that seems to wish this was part of window.open() or some other more-general API, but I think that advice is not correct, and the current API design is good, due to the singleton-per-top-level-traversable nature of a document PiP window. In my opinion this makes the window.documentPictureInPicture entrypoint, with its requestWindow(), window, and onenter properties, a good API for the use case.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE-AwAr2EHEEbqD0X-7bt_4NGAxFhyRz_0wv2c6Bvi65iYw30Q%40mail.gmail.com.

[Philip Jägenstedt]

Thanks Domenic for mentoring and vouching :) I took a very cursory look at the API shape and sent a small fix, thanks for reviewing that.

The concerns on https://github.com/WebKit/standards-positions/issues/41 are device independence and portability. For the Android support, can you say anything about future expectations? Is this feature simply not important on mobile and it might be a desktop-only API for the foreseeable future, or would Android support make sense for some use cases? If so, does it seem tractable given the requisite changes to Android APIs?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra9xZXrXi9f8upH7EZdaF2mOgn%3DVz5ZnDR%2B3xdEaHiG8LQ%40mail.gmail.com.

[Tommy Steimel]

In the future, we may be able to do something on Android that doesn't support input without any change to Android APIs. If we wanted input in the PiP window, it would require changes to Android itself. That said, we haven't seen any interest from web developers for an inputless document PiP on Android, and there isn't really anything you can do with an inputless document PiP that you can't do with a canvas-back video PiP, so we haven't pursued anything on Android.

[Yoav Weiss]

On Tue, Jun 13, 2023 at 7:25 PM 'Tommy Steimel' via blink-dev <blin...@chromium.org> wrote:

In the future, we may be able to do something on Android that doesn't support input without any change to Android APIs. If we wanted input in the PiP window, it would require changes to Android itself. That said, we haven't seen any interest from web developers for an inputless document PiP on Android, and there isn't really anything you can do with an inputless document PiP that you can't do with a canvas-back video PiP, so we haven't pursued anything on Android.

On Tue, Jun 13, 2023 at 7:06 AM Philip Jägenstedt <foo...@chromium.org> wrote:

Thanks Domenic for mentoring and vouching :) I took a very cursory look at the API shape and sent a small fix, thanks for reviewing that.

The concerns on https://github.com/WebKit/standards-positions/issues/41 are device independence and portability. For the Android support, can you say anything about future expectations? Is this feature simply not important on mobile and it might be a desktop-only API for the foreseeable future, or would Android support make sense for some use cases? If so, does it seem tractable given the requisite changes to Android APIs?

On Tue, Jun 13, 2023 at 4:16 AM Domenic Denicola <dom...@chromium.org> wrote:

I was the spec mentor for Tommy's work on this feature, and am chiming in to provide the requested spec maturity summary.

I'm very happy with the spec work for this feature. Tommy worked hard to incorporate all my feedback about how to ensure everything was rigorously defined, and integrated well with existing concepts like opening new windows (navigables). The monkey patches are small, focused, and easy to understand.

There is some TAG feedback that seems to wish this was part of window.open() or some other more-general API, but I think that advice is not correct, and the current API design is good, due to the singleton-per-top-level-traversable nature of a document PiP window. In my opinion this makes the window.documentPictureInPicture entrypoint, with its requestWindow(), window, and onenter properties, a good API for the use case.

On Tue, Jun 13, 2023 at 1:51 AM 'Tommy Steimel' via blink-dev <blin...@chromium.org> wrote:

Contact emails

ste...@chromium.org, libe...@chromium.org, y...@chromium.org

Explainer

https://github.com/WICG/document-picture-in-picture/blob/main/README.md

The examples use the unload event for the PiP document window where, where elsewhere we're working on getting rid of it..

Is there an alternative API we can recommend instead?

 

Specification

https://wicg.github.io/document-picture-in-picture

Summary

Document PiP adds a new API to open an always-on-top window that can be populated with arbitrary HTMLElements. This is an expansion upon the existing HTMLVideoElement API that only allows for an HTMLVideoElement to be put into a PiP window. This allows web developers to provide a better PiP experience to users.

Blink component

Blink>Media>PictureInPicture

TAG review

https://github.com/w3ctag/design-reviews/issues/798

TAG review status

Issues open

Risks

Interoperability and Compatibility

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/670)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/41) no official position, but have some concerns

Web developers: Positive (https://discourse.wicg.io/t/proposal-document-picture-in-picture/5736/2?u=steimel) In addition to the linked comment, we have some signals from partners that this would be valuable, and are checking to see if they are OK making that public

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

N/A since we are not enabling this API on Android

Debuggability

The new methods and properties proposed in this spec will show up in autocomplete functionality (e.g. window.documentPictureInPicture). The enter event will support event listener breakpoints in the "Picture-in-Picture" category.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

At least initially, we're not supporting Android since the Android APIs don't easily support always-on-top windows with arbitrary interaction

Do I understand correctly that the Android apps will not have a PiP experience advantage over the web, since the other PiP APIs enable similar (input-less) experiences on the mobile web compared to mobile apps?

  

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAE-AwAr6O6E2Ds1yw%3DDPyMGDRoLOqyRzVzA6%2BoyOoAQ3Tw_SKg%40mail.gmail.com.

[François Beaufort]

On Wed, Jun 14, 2023 at 8:17 AM Yoav Weiss <yoav...@chromium.org> wrote:

On Tue, Jun 13, 2023 at 7:25 PM 'Tommy Steimel' via blink-dev <blin...@chromium.org> wrote:

In the future, we may be able to do something on Android that doesn't support input without any change to Android APIs. If we wanted input in the PiP window, it would require changes to Android itself. That said, we haven't seen any interest from web developers for an inputless document PiP on Android, and there isn't really anything you can do with an inputless document PiP that you can't do with a canvas-back video PiP, so we haven't pursued anything on Android.

On Tue, Jun 13, 2023 at 7:06 AM Philip Jägenstedt <foo...@chromium.org> wrote:

Thanks Domenic for mentoring and vouching :) I took a very cursory look at the API shape and sent a small fix, thanks for reviewing that.

The concerns on https://github.com/WebKit/standards-positions/issues/41 are device independence and portability. For the Android support, can you say anything about future expectations? Is this feature simply not important on mobile and it might be a desktop-only API for the foreseeable future, or would Android support make sense for some use cases? If so, does it seem tractable given the requisite changes to Android APIs?

On Tue, Jun 13, 2023 at 4:16 AM Domenic Denicola <dom...@chromium.org> wrote:

I was the spec mentor for Tommy's work on this feature, and am chiming in to provide the requested spec maturity summary.

I'm very happy with the spec work for this feature. Tommy worked hard to incorporate all my feedback about how to ensure everything was rigorously defined, and integrated well with existing concepts like opening new windows (navigables). The monkey patches are small, focused, and easy to understand.

There is some TAG feedback that seems to wish this was part of window.open() or some other more-general API, but I think that advice is not correct, and the current API design is good, due to the singleton-per-top-level-traversable nature of a document PiP window. In my opinion this makes the window.documentPictureInPicture entrypoint, with its requestWindow(), window, and onenter properties, a good API for the use case.

On Tue, Jun 13, 2023 at 1:51 AM 'Tommy Steimel' via blink-dev <blin...@chromium.org> wrote:

Contact emails

ste...@chromium.org, libe...@chromium.org, y...@chromium.org

Explainer

https://github.com/WICG/document-picture-in-picture/blob/main/README.md

The examples use the unload event for the PiP document window where, where elsewhere we're working on getting rid of it..

Is there an alternative API we can recommend instead?

Good catch! We should indeed recommend using the pagehide event instead.

I'll be updating spec/explainer and documentation.
 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWLB8nFz8S%3DoFEudtHs3AgJT_vmO%2BQJqE%2BS-oCH_Cs3Bg%40mail.gmail.com.

[François Beaufort]

Android apps have the ability today to show different icons in the PiP window than what is currently in https://w3c.github.io/mediasession/#enumdef-mediasessionaction

However, regarding user input, Android apps and Web apps behave the same. Users can only interact with the PiP window icons, provided by the app.

[Yoav Weiss]

LGTM1

Thanks for addressing the `unload` documentation and for your answers! :)

[slightlyoff via Chromestatus]

LGTM2

[Mike Taylor]

LGTM3

On 6/14/23 11:47 AM, slightlyoff via Chromestatus wrote:

LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/000000000000d6062405fe18ddcb%40google.com.

[Mariano M]

Hey, is there any official release date for the new PiP window without having to have the chrome flag enabled? Thanks!

[François Beaufort]

This API shipped in Chrome 116. See https://developer.chrome.com/docs/web-platform/document-picture-in-picture/

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/91d4591b-052b-42e9-8925-06a0d211e869n%40chromium.org.

[Mariano M]

Oh!! You're right, i no longer need the Chrome flag. So awesome! Good job on this feature, love it!

[Kevin Groat]

Love this feature, but it appears to only be available on desktop.  Is there any word on when it might be made available for Android?

[ari picker]

Thank you for all your work on PiP, it's awesome!

I am very interested in PiP on Android. Is any progress being made, and is there an ETA for PiP landing on Android?

[Sahil Talwar]

+1 - would love PiP on Android as well.

[Tommy Steimel]

Thanks for the interest in Document PiP on Android!

Currently, we're not sure that it's feasible to make the pip window interactive on Android (e.g. allowing HTML buttons to be clicked). Would an API that only allows non-interactive pip (so you'd create an HTML document that we'd show but would not receive user inputs) be useful in your opinion? Or would it only be a useful API if the user could actually interact with it?

Thanks,

Tommy

On Wed, May 1, 2024 at 6:11 PM Maratona app <man...@maratona.app> wrote:

+1 for PiP on Android, it works really well for features like Pomodoro Timer and To-do tasks and since PWA is growing... it seems a great feature to support. (Currently using PWA -> TWA)

I know there is an workaround that is to build the feature using a <canvas>, capture stream and pass to video for original PiP support.... but it makes really hard to map all interactions.

Anyway, awesome work !

[Maratona app]

+1 for PiP on Android, it works really well for features like Pomodoro Timer and To-do tasks and since PWA is growing... it seems a great feature to support. (Currently using PWA -> TWA)

I know there is an workaround that is to build the feature using a <canvas>, capture stream and pass to video for original PiP support.... but it makes really hard to map all interactions.

Anyway, awesome work !

On Wednesday, May 1, 2024 at 4:59:37 PM UTC-3 Sahil Talwar wrote:

[Maratona app]

Hi Tommy, thank you for replying.

My first instinct is not really about what to do on Android, but feature parity with Desktop, for example: https://dlitsman.github.io/document-pip/ since it's already possible to make interactive PiP, it seems odd to not have the same experience for mobile, I know the implementations for Android may be a restriction for this, so only sharing my thoughts as an user. Other APIs like Push Notifcation, Web Share and Badging all seem to have feature parity, even if the UI is different for each platform, they are supported somehow. 

It feels unmotivated to adopt a new API that will only work for a specific platform, if there is a security reason behind this, it would be nice to prompt the user to allow the interaction or not, similar to allowing/denying Push Notifcations. At least I really enjoy this new API as a temporary widget for mobile devices, since it's not possible to build native widgets through PWA. I don't wanna be mean, only sharing initial thoughts and open to understand the real blockers.

[Thomas Steiner]

FWIW, the video PiP already allows for interactive controls (see the attached screenshot from a German news web app https://tagesschau.de). It's a small surface, but it seems fine to leave it to app authors to make sure the controls for their app work.

[ari picker]

+1 for feature parity, but honestly, what I need right now is Android no-video PiP content.

[Sahil Talwar]

My 2 cents - rendering a document PiP would be great, but it wouldn't help our use case unless we also have interactivity - the reason we want a document is precisely because we want the user to be able to interact with the PiP. Rendering a document while keeping the video controls would be an acceptable fallback, but wouldn't really add too much value for our use case.