Intent to Implement and Ship: Media Remoting

150 views
Skip to first unread message

Yuri Wiitala

unread,
Nov 4, 2016, 8:49:33 PM11/4/16
to blin...@chromium.org

Contact emails

m...@chromium.org, sko...@chromium.org


Spec

N/A, since this is a browser feature (with possible web-exposure). Design doc here.


Note: This is similar in functionality to the Remote Playback API, but as an optimization within an existing UA, user-controlled feature; rather than site-controlled. Future Remote Playback API intents will describe any new points of integration (if any) with this feature.


Summary

For Chrome Tab Mirroring, one of the most common use cases is to stream video content to a remote device, such as a Chromecast or Android TV. However, by design, mirroring introduces some loss of quality and also consumes a rather large portion of the sending system's CPU, GPU, memory, and networking resources. Video playback occurs locally in the HTMLMediaElement on the sender device (download → decryption → decoding → rendering → compositing), and then the result is re-captured → scaled → re-encoded → transmitted to the receiver device.


Media Remoting is an optimization that activates during Tab Mirroring. It redirects the original content bitstream of a video directly to the receiver device. In doing this, all of the steps listed above are eliminated. This both saves huge amounts of system resources while also eliminating the main causes of quality degradation.


Intervention/Web-Exposure concern: While a video is being remoted, the HTMLMediaElement will not be rendering the video locally. Instead, a still-frame interstitial, based on the poster image and browser-specific graphics, will be shown to indicate the video content is being remoted. There are concerns that this could break the user experience for certain web sites and, as such, that there needs to be a way for users and/or authors to workaround problems. We propose resolving issues with a combination of manual user intervention and/or the Remote Playback API's disableRemotePlayback attribute (on HTMLMediaElement).


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Desktop platforms only, at this point.


Demo link

Implementation in-progress. For now, we have UX mocks.


Interoperability and Compatibility Risk

Low. Such issues should be rare; and resolved by a combination of manual user intervention or a site author's use of the disableRemotePlayback attribute.


This feature intends to provide the most compelling representation of a page's content for a user that has turned on a second screen for viewing its content (i.e., by mirroring a Chrome browser tab). It's also explicitly intended for use only with pages that do not use other "remoting" web platform APIs, such as the Remote Playback API. As such, when a video occupies the vast majority of the viewport, local rendering of the video content is automatically shut-off and diverted to a receiver device (the second screen; e.g., a TV).


One can easily come up with examples of sites where this behavior breaks the user experience. For example, a page author might choose to layer video and other DOM elements together in complex ways to provide a composed, interactive video game. If the video content were to be remoted, this might result in a very broken display on both the local and remote screens. Nevertheless, we believe such breakage will be rare; especially when compared to the majority of use cases (e.g., streaming video sites) where Media Remoting will vastly improve the user experience.


This feature can only activate after the user takes the manual step of starting Chrome Tab Mirroring. Therefore, in terms of interoperability/compatibility issues, we believe it's reasonable to have the user resolve these by either turning off tab mirroring or using a per-domain "remoting disabled" override setting. The latter means that only Tab Mirroring, which is transparent to web content by design, would be used. Where possible (e.g., an external failure), we will provide UI to aid the user in restoring normal operation.


Also, site authors that have identified problems can use the disableRemotePlayback attribute to preempt the need for manual user intervention. Note that using this attribute is consistent with how other browsers disable their video remoting interventions.


Launch tracking bug

https://crbug.com/632399


Entry on the feature dashboard

None, since this is a browser feature.

Chris Harrelson

unread,
Nov 4, 2016, 9:22:06 PM11/4/16
to Yuri Wiitala, blink-dev
Sounds like a really nice feature. A question below.

On Fri, Nov 4, 2016 at 5:49 PM, Yuri Wiitala <m...@chromium.org> wrote:

Contact emails

m...@chromium.org, sko...@chromium.org


Spec

N/A, since this is a browser feature (with possible web-exposure). Design doc here.


Note: This is similar in functionality to the Remote Playback API, but as an optimization within an existing UA, user-controlled feature; rather than site-controlled. Future Remote Playback API intents will describe any new points of integration (if any) with this feature.


Summary

For Chrome Tab Mirroring, one of the most common use cases is to stream video content to a remote device, such as a Chromecast or Android TV. However, by design, mirroring introduces some loss of quality and also consumes a rather large portion of the sending system's CPU, GPU, memory, and networking resources. Video playback occurs locally in the HTMLMediaElement on the sender device (download → decryption → decoding → rendering → compositing), and then the result is re-captured → scaled → re-encoded → transmitted to the receiver device.


Media Remoting is an optimization that activates during Tab Mirroring. It redirects the original content bitstream of a video directly to the receiver device. In doing this, all of the steps listed above are eliminated. This both saves huge amounts of system resources while also eliminating the main causes of quality degradation.


Intervention/Web-Exposure concern: While a video is being remoted, the HTMLMediaElement will not be rendering the video locally. Instead, a still-frame interstitial, based on the poster image and browser-specific graphics, will be shown to indicate the video content is being remoted. There are concerns that this could break the user experience for certain web sites and, as such, that there needs to be a way for users and/or authors to workaround problems. We propose resolving issues with a combination of manual user intervention and/or the Remote Playback API's disableRemotePlayback attribute (on HTMLMediaElement).


Will the site author be able to observe that mirroring is happening?
 

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Desktop platforms only, at this point.


Demo link

Implementation in-progress. For now, we have UX mocks.


Interoperability and Compatibility Risk

Low. Such issues should be rare; and resolved by a combination of manual user intervention or a site author's use of the disableRemotePlayback attribute.


This feature intends to provide the most compelling representation of a page's content for a user that has turned on a second screen for viewing its content (i.e., by mirroring a Chrome browser tab). It's also explicitly intended for use only with pages that do not use other "remoting" web platform APIs, such as the Remote Playback API. As such, when a video occupies the vast majority of the viewport, local rendering of the video content is automatically shut-off and diverted to a receiver device (the second screen; e.g., a TV).


One can easily come up with examples of sites where this behavior breaks the user experience. For example, a page author might choose to layer video and other DOM elements together in complex ways to provide a composed, interactive video game. If the video content were to be remoted, this might result in a very broken display on both the local and remote screens. Nevertheless, we believe such breakage will be rare; especially when compared to the majority of use cases (e.g., streaming video sites) where Media Remoting will vastly improve the user experience.


This feature can only activate after the user takes the manual step of starting Chrome Tab Mirroring. Therefore, in terms of interoperability/compatibility issues, we believe it's reasonable to have the user resolve these by either turning off tab mirroring or using a per-domain "remoting disabled" override setting. The latter means that only Tab Mirroring, which is transparent to web content by design, would be used. Where possible (e.g., an external failure), we will provide UI to aid the user in restoring normal operation.


Also, site authors that have identified problems can use the disableRemotePlayback attribute to preempt the need for manual user intervention. Note that using this attribute is consistent with how other browsers disable their video remoting interventions.


Launch tracking bug

https://crbug.com/632399


Entry on the feature dashboard

None, since this is a browser feature.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Anton Vayvod

unread,
Nov 4, 2016, 9:45:01 PM11/4/16
to Chris Harrelson, Yuri Wiitala, blink-dev
On Fri, Nov 4, 2016 at 9:22 PM Chris Harrelson <chri...@chromium.org> wrote:
Sounds like a really nice feature. A question below.

On Fri, Nov 4, 2016 at 5:49 PM, Yuri Wiitala <m...@chromium.org> wrote:

Contact emails

m...@chromium.org, sko...@chromium.org


Spec

N/A, since this is a browser feature (with possible web-exposure). Design doc here.


Note: This is similar in functionality to the Remote Playback API, but as an optimization within an existing UA, user-controlled feature; rather than site-controlled. Future Remote Playback API intents will describe any new points of integration (if any) with this feature.


Summary

For Chrome Tab Mirroring, one of the most common use cases is to stream video content to a remote device, such as a Chromecast or Android TV. However, by design, mirroring introduces some loss of quality and also consumes a rather large portion of the sending system's CPU, GPU, memory, and networking resources. Video playback occurs locally in the HTMLMediaElement on the sender device (download → decryption → decoding → rendering → compositing), and then the result is re-captured → scaled → re-encoded → transmitted to the receiver device.


Media Remoting is an optimization that activates during Tab Mirroring. It redirects the original content bitstream of a video directly to the receiver device. In doing this, all of the steps listed above are eliminated. This both saves huge amounts of system resources while also eliminating the main causes of quality degradation.


Intervention/Web-Exposure concern: While a video is being remoted, the HTMLMediaElement will not be rendering the video locally. Instead, a still-frame interstitial, based on the poster image and browser-specific graphics, will be shown to indicate the video content is being remoted. There are concerns that this could break the user experience for certain web sites and, as such, that there needs to be a way for users and/or authors to workaround problems. We propose resolving issues with a combination of manual user intervention and/or the Remote Playback API's disableRemotePlayback attribute (on HTMLMediaElement).


Will the site author be able to observe that mirroring is happening?

The rest of the Remote Playback API has not shipped yet (stay tuned!) but we plan to expose media remoting (not tab mirroring) to the web pages via the RemotePlayback.state and the corresponding state change event handlers when it does.

So the short answer is yes.
 
 

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Desktop platforms only, at this point.


Demo link

Implementation in-progress. For now, we have UX mocks.


Interoperability and Compatibility Risk

Low. Such issues should be rare; and resolved by a combination of manual user intervention or a site author's use of the disableRemotePlayback attribute.


This feature intends to provide the most compelling representation of a page's content for a user that has turned on a second screen for viewing its content (i.e., by mirroring a Chrome browser tab). It's also explicitly intended for use only with pages that do not use other "remoting" web platform APIs, such as the Remote Playback API. As such, when a video occupies the vast majority of the viewport, local rendering of the video content is automatically shut-off and diverted to a receiver device (the second screen; e.g., a TV).


One can easily come up with examples of sites where this behavior breaks the user experience. For example, a page author might choose to layer video and other DOM elements together in complex ways to provide a composed, interactive video game. If the video content were to be remoted, this might result in a very broken display on both the local and remote screens. Nevertheless, we believe such breakage will be rare; especially when compared to the majority of use cases (e.g., streaming video sites) where Media Remoting will vastly improve the user experience.


This feature can only activate after the user takes the manual step of starting Chrome Tab Mirroring. Therefore, in terms of interoperability/compatibility issues, we believe it's reasonable to have the user resolve these by either turning off tab mirroring or using a per-domain "remoting disabled" override setting. The latter means that only Tab Mirroring, which is transparent to web content by design, would be used. Where possible (e.g., an external failure), we will provide UI to aid the user in restoring normal operation.


Also, site authors that have identified problems can use the disableRemotePlayback attribute to preempt the need for manual user intervention. Note that using this attribute is consistent with how other browsers disable their video remoting interventions.


Launch tracking bug

https://crbug.com/632399


Entry on the feature dashboard

None, since this is a browser feature.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Chris Harrelson

unread,
Nov 5, 2016, 3:36:20 PM11/5/16
to Anton Vayvod, Yuri Wiitala, blink-dev
On Fri, Nov 4, 2016 at 6:44 PM, Anton Vayvod <ava...@chromium.org> wrote:
On Fri, Nov 4, 2016 at 9:22 PM Chris Harrelson <chri...@chromium.org> wrote:
Sounds like a really nice feature. A question below.

On Fri, Nov 4, 2016 at 5:49 PM, Yuri Wiitala <m...@chromium.org> wrote:

Contact emails

m...@chromium.org, sko...@chromium.org


Spec

N/A, since this is a browser feature (with possible web-exposure). Design doc here.


Note: This is similar in functionality to the Remote Playback API, but as an optimization within an existing UA, user-controlled feature; rather than site-controlled. Future Remote Playback API intents will describe any new points of integration (if any) with this feature.


Summary

For Chrome Tab Mirroring, one of the most common use cases is to stream video content to a remote device, such as a Chromecast or Android TV. However, by design, mirroring introduces some loss of quality and also consumes a rather large portion of the sending system's CPU, GPU, memory, and networking resources. Video playback occurs locally in the HTMLMediaElement on the sender device (download → decryption → decoding → rendering → compositing), and then the result is re-captured → scaled → re-encoded → transmitted to the receiver device.


Media Remoting is an optimization that activates during Tab Mirroring. It redirects the original content bitstream of a video directly to the receiver device. In doing this, all of the steps listed above are eliminated. This both saves huge amounts of system resources while also eliminating the main causes of quality degradation.


Intervention/Web-Exposure concern: While a video is being remoted, the HTMLMediaElement will not be rendering the video locally. Instead, a still-frame interstitial, based on the poster image and browser-specific graphics, will be shown to indicate the video content is being remoted. There are concerns that this could break the user experience for certain web sites and, as such, that there needs to be a way for users and/or authors to workaround problems. We propose resolving issues with a combination of manual user intervention and/or the Remote Playback API's disableRemotePlayback attribute (on HTMLMediaElement).


Will the site author be able to observe that mirroring is happening?

The rest of the Remote Playback API has not shipped yet (stay tuned!) but we plan to expose media remoting (not tab mirroring) to the web pages via the RemotePlayback.state and the corresponding state change event handlers when it does.

So the short answer is yes.

Yes, but only after that API ships, right? When do you expect that would be? Is it feasible to ship it or some of it together with this feature?
 
 
 

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Desktop platforms only, at this point.


Demo link

Implementation in-progress. For now, we have UX mocks.


Interoperability and Compatibility Risk

Low. Such issues should be rare; and resolved by a combination of manual user intervention or a site author's use of the disableRemotePlayback attribute.


This feature intends to provide the most compelling representation of a page's content for a user that has turned on a second screen for viewing its content (i.e., by mirroring a Chrome browser tab). It's also explicitly intended for use only with pages that do not use other "remoting" web platform APIs, such as the Remote Playback API. As such, when a video occupies the vast majority of the viewport, local rendering of the video content is automatically shut-off and diverted to a receiver device (the second screen; e.g., a TV).


One can easily come up with examples of sites where this behavior breaks the user experience. For example, a page author might choose to layer video and other DOM elements together in complex ways to provide a composed, interactive video game. If the video content were to be remoted, this might result in a very broken display on both the local and remote screens. Nevertheless, we believe such breakage will be rare; especially when compared to the majority of use cases (e.g., streaming video sites) where Media Remoting will vastly improve the user experience.


This feature can only activate after the user takes the manual step of starting Chrome Tab Mirroring. Therefore, in terms of interoperability/compatibility issues, we believe it's reasonable to have the user resolve these by either turning off tab mirroring or using a per-domain "remoting disabled" override setting. The latter means that only Tab Mirroring, which is transparent to web content by design, would be used. Where possible (e.g., an external failure), we will provide UI to aid the user in restoring normal operation.


Also, site authors that have identified problems can use the disableRemotePlayback attribute to preempt the need for manual user intervention. Note that using this attribute is consistent with how other browsers disable their video remoting interventions.


Launch tracking bug

https://crbug.com/632399


Entry on the feature dashboard

None, since this is a browser feature.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Mounir Lamouri

unread,
Nov 6, 2016, 9:21:58 AM11/6/16
to Chris Harrelson, Anton Vayvod, Yuri Wiitala, blink-dev
On Sat, 5 Nov 2016, at 19:35, Chris Harrelson wrote:
> On Fri, Nov 4, 2016 at 6:44 PM, Anton Vayvod <ava...@chromium.org>
> wrote:
>
> > On Fri, Nov 4, 2016 at 9:22 PM Chris Harrelson <chri...@chromium.org>
> > wrote:
> >
> > Sounds like a really nice feature. A question below.
> >
> > On Fri, Nov 4, 2016 at 5:49 PM, Yuri Wiitala <m...@chromium.org> wrote:
> >
> > Contact emails
> >
> > m...@chromium.org, sko...@chromium.org
> >
> > Spec
> >
> > N/A, since this is a browser feature (with possible web-exposure). Design
> > doc here
> > <https://docs.google.com/document/d/1GUieKmNNOKN6fZqMqfjXMfA4xZHhzT6Y2e-FrszvdxE/edit?usp=sharing>
> > .
> >
> > Note: This is similar in functionality to the Remote Playback API
> > <https://w3c.github.io/remote-playback/>, but as an optimization within
> > an existing UA, user-controlled feature; rather than site-controlled.
> > Future Remote Playback API intents will describe any new points of
> > integration (if any) with this feature.
> >
> > Summary
> >
> > For Chrome Tab Mirroring
> > <https://support.google.com/chromecast/answer/3228332?hl=en>, one of the
> > most common use cases is to stream video content to a remote device, such
> > as a Chromecast or Android TV. However, by design, mirroring introduces
> > some loss of quality and also consumes a rather large portion of the
> > sending system's CPU, GPU, memory, and networking resources. Video playback
> > occurs locally in the HTMLMediaElement on the sender device (download →
> > decryption → decoding → rendering → compositing), and then the result is
> > re-captured → scaled → re-encoded → transmitted to the receiver device.
> >
> >
> > Media Remoting is an optimization that activates during Tab Mirroring. It
> > redirects the original content bitstream of a video directly to the
> > receiver device. In doing this, all of the steps listed above are
> > eliminated. This both saves huge amounts of system resources while also
> > eliminating the main causes of quality degradation.
> >
> > Intervention/Web-Exposure concern: While a video is being remoted, the
> > HTMLMediaElement will not be rendering the video locally. Instead, a
> > still-frame interstitial, based on the poster image and browser-specific
> > graphics, will be shown to indicate the video content is being remoted.
> > There are concerns that this could break the user experience for certain
> > web sites and, as such, that there needs to be a way for users and/or
> > authors to workaround problems. We propose resolving issues with a
> > combination of manual user intervention and/or the Remote Playback API's
> > disableRemotePlayback
> > <https://w3c.github.io/remote-playback/#dom-htmlmediaelement-disableremoteplayback>
> > attribute (on HTMLMediaElement).
> >
> >
> > Will the site author be able to observe that mirroring is happening?
> >
> >
> > The rest of the Remote Playback API has not shipped yet (stay tuned!) but
> > we plan to expose media remoting (not tab mirroring) to the web pages via
> > the RemotePlayback.state
> > <https://w3c.github.io/remote-playback/#the-state-attribute> and the
> > corresponding state change event handlers
> > <https://w3c.github.io/remote-playback/#event-handlers> when it does.
> >
> > So the short answer is yes.
> >
>
> Yes, but only after that API ships, right? When do you expect that would
> be? Is it feasible to ship it or some of it together with this feature?

The Intent to Ship for the Remote Playback API should be sent soon. It's
most likely it will ship before the Media Remoting project.

-- Mounir

Chris Harrelson

unread,
Nov 7, 2016, 12:31:05 PM11/7/16
to Mounir Lamouri, Anton Vayvod, Yuri Wiitala, blink-dev
LGTM1 then!

Philip Jägenstedt

unread,
Nov 10, 2016, 9:41:22 AM11/10/16
to Chris Harrelson, Mounir Lamouri, Anton Vayvod, Yuri Wiitala, blink-dev
LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Rick Byers

unread,
Nov 10, 2016, 10:27:14 AM11/10/16
to Philip Jägenstedt, Chris Harrelson, Mounir Lamouri, Anton Vayvod, Yuri Wiitala, blink-dev
LGTM3

Thanks for the detailed compat analysis (and specifically ensuring there was a simple opt-out API).  This sounds great!

LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.


Reply all
Reply to author
Forward
0 new messages