Question regarding page load processing model for media
85 views
Skip to first unread message
Min Qin
Mar 29, 2016, 3:23:01 PM3/29/16
to blin...@chromium.org
Hi, blink-dev experts
According to https://html.spec.whatwg.org/multipage/browsers.html#read-media, the MediaDocument/ImageDocument should append an media or image element to <body>, but it doesn't specify whether there can be any other elements on the page. Or whether we can put the media/image element inside a div element before appending it to <body>. The android team is doing some experiment on adding a download button to the MediaDocument. And in order to position the button and the media element correctly, we need to put it inside a div element. That causes imported/web-platform-tests/html/browsers/browsing-the-web/read-media/pageload-video.html test to fail.
So I am wondering whether it is allowed for us to put a media element inside a div element before appending it to the body.
Thanks
Min
Elliott Sprehn
Mar 29, 2016, 3:33:08 PM3/29/16
to Qin Min, dom...@chromium.org, blink-dev
+Domenic
I've always wondered how important the shape of the image, plain text, video, audio, and plugin embedded documents was for web compat.
In theory they're real API that is web exposed. In practice I think browsers put random not always the same stuff down there.
If we want to keep the current shape we could instead use a UA shadow root on the body and put the UA wrappers and content there. That might be the easiest route.
Boris Zbarsky
Mar 29, 2016, 4:03:09 PM3/29/16
to Min Qin, blin...@chromium.org
On 3/29/16 3:22 PM, Min Qin wrote:
> According to
> https://html.spec.whatwg.org/multipage/browsers.html#read-media, the
> MediaDocument/ImageDocument should append an media or image element to
> <body>, but it doesn't specify whether there can be any other elements
> on the page. Or whether we can put the media/image element inside a div
> element before appending it to <body>.
For what it's worth, it does specify those: you can't do them. The spec
> According to
> https://html.spec.whatwg.org/multipage/browsers.html#read-media, the
> MediaDocument/ImageDocument should append an media or image element to
> <body>, but it doesn't specify whether there can be any other elements
> on the page. Or whether we can put the media/image element inside a div
> element before appending it to <body>.
says the steps to follow. There's no reading between the lines in this
spec, so absent any other statements any implementation that does not
produce output that matches following those steps is not conforming.
That said, the spec also explicitly says:
User agents may add content to the head element of the Document,
or attributes to the element host element, e.g. to link to a style
sheet or a binding, to provide a script, to give the document a
title, to make the media autoplay, etc.
which means you could in fact inject styles or scripts that add shadow
trees, as far as I can tell.
There's also the "change the spec" option, of course.
-Boris
Janice Hourly
Mar 29, 2016, 8:48:27 PM3/29/16
to Boris Zbarsky, Min Qin, blin...@chromium.org, bigtop...@gmail.com
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
Elliott Sprehn
Mar 29, 2016, 10:48:30 PM3/29/16
to Boris Zbarsky, blink-dev, Qin Min
Yeah I'm aware the spec says what to do. That said I don't think browsers agree on all the various UA generated documents today... But if we all agree on what to do for a video maybe we should explore UA shadow roots.
I'll have to think about the implications of UA shadows. If an author then adds their own shadow root to the <body> the download button would disappear. I think shadow dom v1 was also careful to say author shadows don't work on things with UA shadows, but the body isn't one of those things. If we instead turduken the video and stick the button inside its UA shadow with a nested <video> the size of the video element on the outside will include the button incorrectly.
I'm also curious how often authors even embed videos in frames or window.open them such that they'd observe this. Why wouldn't they use video directly? I guess that's something to UseCount.
Does Firefox have any UI controls for media documents that are not inside the video itself as an overlay?
Boris Zbarsky
Mar 29, 2016, 11:08:15 PM3/29/16
to Elliott Sprehn, blink-dev, Qin Min
On 3/29/16 10:48 PM, Elliott Sprehn wrote:
> Yeah I'm aware the spec says what to do. That said I don't think
> browsers agree on all the various UA generated documents today...
That's entirely possible; would need testing. I also don't have a good
> Yeah I'm aware the spec says what to do. That said I don't think
> browsers agree on all the various UA generated documents today...
feeling for what the real web compat constraints here are...
> I'm also curious how often authors even embed videos in frames
text documents, etc actually opened in a way where someone is observing
the DOM? The usual way I run into them is following some link that
points directly to an image or video, not because some page explicitly
wanted to present that as UI.
> Does Firefox have any UI controls for media documents that are not
> inside the video itself as an overlay?
that propagates most keypresses with the viewport, but not the video,
focused to the video element. The idea being that you don't have to
focus the video to hit the space bar to pause/unpause, for example.
But I think that's it.
-Boris
Anne van Kesteren
Mar 30, 2016, 1:53:23 AM3/30/16
to Elliott Sprehn, Boris Zbarsky, blink-dev, Qin Min
On Wed, Mar 30, 2016 at 4:48 AM, Elliott Sprehn <esp...@chromium.org> wrote:
> I'll have to think about the implications of UA shadows. If an author then
> adds their own shadow root to the <body> the download button would
> disappear. I think shadow dom v1 was also careful to say author shadows
> don't work on things with UA shadows, but the body isn't one of those
> things.
Indeed, attaching a user-agent shadow root to <body> would be bad. Per
> I'll have to think about the implications of UA shadows. If an author then
> adds their own shadow root to the <body> the download button would
> disappear. I think shadow dom v1 was also careful to say author shadows
> don't work on things with UA shadows, but the body isn't one of those
> things.
https://dom.spec.whatwg.org/#dom-element-attachshadow <html> is
available though.
--
https://annevankesteren.nl/
Min Qin
Mar 30, 2016, 12:37:29 PM3/30/16
to Anne van Kesteren, Elliott Sprehn, Boris Zbarsky, blink-dev
Some browsers don't support MediaDocument today. For example, UC browser on Android and earlier versions of safari on ios will intercept navigation request to a video url, and launch the system mediaplayer.
So I am wondering whether we should allow different UAs to have different controls on the MediaDocument
Domenic Denicola
Apr 1, 2016, 5:57:42 PM4/1/16
to Elliott Sprehn, Qin Min, dom...@chromium.org, blink-dev
Sorry for the delay in responding; I was stuck in standards meetings this week.
On Mar 29, 2016 12:23 PM, "Min Qin" <mailto:qin...@chromium.org> wrote:
> According to https://html.spec.whatwg.org/multipage/browsers.html#read-media, the MediaDocument/ImageDocument should append an media or image element to <body>, but it doesn't specify whether there can be any other elements on the page. Or whether we can put the media/image element inside a div element before appending it to <body>.
>
> The android team is doing some experiment on adding a download button to the MediaDocument. And in order to position the button and the media element correctly, we need to put it inside a div element. That causes imported/web-platform-tests/html/browsers/browsing-the-web/read-media/pageload-video.html test to fail.
>
> So I am wondering whether it is allowed for us to put a media element inside a div element before appending it to the body.
The spec is pretty clear that you should not be doing this; it tells you specifically the steps you're supposed to carry out, and inserting a div is not one of them.
It might be web-compatible to insert new things here, but I think we should stick to the spirit of trying to minimize observable differences which could potentially cause compatibility issues here. For example, you could imagine Chrome starting to do this, and developers starting to access the resulting video via `document.body.firstChild.firstChild.play()`, which would only work in Chrome due to Chrome's wrapper div. Or worse, something like `document.querySelector("button").click()` to try to click the download button.
So I think something non-observably, like using user-agent shadow root, will be a better bet. The invariant you need to preserve is that the resulting DOM tree looks like it is in the spec, namely:
html
- head
- body
- video[src]
The spec does give you freedom to add things to the <head>, or add attributes to the <video>, which combined with UA shadow root is hopefully enough degrees of freedom and won't cause interop problems.
If the restrictions here are too restrictive, then we should consider changing the spec in some way so that all browsers can have a new interoperable structure. However...
> And in order to position the button and the media element correctly, we need to put it inside a div element.
Are you sure that this is necessary? You already have three things you can independently vary the position of with CSS: two containers, <html> and <body>, and the <video> itself. Do you really need a third container layer? Or could you maybe style <body> in the way you were planning to style the <div>?
On Mar 29, 2016 12:23 PM, "Min Qin" <mailto:qin...@chromium.org> wrote:
> According to https://html.spec.whatwg.org/multipage/browsers.html#read-media, the MediaDocument/ImageDocument should append an media or image element to <body>, but it doesn't specify whether there can be any other elements on the page. Or whether we can put the media/image element inside a div element before appending it to <body>.
>
> The android team is doing some experiment on adding a download button to the MediaDocument. And in order to position the button and the media element correctly, we need to put it inside a div element. That causes imported/web-platform-tests/html/browsers/browsing-the-web/read-media/pageload-video.html test to fail.
>
> So I am wondering whether it is allowed for us to put a media element inside a div element before appending it to the body.
It might be web-compatible to insert new things here, but I think we should stick to the spirit of trying to minimize observable differences which could potentially cause compatibility issues here. For example, you could imagine Chrome starting to do this, and developers starting to access the resulting video via `document.body.firstChild.firstChild.play()`, which would only work in Chrome due to Chrome's wrapper div. Or worse, something like `document.querySelector("button").click()` to try to click the download button.
So I think something non-observably, like using user-agent shadow root, will be a better bet. The invariant you need to preserve is that the resulting DOM tree looks like it is in the spec, namely:
html
- head
- body
- video[src]
The spec does give you freedom to add things to the <head>, or add attributes to the <video>, which combined with UA shadow root is hopefully enough degrees of freedom and won't cause interop problems.
If the restrictions here are too restrictive, then we should consider changing the spec in some way so that all browsers can have a new interoperable structure. However...
> And in order to position the button and the media element correctly, we need to put it inside a div element.
Min Qin
Apr 4, 2016, 6:38:53 PM4/4/16
to Domenic Denicola, Elliott Sprehn, dom...@chromium.org, blink-dev
Yes, Domenic, we could style <body> instead of adding a new <div>. However, we still need to add the download button. Does introducing the button against the spec? we will have the DOM tree like this
html
- head
- body
- video[src]
- head
- body
- video[src]
-anchor
If that's not allowed, is everyone ok with UA shadow root?
Domenic Denicola
Apr 4, 2016, 6:42:00 PM4/4/16
to Min Qin, Elliott Sprehn, dom...@chromium.org, blink-dev
From: qin...@google.com [mailto:qin...@google.com] On Behalf Of Min Qin
> Yes, Domenic, we could style <body> instead of adding a new <div>.
Good to hear!
> However, we still need to add the download button. Does introducing the button against the spec? we will have the DOM tree like this
It would still be against the spec, indeed. A UA shadow root solution would be preferable IMO.
If there are reasons why you or others think the UA shadow root solution is not good, we should consider loosening the spec requirements somehow, or maybe explicitly adding a download button to the spec (so that code like `document.querySelector("button").click()` works in all browsers, instead of throwing an error). But UA shadow root seems like a nice way to avoid such complications entirely.
> Yes, Domenic, we could style <body> instead of adding a new <div>.
> However, we still need to add the download button. Does introducing the button against the spec? we will have the DOM tree like this
If there are reasons why you or others think the UA shadow root solution is not good, we should consider loosening the spec requirements somehow, or maybe explicitly adding a download button to the spec (so that code like `document.querySelector("button").click()` works in all browsers, instead of throwing an error). But UA shadow root seems like a nice way to avoid such complications entirely.
Christian Biesinger
Apr 4, 2016, 6:47:56 PM4/4/16
to Domenic Denicola, Min Qin, Elliott Sprehn, dom...@chromium.org, blink-dev
We already put the video controls into a UA shadow root, seems like
the download button could be added there relatively easily...?
-Christian
the download button could be added there relatively easily...?
-Christian
Elliott Sprehn
Apr 4, 2016, 6:49:50 PM4/4/16
to Domenic Denicola, Min Qin, dom...@chromium.org, blink-dev
I'm curious to know why the DOM needs to be the same between all browsers here, but the style of the page doesn't. If we use a UA shadow root or style the body you'll get very different results from getBoundingClientRect(), document.body.clientHeight, etc. The body would also end up styled with totally different UA sheets across browsers (ex. flex box in one, and not with the other, we'd also be removing the margins, but other browsers wouldn't...).
Min Qin
Apr 5, 2016, 4:02:01 PM4/5/16
to Elliott Sprehn, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
What about adding an onclick handler on the page?
When clicking the page, we change the css and append the button, and when clicking again, we hide the button.
The HTML spec mentions that javascript is ok with the MediaDocument, but is this reasonable workaround?
Domenic Denicola
Apr 5, 2016, 4:08:17 PM4/5/16
to Elliott Sprehn, Min Qin, dom...@chromium.org, blink-dev
From: Elliott Sprehn [mailto:esp...@chromium.org]
> I'm curious to know why the DOM needs to be the same between all browsers here, but the style of the page doesn't. If we use a UA shadow root or style the body you'll get very different results from getBoundingClientRect(), document.body.clientHeight, etc. The body would also end up styled with totally different UA sheets across browsers (ex. flex box in one, and not with the other, we'd also be removing the margins, but other browsers wouldn't...).
I agree it's a fuzzy line. Clearly there's a tension here between the desire for compatibility and the desire to allow customization by UAs. In my opinion the idea of preserving the spec's DOM tree (modulo the known extension point of the head element) is a good line to draw, but as I've said, I'm willing to open a spec discussion on budging that line and make sure other implementers are on board.
> I'm curious to know why the DOM needs to be the same between all browsers here, but the style of the page doesn't. If we use a UA shadow root or style the body you'll get very different results from getBoundingClientRect(), document.body.clientHeight, etc. The body would also end up styled with totally different UA sheets across browsers (ex. flex box in one, and not with the other, we'd also be removing the margins, but other browsers wouldn't...).
Domenic Denicola
Apr 5, 2016, 4:09:37 PM4/5/16
to Min Qin, Elliott Sprehn, dom...@chromium.org, blink-dev, Kingston Tam
> What about adding an onclick handler on the page?
> When clicking the page, we change the css and append the button, and when clicking again, we hide the button.
> The HTML spec mentions that javascript is ok with the MediaDocument, but is this reasonable workaround?
Hah, that's an amazing workaround. I guess per that interpretation you could always insert <script>s into the head to mutate the DOM however you want (without even necessarily waiting for click). I think it is indeed legal per the letter of the spec, although not per the spirit.
> When clicking the page, we change the css and append the button, and when clicking again, we hide the button.
> The HTML spec mentions that javascript is ok with the MediaDocument, but is this reasonable workaround?
But, can you explain why this workaround is a better solution than the UA shadow root approach? That still seems like the best route to me, and I've seen nobody object to it so far.
Min Qin
Apr 5, 2016, 4:34:59 PM4/5/16
to Domenic Denicola, Elliott Sprehn, dom...@chromium.org, blink-dev, Kingston Tam
With this work around, we can show and hide the download button when needed and won't impact user's experience on video watching.
And this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned.
And since javascript is allowed, we could even put it in onload(), but we can always leave that as a TODO
Domenic Denicola
Apr 5, 2016, 4:36:27 PM4/5/16
to Min Qin, Elliott Sprehn, dom...@chromium.org, blink-dev, Kingston Tam
> With this work around, we can show and hide the download button when needed and won't impact user's experience on video watching.
> And this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned.
>
> And since javascript is allowed, we could even put it in onload(), but we can always leave that as a TODO
Can you answer my question from earlier?
> And this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned.
>
> And since javascript is allowed, we could even put it in onload(), but we can always leave that as a TODO
Min Qin
Apr 5, 2016, 4:39:41 PM4/5/16
to Domenic Denicola, Elliott Sprehn, dom...@chromium.org, blink-dev, Kingston Tam
On Tue, Apr 5, 2016 at 1:36 PM, Domenic Denicola <d...@domenic.me> wrote:
From: qin...@google.com [mailto:qin...@google.com] On Behalf Of Min Qin
> With this work around, we can show and hide the download button when needed and won't impact user's experience on video watching.
> And this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned.
>
> And since javascript is allowed, we could even put it in onload(), but we can always leave that as a TODO
Can you answer my question from earlier?
Do you mean this question: " can you explain why this workaround is a better solution than the UA shadow root approach?"
As I mentioned , this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned, and it allows us to show/hide the button when needed.
Domenic Denicola
Apr 5, 2016, 4:42:55 PM4/5/16
to Min Qin, Elliott Sprehn, dom...@chromium.org, blink-dev, Kingston Tam
> As I mentioned , this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned, and it allows us to show/hide the button when needed.
Well, it will, just only when the button is shown. But I guess I understand what you're saying. Thanks; that seems reasonable.
Elliott Sprehn
Apr 5, 2016, 4:43:48 PM4/5/16
to Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
On Tue, Apr 5, 2016 at 1:39 PM, Min Qin <qin...@chromium.org> wrote:
On Tue, Apr 5, 2016 at 1:36 PM, Domenic Denicola <d...@domenic.me> wrote:From: qin...@google.com [mailto:qin...@google.com] On Behalf Of Min Qin
> With this work around, we can show and hide the download button when needed and won't impact user's experience on video watching.
> And this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned.
>
> And since javascript is allowed, we could even put it in onload(), but we can always leave that as a TODO
Can you answer my question from earlier?Do you mean this question: " can you explain why this workaround is a better solution than the UA shadow root approach?"As I mentioned , this won't impact the getBoundingClientRect() or document.body.clientHeight as Elliot mentioned, and it allows us to show/hide the button when needed.
I don't understand how this works, what is the user clicking on that makes the download button appear? While the button is visible the DOM doesn't match the spec either.
- E
Min Qin
Apr 5, 2016, 4:51:13 PM4/5/16
to Elliott Sprehn, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
It would be clicking on the body. Or if we use onload(), then it doesn't matter.
The spec only mentions everything should happen before page loads, the last step it mentioned is "After creating the
Document object, but potentially before the page has finished fully loading, the user agent must update the session history with the new page."So any script running after the page finished loading doesn't matter with the spec.
- E
Elliott Sprehn
Apr 5, 2016, 4:57:44 PM4/5/16
to Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
I'd rather we just violated the spec than use script to work around the funny wording in the spec. Using script doesn't preserve any useful author invariants. Showing the button after a click seems like very weird UX as well. I'd say lets use a UA shadowRoot for now.
- E
Philip Jägenstedt
Apr 6, 2016, 6:00:15 AM4/6/16
to Elliott Sprehn, Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
Catching up with this thread now. To which element would then UA shadowRoot be attached? It can't be the video itself, and in my ad-hoc testing, document.body.createShadowRoot causes the video itself to become invisible.
I think it would be worth investigating whether the internal structure of MediaDocument needs to be exposed for web compat at all. If it doesn't, maybe the entire iframe could be treated as cross-origin? It's less likely that this could be done for images, but still worth measuring if video is measured.
Philip
Elliott Sprehn
Apr 6, 2016, 1:18:34 PM4/6/16
to Philip Jägenstedt, Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
I'd rather we just violated the spec than use script to work around the funny wording in the spec. Using script doesn't preserve any useful author invariants. Showing the button after a click seems like very weird UX as well. I'd say lets use a UA shadowRoot for now.
Catching up with this thread now. To which element would then UA shadowRoot be attached? It can't be the video itself, and in my ad-hoc testing, document.body.createShadowRoot causes the video itself to become invisible.
You need to add an insertion point to the shadow root. Try createShadowRoot().appendChild(document.createElement("content")).
I think it would be worth investigating whether the internal structure of MediaDocument needs to be exposed for web compat at all. If it doesn't, maybe the entire iframe could be treated as cross-origin? It's less likely that this could be done for images, but still worth measuring if video is measured.
It would certainly be nice if we didn't need to expose the MediaDocument. Someone should UseCounter it, I can't imagine why anyone would use the MediaDocument instead of using a <video> directly.
- E
Christian Biesinger
Apr 6, 2016, 1:45:17 PM4/6/16
to Philip Jägenstedt, Elliott Sprehn, Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
On Wed, Apr 6, 2016 at 6:00 AM, Philip Jägenstedt <phi...@opera.com> wrote:
> Catching up with this thread now. To which element would then UA shadowRoot
> be attached? It can't be the video itself, and in my ad-hoc testing,
> document.body.createShadowRoot causes the video itself to become invisible.
Why can't it be the video itself? We could have mediaelement check if
> Catching up with this thread now. To which element would then UA shadowRoot
> be attached? It can't be the video itself, and in my ad-hoc testing,
> document.body.createShadowRoot causes the video itself to become invisible.
it's in a mediadocument and add a download button to its shadow
root...?
-Christian
Elliott Sprehn
Apr 6, 2016, 1:52:38 PM4/6/16
to Christian Biesinger, Philip Jägenstedt, Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
There's no way to get the layout they want without putting the button inside the video without nesting another video of it. It would also make the video taller than the video it's displaying which is strange.
- E
Philip Jägenstedt
Apr 7, 2016, 7:57:24 AM4/7/16
to Elliott Sprehn, Min Qin, Domenic Denicola, dom...@chromium.org, blink-dev, Kingston Tam
On Wed, Apr 6, 2016 at 7:17 PM, Elliott Sprehn <esp...@chromium.org> wrote:
On Wed, Apr 6, 2016 at 3:00 AM, Philip Jägenstedt <phi...@opera.com> wrote:...I'd rather we just violated the spec than use script to work around the funny wording in the spec. Using script doesn't preserve any useful author invariants. Showing the button after a click seems like very weird UX as well. I'd say lets use a UA shadowRoot for now.Catching up with this thread now. To which element would then UA shadowRoot be attached? It can't be the video itself, and in my ad-hoc testing, document.body.createShadowRoot causes the video itself to become invisible.You need to add an insertion point to the shadow root. Try createShadowRoot().appendChild(document.createElement("content")).
Ah, thanks. So I guess that would work then.
I think it would be worth investigating whether the internal structure of MediaDocument needs to be exposed for web compat at all. If it doesn't, maybe the entire iframe could be treated as cross-origin? It's less likely that this could be done for images, but still worth measuring if video is measured.It would certainly be nice if we didn't need to expose the MediaDocument. Someone should UseCounter it, I can't imagine why anyone would use the MediaDocument instead of using a <video> directly.
Reply all
Reply to author
Forward
0 new messages