Intent to Ship: Media Capture from Canvas

[Emircan]

Contact emails

emi...@chromium.org, nik...@chromium.org

Spec

Media Capture from DOM Elements: HTML Canvas Element Media Capture Extensions

Summary

“Media Capture from DOM Elements” document by W3C defines captureStream() method that allows the capture of the <canvas> element in the form of a MediaStream. We implemented the necessary Blink and Chromium sections that would create this stream by accessing the canvas output according to the given frame rate constraints.

Link to “Intent to Implement” blink-dev discussion

Intent to Implement: Media Capture from Canvas

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

https://cdn.rawgit.com/uysalere/js-demos/master/intro.html

Debuggability

-

Interoperability and Compatibility Risk

This feature is defined by the W3C spec and already supported by Mozilla Firefox.

During the capture, we can observe some unintended side effects of our canvas implementation. The common mechanisms that animate the content of a canvas (like requestAnimationFrame and setTimeout) may stop or get throttled down in the cases below.

1) The page with <canvas> is not visible.

2) Canvas created with document.createElement() but never inserted to the DOM.

3) Canvas with "display:none".

4) Canvas in iframes scrolled out of view.

We will continue the discussion the discussion and work on possible solutions if necessary for these cases in the meantime.

OWP launch tracking bug

https://crbug.com/524218

Entry on the feature dashboard

https://www.chromestatus.com/features/4817998447640576

[PhistucK]

Beside Firefox, does any other browser show interest? Any signal?

Can you try and get some responses?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Does shipping this amount to switching MediaCaptureFromElement to stable, or are the other changes involved?

The code enabled by MediaCaptureFromElement doesn't match the spec, it has a CanvasCaptureMediaStreamTrack instead CanvasCaptureMediaStream, which would be entirely non-interoperable with the spec and Firefox.

Reading the spec, I struggle to understand what the intended processing model is. Why does it make sense to have a framerate and a mechanism like requestFrame() that "allows applications to manually request that a frame from the canvas be captured and rendered into the media stream"?

I naively wonder why this feature cannot be an argument-less captureStream() that returns a plain MediaStream, and that the same things that trigger a canvas to be updated on-screen is also what causes a new frame to encoded for the MediaStream? Then it up to the web developer to update the canvas however often is desired, which could be faster or slower than rAF callbacks.

--

[Emircan]

Shipping this would mean changing MediaCaptureFromCanvas to stable. We are splitting media capture from <canvas> and <video>/<audio> behind different flags here.[0] We want to enable <canvas> case and still keep the rest behind test flag.

CanvasCaptureMediaStreamTrack issue was discussed and agreed here[1]. w3c.github.io link is updated[2] but w3.org is not. I will ask about why that is the case.

Regarding the API choice on requestFrame() and args, I cannot give you the whole reasoning behind as I wasn't involved in the initial discussions. However, captureStream() that is updated whenever canvas is updated is a supported case currently. The other restrictions help the developer to show the local user 30 fps while encoding&sending a smaller number of frames to the peer connection. We can bring up a topic on the spec discussion board if you want. 

[0] https://codereview.chromium.org/1689393002/

[1] https://github.com/w3c/mediacapture-fromelement/pull/18

[2] https://w3c.github.io/mediacapture-fromelement/#the-canvascapturemediastreamtrack

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Emircan]

AFAIK there aren't. I will try to get responses in the meantime and keep this updated.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Harald Alvestrand]

Re CanvasCaptureMediaStreamTrack vs CanvasCaptureMediaStream: This is changed in the spec and the editors' draft updated; I'll take an action to get another WD published.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Ah, so the frame rate is in fact a limit on the frame rate in case updates should happen faster, and not a mechanism to cause updates to happen faster? That makes a lot more sense. But then what is the purpose of requestFrame(), wherever it is located? That seems to only cater to more frequent updates.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Do you have any background on the CanvasCaptureMediaStreamTrack vs. CanvasCaptureMediaStream change? Are the Gecko developers on board with this change and planning to change soon? I am kind of hoping no CanvasCapture* specialization will be needed of any interface, but if it is interoperability is worth a lot.

P.S. If you need any ammunition to change the policy on updating w3c.github.io for each commit, you can argue that it has caused confusion on two occasions recently, the other being the webrtc-pc changes around exceptions.

[Michael[tm] Smith]

Emircan <emi...@chromium.org>, 2016-02-16 11:30 -0800:

> On Tuesday, February 16, 2016 at 11:32:31 AM UTC-5, PhistucK wrote:
> >
> > Beside Firefox, does any other browser show interest? Any signal?
> > Can you try and get some responses?
>
> AFAIK there aren't. I will try to get responses in the meantime and keep
> this updated.

Please see the WebKit bug https://bugs.webkit.org/show_bug.cgi?id=152591
for it (full disclosure: I raised it) and note that it has a reference to a
Radar issue and also that the implementors who are the ones there that you’d
probably want to chat with about it are already (self) Cc’ed on that bug.

That all looks to me at least like a signal of interest.

—Mike

P.S. I also raised an Edge uservoice thing for it:

https://wpdev.uservoice.com/forums/257854-microsoft-edge-developer/suggestions/11252697-video-capture-from-canvas

...but no similar signals of interest have been shown there yet.

--
Michael[tm] Smith https://people.w3.org/mike

[Emircan]

captureStream(0) is a valid case where users doesn't expect any updates on the captured MediaStream, unless they manually ask for frames by using requestFrame(). It gives them a manual control.

Regarding CanvasCaptureMediaStreamTrack; you can find the discussion here: https://github.com/w3c/mediacapture-fromelement/issues/16 .

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Philip Jägenstedt]

OK, this makes sense. By default, no frame rate is given and the MediaStream is updated whenever the canvas is. If that's too much, on can either limit the framerate or say precisely when to capture the contents.

The kDefaultFrameRate = 60 looks a bit suspicious. I'm not sure it actually ends up being used, but if it is that sounds like a problem, as one risks dropping a frame if the frame rate is very close to 60 but a slight jitter. Is this an actual problem?

I took a look at the spec, and filed issues for things seemingly left undefined:

https://github.com/w3c/mediacapture-fromelement/issues/26

https://github.com/w3c/mediacapture-fromelement/issues/27

https://github.com/w3c/mediacapture-fromelement/issues/28

https://github.com/w3c/mediacapture-fromelement/issues/29

https://github.com/w3c/mediacapture-fromelement/issues/30

https://github.com/w3c/mediacapture-fromelement/issues/31

Doing some experimentation it was very easy to find cases where behavior was different from Firefox, and it seems likely that web developers will stumble across some of the same differences. Are there any shared tests in web-platform-tests or anything else that gives you any idea about how interoperable the two implementations are?

I also filed some issues blocking crbug.com/524218 which are hopefully invalid or simple to fix.

Can you look over the issues and see which of the valid ones would be worth fixing before shipping, and which you'd prefer to punt on? Overall, I think the framerate throttling and requestFrame() mechanism is where most of the trouble is, so shipping an uncomplicated subset first in M50 might be an option, but only if the rest turns out to be complicated and that's what you'd really prefer to do.

Overall, this looks like a great and useful feature. If you can get yourself spec editing powers, it should be easy enough to sort out the interop issues, I think.

Philip

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Emircan]

Thanks Philip for the bugs/interop issues you pointed out. I responded to them as much as I could. Regarding spec editing powers, I will contact hta@ and see what I can do to go through them.

We have added a UMA count here to track the usage, and we can see that people have been trying it out. We aimed at M50 when we sent I2I and started implementing. It would be nice to reach that goal if we agree on the discussion points by the cut.  

kDefaultFrameRate is a Chrome-side implementation requirement coming from VideoCapturerSource interface. GetCurrentSupportedFormats() is expected to return the supported frame rate of the source. For captureStream(frameRate) cases, I return the user given frameRate. For captureStream() case I return 60, assuming it is the cap for canvas capture per second. 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Harald Alvestrand]

Emircan, Martin Thomson has been responsive with spec changes.

File issues or pull requests at http://w3c.github.io/mediacapture-fromelement/ - if you have a pull request, it's one click for Martin to integrate it.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Have you also taken a look at the issues on GitHub? I couldn't figure out what your GitHub account was or I would have poked you when I created them. The issues don't look particularly complicated, and could presumably be resolved easily if you compare notes with the Firefox implementors.

About kDefaultFrameRate, is that information only, does the actual frame rate have to match? Or, to get right to the problem, if 17ms ever pass between two canvas updates, will that cause a dropped frame?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Emircan]

I am taking a look at GitHub issues as well, still new to the github W3C workflow. We can carry the discussion over crbugs and github issues at this point.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Emircan]

I wanted to give an update on this. We are looking to ship this for M51 and I am looking forward to your feedbacks or approvals.

Issues addressed and closed since I2S is sent(Thanks philipj@ for raising most of them):

- Handle alpha-channel

- Fix timed capture drift

- Fix timestamps

- Some spec discussions and improvements [0] [1] [2]

[nik...@chromium.org]

Hi API OWNERS, how can we make this i2s to move forward? As Emircan said we want to ship this in 51 and unflag it as soon as possible. If there's anything that blocks that we want to start fixing that immediately.

Niklas

[Philip Jägenstedt]

LGTM1 with commentary:

Most of the spec issues still remain open, but for most it's now clear what should happen. Can someone comment on each open issue what behavior Blink will now ship, and that it's expected that the spec will be made to match? Shipping closes the door on certain kinds of changes, and it's just as well to say it explicitly up front.

There is still one issue that's worth worrying about, I think:

https://github.com/w3c/mediacapture-fromelement/issues/29

https://bugs.chromium.org/p/chromium/issues/detail?id=587789

There are two things going on here. First, no spec seems to define precisely when a canvas is updated. I've filed https://github.com/whatwg/html/issues/924 for this. It's probably straightforward in the case of visible canvases, but potentially hairy for canvases that aren't in the document at all.

Second, a MediaStream in Gecko holds on to the latest frame, but not in Chromium. This is likely to cause occasional differences in observable behavior. At the end of the day, some spec needs to spell out whether a MediaStream ever holds on to anything that flows through it. Without giving it too much thought, spec'ing Chromium's behavior seems like an option at least. tommi@, peter@, as modules/mediastream/OWNERS, how would you propose to move forward with that?

Philip

Niklas

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Rick Byers]

On Wed, Mar 23, 2016 at 2:32 AM, Philip Jägenstedt <phi...@opera.com> wrote:

LGTM1 with commentary:

Most of the spec issues still remain open, but for most it's now clear what should happen. Can someone comment on each open issue what behavior Blink will now ship, and that it's expected that the spec will be made to match? Shipping closes the door on certain kinds of changes, and it's just as well to say it explicitly up front.

This makes me a little nervous in the abstract.  In general I think our policy is that we only ship features which have been well specified somewhere, and I don't think "open github issues where the conclusion is mostly clear" really counts as "well specified" (assuming developer-impacting spec changes are required based on the conclusion).  A pull request with concrete changes may (assuming there are no major outstanding concerns raised with that PR) - chromestatus can always link to the forked version of the spec until the PR is merged.  This is to ensure we're only shipping behavior which another browser can reasonably reproduce exactly and developers can reasonably understand the fine details of.

But I don't know much about this space and I haven't looked through the specific issue.  Philip, what open design issues are there which aren't covered by your "most"?  Are any of the open issues about API shape / substantial behavior (i.e. things we'd change MDN docs / code samples based on the results)?  Or are they all just spec formalities that developers won't generally need to be aware of (and so could potentially be changed later with a minimum of compat risk)?  I trust your judgement in this space, so if you think the spec is good enough to describe the behavior to developers, and the spec+issues are good enough to permit interoperability, then that's good enough for me.

Another thing that would reduce the risk here is a test suite.  Are there any interop tests (Eg. web-platform-test) for this API?

[Emircan Uysaler]

Thanks for all the help and issues pointed out Philip.

On Wed, Mar 23, 2016 at 8:41 AM Rick Byers <rby...@chromium.org> wrote:

This makes me a little nervous in the abstract.  In general I think our policy is that we only ship features which have been well specified somewhere, and I don't think "open github issues where the conclusion is mostly clear" really counts as "well specified" (assuming developer-impacting spec changes are required based on the conclusion).  A pull request with concrete changes may (assuming there are no major outstanding concerns raised with that PR) - chromestatus can always link to the forked version of the spec until the PR is merged.  This is to ensure we're only shipping behavior which another browser can reasonably reproduce exactly and developers can reasonably understand the fine details of.

Thanks for the explanation and reference Rick. I understand your concerns and I'll try to clarify things as much as I can.

The issues where verdict is clear and can be closed are: #31, #30, #28, #27 . One issue that is still discussed and philipj@ also pointed out is #29. This is about MediaStreamTrack behavior, holding onto the last frame or not: Firefox does and Chrome doesn't. This discussion actually belongs to Media Capture Main spec, and probably affects other issues as well as this one. 

Note that the rest of the open github issues of the spec are related to <video> capture. That is behind a seperate flag, and it is not going to be shipped here. This I2S is only about <canvas> capture.

[Philip Jägenstedt]

On Thu, Mar 24, 2016 at 6:38 AM, Emircan Uysaler <emi...@google.com> wrote:

Thanks for all the help and issues pointed out Philip.

On Wed, Mar 23, 2016 at 8:41 AM Rick Byers <rby...@chromium.org> wrote:

This makes me a little nervous in the abstract.  In general I think our policy is that we only ship features which have been well specified somewhere, and I don't think "open github issues where the conclusion is mostly clear" really counts as "well specified" (assuming developer-impacting spec changes are required based on the conclusion).  A pull request with concrete changes may (assuming there are no major outstanding concerns raised with that PR) - chromestatus can always link to the forked version of the spec until the PR is merged.  This is to ensure we're only shipping behavior which another browser can reasonably reproduce exactly and developers can reasonably understand the fine details of.

Thanks for the explanation and reference Rick. I understand your concerns and I'll try to clarify things as much as I can.

The issues where verdict is clear and can be closed are: #31, #30, #28, #27 . One issue that is still discussed and philipj@ also pointed out is #29. This is about MediaStreamTrack behavior, holding onto the last frame or not: Firefox does and Chrome doesn't. This discussion actually belongs to Media Capture Main spec, and probably affects other issues as well as this one. 

Note that the rest of the open github issues of the spec are related to <video> capture. That is behind a seperate flag, and it is not going to be shipped here. This I2S is only about <canvas> capture.

Yep, that's a good summary. Issue #29 started out with a rather specific test case, and as I say turned into two different concerns. The HTML integration suggested by junov@ looks about right to me, and crucially it would mean that if one starts canvas capture and pokes at the canvas in the same script execution, the new state of the canvas would reliably be the thing that ends up captured.

The main compat risk I think is whether a MediaStream holds on to a frame or not, and I'd still like feedback about how to proceed with that. It could be that it's actually controversial and that Gecko's behavior is very intentional in order to make things more predictable, for example.

As for spec maturity, one concern I have is that this can be outside of the control of the implementor. The implementor and spec editor will not always be the same person or work for the same company, and that's a good thing more often than not. I'm not sure that the editor in this case even knows about this Intent to Ship, so making it explicit on the relevant issues should flush out any concerns if they exist.

Philip

[Emircan Uysaler]

Update:

After the fix discussed here , the issue in #29 is solved for canvas capture. (Check the demo mentioned) MediaStreamTrack behavior -holding onto the last frame or not- can continue to be discussed, but it no longer affects this feature. 

As I said, we are aiming for M51, and I am looking forward to your feedbacks or approvals. Thanks.

[Dimitri Glazkov]

Rick, have the interop concerns you raised been addressed adequately?

:DG<

[Philip Jägenstedt]

Because I've read my inbox in order I've already commented on the review and issue 597876, but maybe this is a better place to discuss this. Is the intention of the fix to behave as if a MediaStreams holds onto the last frame that passes through it, or something else? How would it be expressed in spec terms?

--

[Emircan Uysaler]

I responded on issue 597876 as well. Sorry for the repeated replies.

Note that the fix doesn't hold onto last frame in MediaStreamTrack or change state. It propagates RequestRefreshFrame() calls down to the Source when a new Sink is added, and Source decides to send a refresh frame or not. It was a part of the plan on issue 486274 about tab capture. I decided to take the same approach to canvas capture so that it solves interoperability issues with Firefox in those examples and we don't have the side effects(really risky) of holding onto all last frame in the MediaStreamTrack. 

Regarding spec changes, AFAIK there isn't a clear description of MediaStreamTrack behavior, i.e. holding onto frames(Firefox vs Chrome) or adding sinks on mediacapture-main spec. I think that discussion belongs to a seperate thread.

[Rick Byers]

Sorry for the delay.  I've reviewed the open issues, and there are definitely still some spec changes required but I agree with the assessment that they are relatively minor and so probably shouldn't block shipping.  Worst case and the spec goes in a different direction from what we've shipped, these are likely things we could change (and if we find otherwise that's a very strong argument for changing the spec).  Please continue to work on improving the spec here (including officially joining the WG so you can make contributions without failing the IPR check).  We can't expect to achieve interop (and so ship something that developers are happy using) without a precise description of the behavior (and ideally tests that validate it).

LGTM2 to ship.

On Tue, Mar 29, 2016 at 3:10 PM, Emircan Uysaler <emi...@google.com> wrote:

I responded on issue 597876 as well. Sorry for the repeated replies.

Note that the fix doesn't hold onto last frame in MediaStreamTrack or change state. It propagates RequestRefreshFrame() calls down to the Source when a new Sink is added, and Source decides to send a refresh frame or not. It was a part of the plan on issue 486274 about tab capture. I decided to take the same approach to canvas capture so that it solves interoperability issues with Firefox in those examples and we don't have the side effects(really risky) of holding onto all last frame in the MediaStreamTrack. 

Regarding spec changes, AFAIK there isn't a clear description of MediaStreamTrack behavior, i.e. holding onto frames(Firefox vs Chrome) or adding sinks on mediacapture-main spec. I think that discussion belongs to a seperate thread.

On Tue, Mar 29, 2016 at 2:29 AM Philip Jägenstedt <phi...@opera.com> wrote:

Because I've read my inbox in order I've already commented on the review and issue 597876, but maybe this is a better place to discuss this. Is the intention of the fix to behave as if a MediaStreams holds onto the last frame that passes through it, or something else? How would it be expressed in spec terms?

On Tue, Mar 29, 2016 at 2:17 AM, 'Emircan Uysaler' via blink-dev <blin...@chromium.org> wrote:

Update:

After the fix discussed here , the issue in #29 is solved for canvas capture. (Check the demo mentioned) MediaStreamTrack behavior -holding onto the last frame or not- can continue to be discussed, but it no longer affects this feature. 

As I said, we are aiming for M51, and I am looking forward to your feedbacks or approvals. Thanks.

On Thu, Mar 24, 2016 at 12:16 AM Philip Jägenstedt <phi...@opera.com> wrote:

On Thu, Mar 24, 2016 at 6:38 AM, Emircan Uysaler <emi...@google.com> wrote:

Thanks for all the help and issues pointed out Philip.

On Wed, Mar 23, 2016 at 8:41 AM Rick Byers <rby...@chromium.org> wrote:

This makes me a little nervous in the abstract.  In general I think our policy is that we only ship features which have been well specified somewhere, and I don't think "open github issues where the conclusion is mostly clear" really counts as "well specified" (assuming developer-impacting spec changes are required based on the conclusion).  A pull request with concrete changes may (assuming there are no major outstanding concerns raised with that PR) - chromestatus can always link to the forked version of the spec until the PR is merged.  This is to ensure we're only shipping behavior which another browser can reasonably reproduce exactly and developers can reasonably understand the fine details of.

Thanks for the explanation and reference Rick. I understand your concerns and I'll try to clarify things as much as I can.

The issues where verdict is clear and can be closed are: #31, #30, #28, #27 . One issue that is still discussed and philipj@ also pointed out is #29. This is about MediaStreamTrack behavior, holding onto the last frame or not: Firefox does and Chrome doesn't. This discussion actually belongs to Media Capture Main spec, and probably affects other issues as well as this one. 

Note that the rest of the open github issues of the spec are related to <video> capture. That is behind a seperate flag, and it is not going to be shipped here. This I2S is only about <canvas> capture.

Yep, that's a good summary. Issue #29 started out with a rather specific test case, and as I say turned into two different concerns. The HTML integration suggested by junov@ looks about right to me, and crucially it would mean that if one starts canvas capture and pokes at the canvas in the same script execution, the new state of the canvas would reliably be the thing that ends up captured.

The main compat risk I think is whether a MediaStream holds on to a frame or not, and I'd still like feedback about how to proceed with that. It could be that it's actually controversial and that Gecko's behavior is very intentional in order to make things more predictable, for example.

As for spec maturity, one concern I have is that this can be outside of the control of the implementor.

I think it's our responsibility as implementors to ensure we're only shipping features that have some reasonably "mature" specification somewhere.  Worst case and the official spec is not being updated to fix the gaps we find, we can always work on our own fork with pull requests (or if necessary, some standalone document describing our behavior in detail).  The spec being outside our control is not a legitimate excuse for shipping underspecified APIs IMHO.

The implementor and spec editor will not always be the same person or work for the same company, and that's a good thing more often than not. I'm not sure that the editor in this case even knows about this Intent to Ship, so making it explicit on the relevant issues should flush out any concerns if they exist. 

Yep, agreed on this point.

[Chris Harrelson]

LGTM3