desktopCapture.chooseDesktopMedia steamId is not usable in offscreen document MV3

[stev...@hotmail.com]

I get "DOMException: Invalid state" error when trying to use a stream id with getUserMedia from chooseDesktopMedia.

I know we can use navigator.mediaDevices.getDisplayMedia in an offscreen document and this is working fine however chooseDesktopMedia is more usable for two reasons:

1. I cannot cancel a request to getDisplayMedia.
2. When using getDisplayMedia with multiple monitors it does not open the dialog on the current tab so users are unlikely to notice the dialog. Also, while the dialog is open no other streams can be used. This can appear as though the extension has a bug if a user tries to capture via another method.

It would be greatly appreciated if this was supported for offscreen documents in a future update.

I have now finished updating my extension to MV3 and I must say the developer experience is terrible. To accomplish the same thing in MV3 vs MV2 it requires a lot of jumping through hoops and extra code.

[Oliver Dunk]

Hi Steve,

I'm assuming this is because you are trying to consume the stream ID from a different place to where you called the extension API.

As a little bit of context, our preference is very much to move over to the web APIs for screen recording where possible. The extension APIs are still supported but even before MV3 were using older code paths and generally falling behind the work being done on the more widely used APIs. In the cases where there are differences we'd like to make sure the web APis can do everything the extension APIs can.

That said, we had a similar issue with the chrome.tabCapture API and this was changed in Chrome 116: https://developer.chrome.com/docs/extensions/reference/api/tabCapture#usage-restrictions.

For desktopCapture, I need to check with the team on our thoughts and figure out if we'd be able to make a similar change. It looks like it might be feasible technically but since usage of desktopCapture is lower than tabCapture, may be slightly harder to prioritise.

I'll try to follow up soon. If you are comfortable sharing more about your specific extension and what it does, that may be helpful to have an example of how this would be used.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/33369774-147f-4307-8e13-23c8494685b5n%40chromium.org.

[stev...@hotmail.com]

Hi Oliver,

Thank you for the response.

Yes, it is because the chrome api desktopCapture is not usable inside an offscreen document. I make the request for the stream id in the service worker and send it to the offscreen doc, similar to the tabCapture api.

Thank you for considering this update. I have already had users say the extension is broken because they failed to select media to share via the dialog.

An alternative could be to improve the web API navigator.mediaDevices.getDisplayMedia so it has the usability improvements that desktopCapture has.

Sure, Chrome Capture and Reelia are two of my screen capturing extensions:

https://chromewebstore.google.com/detail/chrome-capture-screenshot/ggaabchcecdbomdcnbahdfddfikjmphe

https://chromewebstore.google.com/detail/reelia-screen-recorder-ed/ajhidbbpmnhjjffefekncbdbeopghnnn

I just updated Chrome Capture to MV3.

For context/example of how this is used:

A user will select the record "full screen or app" option from the extension pop-up. I then close the popup and open the getDisplayMedia dialog for a user to select what to record/share. If the user has multiple monitors, the dialog may not open on the monitor they are currently interacting with. This causes confusion and until the user interacts with the dialog all recording is broken - including using tabCapture.

Cheers,

Steven.

[Oliver Dunk]

Hi Steve,

Thanks so much for the extra context.

I had a chance to speak to the team yesterday. A few thoughts:

• Being able to cancel the request and focus it on the right window both seem like reasonable things that should be supported in the web platform API. That would be the preferred approach.
• It's technically feasible to support passing a stream ID from chrome.desktopCapture to an offscreen document, but it would be hard to prioritise given the small number of requests so far.

With that in mind, could you open Chromium issues (https://issues.chromium.org/) for the two web platform API requests you have? I suspect you may be able to better explain your use case and the situations where the dialog can appear on the wrong window.

Once you've done that, I can open a bug in the extension component that points to these. I can't promise we will be able to prioritise it, but the team will keep an eye out for activity on the bug and other requests in the mailing list, as well as what happens on the web platform bugs, and we can decide what to do from there.

Thanks,

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Vibhash Chandra]

Hi,

How did you update Chrome Capture to MV3?

When the user selects the full screen option, do you send a message to the service worker and call desktopCapture.chooseDesktopMedia to get stream ID? After that do you pass stream ID to offscreen documents?

I had tried something similar and got DOMException: Invalid state.

Regards,

Vibhash.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAOsQqB%3DOKthp%2BMvpYKsKzaVmAs7nPF0WAqCuMSeO86xLLwAdMQ%40mail.gmail.com.

[stev...@hotmail.com]

Hi Oliver,

Thank you for getting back to me and sorry for the late reply.

I created the tickets:

https://issues.chromium.org/issues/326509126

https://issues.chromium.org/issues/326508296

Cheers,

Steven.

[stev...@hotmail.com]

Hi Vibhash,

Just manually after reading the documentation.

Yes, that's exactly what I was getting.

Cheers,

Steven.

On Saturday, February 17, 2024 at 5:53:53 AM UTC+13 Vibhash Chandra wrote:

[Vibhash Chandra]

Did you find a workaround after this error?

Regards,

Vibhash.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/81033cd7-9c60-4773-9406-3cc95df955b7n%40chromium.org.

[Kieran Kelk]

Hi all,

In the hopes of progressing a fix (or at least improving the workarounds) for this issue, I thought it'd be useful to write up minimal examples of each of the APIs and potential workarounds I'm aware of. Those examples (and a write-up of current behaviour in the README) can be found at https://github.com/kierankelk-synth/desktopCapture-examples.

In my view, fixing desktopCapture to work like tabCapture does in offscreen documents is the ideal solution, because the desktopCapture API can do things that getDisplayMedia can't and likely never will be able to do. Failing that, fixing the strange behaviour of dialogs in offscreen documents could be an option, but my suspicion is that that would actually be more difficult to correctly handle - and would still leave us with a theoretically more-useful but broken desktopCapture API to trip people up in the future.

I totally get that this is unlikely to be top-priority, but I'd love to hear the thoughts of a Chromium dev on what solution is favoured, and whether there is any realistic timeline for its implementation. If it's just a prioritisation issue but there is general consensus, would the situation change if I had a go at contributing a fix (no promises at this stage, but just so I have a data point, and know that at least someone would be interested in approving a CL if I did go that route).

Also, it looked to me that this was the best place to have this discussion, but I note that in addition to the issue opened in this thread, there's also this one with with some discussion on it.

Thanks,

Kieran

[Dewde]

I 100% agree with the case being made here https://github.com/kierankelk-synth/desktopCapture-examples.

A streamId generated by desktopCapture in the service worker needs to be usable by the offscreen document in the same way it uses the streamId generated by tabCapture.

This is already postponed since 116. Help us unify recording behaviors!

[Oliver Dunk]

Generally we're supportive of having desktopCapture work like tabCapture. Ideally, we would bring the missing pieces and behaviors to the web API, and then not need the desktopCapture API at all. However, that is likely a longer term solution.

Unfortunately, at the moment we haven't seen enough demand to prioritise the changes to the desktopCapture API. If you have use cases and existing extensions which are impacted, please do let us know, as we can use that to evaluate the impact of doing work there.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB