Streaming canvas.getImageData() from Background to Content scripts?

[Oz Ramos]

Hi! I've been working on a chrome extension to help you use the browser handsfree with support for custom mappable Face and Hand gestures (here's me scrolling Reddit with a pinch gesture):

To get this to work, I run computer vision in the background script so that it's isolated from every other context and so that you don't have to approve the webcam to any domain (only the extension itself).

The challenge is this: I'd like to add a button that allows the user to see themselves, which would be useful for recording yourself (like in the composed video above) or even as an obvious reminder that your camera is on. The problem is that the webcam is running in the background page and not in the content page so I need to stream it over.

The usual way I do it is to use $canvas.getContext('2d').getImageData() but when I actually try to sendMessage from the background it is extremely slow (1 or 2FPS per second). The data also comes in mangled...getImageData() returns a Unit8ClampedArray but the sendMessage converts it into a monstrously huge object.

Does anyone know how I can send a canvas/video from the background to the content script?

Thanks!

[James Augeri]

a very interesting question and one that we're exploring at Jingle (since we may on occasion need to pass large blobs); previous company was image processing & CV related so doubly interested :) suppose that the first thing that comes to mind is creating a connect pipe to see if that's faster than sendMessage?

re: connect, port, etc., e.g.,

1. https://developer.chrome.com/docs/extensions/reference/runtime/
2. https://developer.chrome.com/docs/extensions/reference/runtime/#type-Port
3. https://developer.chrome.com/docs/extensions/reference/runtime/#method-connect
4. https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/connect

in theory with the port being continually open, should be able to boost FPS?

James / https://twitter.com/DotDotJames 

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/11634b15-5c8d-40a9-9655-2f7c8a3cfe6en%40chromium.org.

--

Thanks,
James

James Augeri, PhD, Founder
Jingle, fix your workflow chaos
Web, https://heyjingle.com/

Email, ja...@heyjingle.com
Cell, 402-522-6787
Twitter, @DotDotJames

[Jackie Han]

Hello, I thought of a very simple solution. You don't need to send webcam video from background script to content script. You can use Picture-in-Picture (PiP) api to swap out your video from background to front end ( and PiP window above all other windows). 

I didn't test whether the PiP api can work in the background page, I've only used it on regular web pages before.

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/11634b15-5c8d-40a9-9655-2f7c8a3cfe6en%40chromium.org.

--

韩国恺(Jackie)

[Jackie Han]

PiP api is supported by most modern browsers in 2020.

https://caniuse.com/?search=Picture-in-Picture

On Wed, Dec 30, 2020 at 5:59 AM James Augeri <ja...@heyjingle.com> wrote:

Jackie's proposed approach should offer more throughput if works -- is the PiP API cross-browser?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAAgdh1%2BfyRXvHAVo0zUBLjtQ_%2Ba%2BPyCDfBqnn4C1fdONhCv2OQ%40mail.gmail.com.

--

Thanks,
James

James Augeri, PhD, Founder
Jingle, fix your workflow chaos
Web, https://heyjingle.com/

Email, ja...@heyjingle.com
Cell, 402-522-6787
Twitter, @DotDotJames

--

韩国恺(Jackie)

[James Augeri]

Jackie for the win!

Thanks,

James

James Augeri, PhD, Founder

Jingle, fix your workflow chaos

Web, https://heyjingle.com/

Twitter, @DotDotJames

On Dec 29, 2020, at 4:07 PM, Jackie Han <han.g...@gmail.com> wrote:



<Screen Shot 2020-12-30 at 6.04.05 AM.png>

PiP api is supported by most modern browsers in 2020.

https://caniuse.com/?search=Picture-in-Picture

On Wed, Dec 30, 2020 at 5:59 AM James Augeri <ja...@heyjingle.com> wrote:

Jackie's proposed approach should offer more throughput if works -- is the PiP API cross-browser?

On Tue, Dec 29, 2020 at 3:55 PM Jackie Han <han.g...@gmail.com> wrote:

Hello, I thought of a very simple solution. You don't need to send webcam video from background script to content script. You can use Picture-in-Picture (PiP) api to swap out your video from background to front end ( and PiP window above all other windows). 

I didn't test whether the PiP api can work in the background page, I've only used it on regular web pages before.

On Wed, Dec 30, 2020 at 5:03 AM Oz Ramos <midib...@gmail.com> wrote:

Hi! I've been working on a chrome extension to help you use the browser handsfree with support for custom mappable Face and Hand gestures (here's me scrolling Reddit with a pinch gesture):

--

韩国恺(Jackie)

[Oz Ramos]

Oh wow this is amazing, thank you both for the speedy responses! Using PiP is so clever I'll give that a try now. In fact I may use that for other areas of my extension too. Also thank you James for the links, even if I end up going with PiP the links you pointed me will help me solve other communication challenges!

I'll report back with what I find, thanks again!

[hrg...@gmail.com]

On Tuesday, December 29, 2020 at 6:03:39 PM UTC-3 midib...@gmail.com wrote:

To get this to work, I run computer vision in the background script

This is quite an interesting use case.

But I worry whether this functionality will be possible to implement in Manifest V3.

Have you thought about how you'll make this work once the background page is no longer supported?

[fx noob]

For v3, background page that holds the core logic for computer vision stuff, can be migrated to option page(page that lies under the extension url ). So when ever user clicks on extension icon, that option page can be opened and utilised as background page.

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/d5dead5d-71f8-4ab7-8b45-214d7ae84d7bn%40chromium.org.

--

fxnoob Noob github.com/fxnoob

[guest271314]

sendMessage() only sends text, does not have support for transferable objects.

PictureInPicture window implementation at Chrome and Chromium restricts maximum width and height to half of screen with and height. 

Since clipboard can be accessed from "background", "content", and main thread you can use (async) clipboard API to write and read SDP and stream the camera from "background" to "content" script. 

Alternatively the clipboard (DataTransfer) itself can be used to write ansd read from, including binary data, e.g., https://stackoverflow.com/a/47172409.

An example of streaming audio from Mozilla Nightly to Chromium https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c , and an example of using Chromium as  Streams API "module" at Mozilla Nightly using RTCDataChannel to execute pipeTo() at Chroniun, which Firefox does not support.

[Oz Ramos]

Thank you all for your suggestions, all of them are applicable to various parts of my extension! I ended up going with Picture in Picture. Actually what I did was render webcam into canvas, render the green computer vision wireframes on top of them, and then use that canvas as a srcObject for a separate <video>. That composed video is what I then PiP and it gives the cool effect of being able to see the wireframes on top of your hands!

Thanks again for the suggestions everyone!

[Jackie Han]

Congratulations! Your extension is very creative!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/db482e64-6c97-44d3-b801-aee3a0ebadd0n%40chromium.org.

--

韩国恺(Jackie)

[James Augeri]

really great work with PiP — congrats!!!

do add me to your beta test list — really like this concept - can DM me on Twitter at @DotDotJames for my email

Thanks,

James

James Augeri, PhD, Founder

Jingle, your turbo button for work

Web, https://heyjingle.com/

On Dec 31, 2020, at 2:09 AM, Jackie Han <han.g...@gmail.com> wrote:



Congratulations! Your extension is very creative!

On Thu, Dec 31, 2020 at 10:22 AM Oz Ramos <midib...@gmail.com> wrote:

Thank you all for your suggestions, all of them are applicable to various parts of my extension! I ended up going with Picture in Picture. Actually what I did was render webcam into canvas, render the green computer vision wireframes on top of them, and then use that canvas as a srcObject for a separate <video>. That composed video is what I then PiP and it gives the cool effect of being able to see the wireframes on top of your hands!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAAgdh1LNvyCD68V1yxMENnQLFVYw5pqe-R_cr%3DmbJ9JEctRH3Q%40mail.gmail.com.

[Oz Ramos]

Thanks Jackie! I still can't believe how well it runs with PiP, I don't notice any performance loss at all. Your suggestion also helped me accidently discover that you can set a canvas as a source to a video which is I've dreamt of for a while but didn't think it was possible.

And thanks James will do!!

[François Beaufort]

It is pretty cool indeed! Nice job and great collaboration folks!

If you need somehow to plug some media icons into the PiP window, check out this cool hack as well https://developers.google.com/web/updates/2018/10/watch-video-using-picture-in-picture#show_canvas_element_in_picture-in-picture_window  ;)

[bhagyash ingale]

Very cool ! All you need now is a toggle button to hide the stream once people are bored of looking at themselves :P

Good luck

[Daniel Fogle]

Thank you very much for giving me now the opportunity again to work alongside of you and being able to become a part of your team to learn how to build inside computer chips and all that good stuff with Google I do appreciate it and good luck to all of you

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/acf64fb1-049f-4068-83e6-20b963c3e22dn%40chromium.org.

[Simeon Vincent]

Thanks Jackie! I still can't believe how well it runs with PiP, I don't notice any performance loss at all. Your suggestion also helped me accidently discover that you can set a canvas as a source to a video which is I've dreamt of for a while but didn't think it was possible.

FWIW, what you describe here is how most web-based camera apps work: they stream video to a canvas that serves as the user's viewfinder and when the user "takes a picture" they save a frame from the camera's stream. See Taking still photos on MDN for a more detailed walkthrough. Hopefully someday we'll actually be able to use the ImageCatpure API on non-Chromium browsers.

I really like the PIP approach as it doesn't require injecting extension UI in every page, which could lead to duplicated rendering. Beyond this use case, it also potentially allows you to avoid injecting content scripts, helping keep user data secure.

For v3, background page that holds the core logic for computer vision stuff, can be migrated to option page(page that lies under the extension url ). So when ever user clicks on extension icon, that option page can be opened and utilised as background page.

This seems like a reasonable approach. Another idea that leaps to mind is to use a dialog window as your video surface and to also handle the image processing work. If necessary, you could also spawn a worker from that dialog to keep the UI thread (main thread) jank-free.

There are two main drawbacks to this approach. 1) While dialogs have reduced window chrome, there's still a title bar that you can't get rid of. 2) As I recall you can only make a (dialog) window draw on top of everything else in Chrome OS. Both of these limitations seem like interesting fodder for feature requests. Just to set expectations, though, the team is hard at work on MV3 capabilities, so they might be a bit … preoccupied.

Simeon - @dotproto

Chrome Extensions DevRel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAK7w0Rra4K6MbXWCT-XNpMaiQ033_6tuct0syqw%2B2a-9i1KWsw%40mail.gmail.com.

[Jackie Han]

A little off topic. "a popup/window that is always on top" is a common feature request. However, it seems that Chrome doesn't want to support this feature because of UI / UX considerations. PIP implements this feature, but only for video.

[hrg...@gmail.com]

On Tuesday, April 13, 2021 at 4:40:53 AM UTC-4 Jackie Han wrote:

A little off topic. "a popup/window that is always on top" is a common feature request. However, it seems that Chrome doesn't want to support this feature because of UI / UX considerations. PIP implements this feature, but only for video.

An always-on-top popup window is the closest thing to achieve that. A few extensions use this as a sort of side panel. It's an ugly hack but it kind of works if you don't move around your browser windows to much.

 

[guest271314]

> PIP implements this feature, but only for video.

What is the requirement?

navigator.mediaDevices.getDisplayMedia() "Application" capture can capture PIP window. A MediaStreamTrack of kind 'audio' can be added to the MediaStream therefrom. Chromium now supports audio capture at getDisplayMedia(), with the exception being Web Speech API speechSynthesis.speak() audio output is not captured.

[Jackie Han]

What is the requirement?

A window that is always on the top over all other windows, even when you click other window.

[guest271314]

Basically an "Always on top" option for Chromw window, similar to a window manager on *nix, correct?

That should be possible, to an appreciable degree, using workarounds.