Manifest V3: how to load an image onto an OffscrenCanvas?

[Kent Brewster]

Making slow progress implementing the same functionality on Manifest V2 and Manifest V3; notes here:

https://github.com/kentbrew/learning-manifest-v3

Next up is loading and manipulating image data in the background process. Since there's no DOM, my usual approach is useless:

let img = new Image();
img.onload = function () {
// make a canvas, draw the image, capture data, do cool stuff
};
img.src = "https://www.example.com/myImage.jpg";

Any pointers? Messing around with the fetch API got me a bunch of CORS trouble.

Thanks very much,

--Kent

PS: If you're wondering about use cases, I will need to convert instant image feature detection to work on v3:

https://medium.com/pinterest-engineering/building-instant-image-feature-detection-c6f394fbc922

[Anton Bershanskiy]

Service workers do not have document and therefore do not have DOM. Instead, canvas API is exposed via OffscreenCanvas, which is pretty much identical to <canvas> element, but without DOM. Instead of this:

var canvas = document.createElement('canvas');

var context = canvas.getContext('2d');

you do this:

new canvas = OffscreenCanvas(100, 1);

var context = canvas.getContext('2d');

[Anton Bershanskiy]

Edit: forgot new in the reply.

Correct example:

var canvas = new OffscreenCanvas(100, 1);

var context = canvas.getContext('2d');

[PhistucK]

The question is not about how to create a canvas in service workers, but about how to load an image into an OffscreenCanvas (the subject already mentions it).

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/fe2d550c-cfa4-4079-8e79-be8f088568cd%40chromium.org.

[Anton Bershanskiy]

You used new Image() which is the same as document.createElement('IMG') . These functions can not be used in a worker because there is NO document. Instead, you should use OffscreenCanvas and populate it via createImageBitmap with a Blob containing your image. That simple!

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.

[Kent Brewster]

Right, got all that.

I'm looking for help getting the image INTO the blob, please.

--Kent

> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/fe2d550c-cfa4-4079-8e79-be8f088568cd%40chromium.org.
>

> --
> You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/03945e87-7c7e-4f33-bdd4-130e72ecc4e7%40chromium.org.

[PhistucK]

I think fetch/XMLHttpRequest are the only option, but you need to add the host to the permissions in the manifest.

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/A7842079-3750-49B9-BF90-82C055F38008%40pinterest.com.

[Kent Brewster]

Pretty sure XMLHttpRequest has been deprecated in service workers and we have to use the Fetch API?

When you say "add the host to the permissions in the manifest," do you mean host_permissions? If so, I've already set it thusly:

"host_permissions": ["*://*/*"],

Thanks again,

--Kent

[PhistucK]

I think you mean "<all_urls>" instead of "*://*/*", because I believe hosts do not have a slash appended to them and they definitely do not have paths. A host is basically a protocol:// and a domain. I believe anything else is a syntax error.

☆PhistucK

[Kent Brewster]

> On Nov 18, 2019, at 12:55 PM, PhistucK <phis...@gmail.com> wrote:
>
> I think you mean "<all_urls>" instead of "*://*/*", because I believe hosts do not have a slash appended to them and they definitely do not have paths. A host is basically a protocol:// and a domain. I believe anything else is a syntax error.

Disagree; I'd be seeing a syntax error when I tried to load it otherwise. The two are functionally identical, unless the docs at https://developer.chrome.com/extensions/match_patterns don't apply to Manifest V3. If this gets you all HTTP urls:

http://*/*

... and this gets you the http and https versions of mail.google.com:

*://mail.google.com/*

... then this:

*://*/*

... should get you the http and https version of all URLs.

In any event, neither <all_urls> nor *://*/* work for me. Off I go to try some dangerous mucking about with invisible iframes....

--Kent

[PhistucK]

> Host permissions are permissions where you want to inject code, not where you can load resources from.
I believe this is not accurate. Host permissions are not only for injection, as far as I know. They (inside the "permissions" key) used to be the only way you could initiate cross-origin requests (XMLHttpRequest or fetch). They were just moved into a new "host_permissions" key. They may also function as permissions for executeScript, but the initial purpose was for cross origin requests.

☆PhistucK

On Mon, Nov 18, 2019 at 9:58 PM Anton Bershanskiy <bersh...@pm.me> wrote:

Host permissions are permissions where you want to inject code, not where you can load resources from. "host_permissions": ["*://*/*"] is a very broad permission and will result in installation-time permission warning like "this extension can read and modify any data on the websites you visit". If you need to download resources from arbitrary URLs, those downloads are restricted by Content Security Policy. The default CSP is script-src 'self'; object-src 'self' which will not allow remote images. Instead, you should specify connect-src and img-src. A good CSP would be script-src 'self'; object-src 'self'; connect-src 'https://example.com/images/*.png' or something similar, where your images are on the example.com domain at paths matching the regex.

Then you can fetch the image and put it into the blob

// fetch image and convert it to a blob

const imageBlob = await fetch('https://example.com/images/1.png').then(r => r.blob())

// convert blob to bitmap

const imageBitmap = await createImageBitmap(imageBlob);

// create canvas

const canvas = new OffscreenCanvas(100, 1);

// get 2D context

const context = canvas.getContext('2d');

// import the bitmap onto the canvas

context.drawImage(imageBitmap ,0,0);

[PhistucK]

https://developer.chrome.com/extensions/declare_permissions#host-permissions shows hosts without a path component (without the last *). Unless they explicitly changed it for "host_permissions" in version 3, this should still be the case... 

Match patterns applied more to content script matches than to host permissions, I am not sure what is going on with that documentation. Maybe things changed since I last experienced those restrictions.

If you put specific hosts, do they at least work?

☆PhistucK

[Kent Brewster]

> On Nov 18, 2019, at 1:21 PM, PhistucK <phis...@gmail.com> wrote:
>
> https://developer.chrome.com/extensions/declare_permissions#host-permissions shows hosts without a path component (without the last *). Unless they explicitly changed it for "host_permissions" in version 3, this should still be the case...
>
> Match patterns applied more to content script matches than to host permissions, I am not sure what is going on with that documentation. Maybe things changed since I last experienced those restrictions.
>
> If you put specific hosts, do they at least work?

Nope. Tried it in permissions and host_permissions:

"permissions": [
"contextMenus",
"tabs",
"<all_urls>",
"https://c2.staticflickr.com/*",
"https://kentbrew.neocities.org/*
],
"host_permissions": ["*://*/*", "https://c2.staticflickr.com/*", "https://kentbrew.neocities.org/*"]

... and the errors point to CORS trouble:

Access to fetch at 'https://c2.staticflickr.com/8/7027/6851755809_df5b2051c9_z.jpg' from origin 'chrome-extension://fppaooplgoamfmeenmdnhmiokhgmogbe' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Access to fetch at 'https://kentbrew.neocities.org/tall.png' from origin 'chrome-extension://fppaooplgoamfmeenmdnhmiokhgmogbe' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

Have tried this with no mode set in fetch and mode: cors; mode: no-cors gets me an opaque response with nothing I can use.

Hopefully I'm missing something important about CORS requests; if we can't somehow gain access to raw image data in our background script it will be a serious problem for the future of our extension.

[Kent Brewster]

Oh, hey. Once again I failed to look under the Errors button (because it is ALWAYS ON due to the manifest-v3-is-only-partially-supported warning) and missed this:

Permission '<all_urls>' is unknown or URL pattern is malformed.

So no, <all_urls> is not a valid thing in the API permissions property any more.

--Kent

> On Nov 18, 2019, at 1:21 PM, PhistucK <phis...@gmail.com> wrote:
>

[PhistucK]

I could not get any recent Chromium snapshot build (I tried like 5, going back 100 revisions on each trial) to work on my Windows 10 64 bit work computer (any tab/service worker crashes after a few seconds), so I cannot verify what works and what does not or troubleshoot the issue.

I still recommend that you remove the /* from all of the hosts, but this is as far as I can help at the moment.

Oh - also, maybe specify mode: "no-cors" in your fetch options (or the various other values).

Maybe I will have a better luck trying them at home, but this will happen in the Israeli evening.

☆PhistucK

[Kent Brewster]

Changing the pattern to *://* in manifest.content_scripts.matches gets me an error pop-up on load:

Invalid value for 'content_scripts[0].matches[0]': Empty path.
Could not load manifest.

Changing the pattern to *://* in manifest.host_perrmissions gets me this when I navigate to the page:

Unchecked runtime.lastError: Cannot access contents of url "https://example.com/foo/". Extension manifest must request permission to access this host.

<all_urls> seems to work in manifest.host_permissions as well as manifest.permissions.

For the file request: I tried no-cors but I get an "opaque" response. Not sure where to go with that; if there's a way to get to the body of an opaque response I'd love to know how.

Thanks again for your help,

--Kent

[Anton Bershanskiy]

Permissions and match expressions are different things. Script injection is always done based on match expressions and browser automatically adds match patterns to permission prompts when you install extension. So really, you should just use match patterns.
The only exception is if you want to request more permissions than you use right now so that an upfate you publish later doesn't trigger an extra pernission prompt. FYI, this pattern is frowned upon.

Also, *://* mathes only URLs with domains, e.g. https://example.com, but not https://example.com/path. In general, ://*/* is better for exaclty that reason. Even better, there is an even more general <all_urls> pattern.

[PhistucK]

I only meant those for permissions/host permissions, not for content scripts and you apparently do need (in permissions/host permissions) the last /, just not the * (I was not concentrated before, sorry).

☆PhistucK

[David Bertoni]

You might want to check out my current CL for dealing with this for browserAction.SetIcon:

https://chromium-review.googlesource.com/c/chromium/src/+/1921640

I hope that helps.

Dave

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CABc02_JL_SsP%2BOsTHob0M4ahD%3DQtK6fFQwk%2BW9Xqhkt6cZuHzg%40mail.gmail.com.

[Deco]

This will actually not work regardless of CORS - you cannot specify wildcard attributes in the manifest file such as /*, all domains must be individually specified or use <all_urls>. It is forbidden to attempt to skirt this.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CABc02_Jgprn82yBNXX2f-zM-Fg705sqByj6yCy-iNZJPbbVyag%40mail.gmail.com.

[Kent Brewster]

Turns out it's possible to do this entirely outside the background process:

https://github.com/kentbrew/learning-manifest-v3/blob/master/ep_006.md

TL;DR: there's message passing, web_accessible_resources, and iframe overlays involved. Not for the faint of heart. :)

--Kent

[PhistucK]

I just tried this manifest with Chromium 80.0.3976.0 (718444) -

{"name": "Foo", "version": "0.0.0.1", "description": "Baz",
"host_permissions": ["<all_urls>"], "background": {"service_worker": "1.js"}, "manifest_version": 3}

And ran in the service worker (either placed in the file itself, or via the console) -

fetch("https://phistuck-app.appspot.com").then(r=>r.text()).then(console.log);
fetch("https://phistuck-app.appspot.com").then(r=>r.blob()).then(console.log);
fetch("https://phistuck-app.appspot.com").then(r=>r.arrayBuffer()).then(console.log);

No CORS error was thrown (and this server does not return CORS-allowing headers).

Maybe it was fixed recently?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/06114F72-E331-4D8A-BB3A-3E0616092B93%40pinterest.com.

[Kent Brewster]

Sorry, I'm getting the same result as before, running 80.0.3977.0 Here's my (actually your) manifest.json:

{

"name": "Foo",
"version": "0.0.0.1",
"description": "Baz",
"host_permissions": ["<all_urls>"],
"background": { "service_worker": "1.js" },
"manifest_version": 3
}

Here's the service worker,1.js, which logs to show it's running:

console.log("Service worker has loaded.");

Dragged into chrome://extensions, saw the usual warning about v3:

The maximum currently-supported manifest version is 2, but this is 3. Certain features may not work as expected.

Opened chrome://serviceworker-internals/, found the worker, clicked Inspect, opened the Console tab. In the service worker console I see the expected message:

Service worker has loaded.

Entered into console:

fetch("https://phistuck-app.appspot.com").then(r=>r.arrayBuffer()).then(console.log);

Received:

Promise {<pending>}
Access to fetch at 'https://phistuck-app.appspot.com/' from origin 'chrome-extension://gfeicciokofhgphkckgmabbnnjkkmdkc' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Thanks again for trying this out but it's not working for me,


--Kent

[Kent Brewster]

Wrote up an explainer; your fetch example works for me on CORS-enabled resources but not the other kind:

https://github.com/kentbrew/learning-manifest-v3/blob/master/CORtest.md

--Kent

> On Nov 23, 2019, at 6:04 AM, PhistucK <phis...@gmail.com> wrote:
>

[PhistucK]

I am guessing some API/code path is only enabled on trunk (Chromium builds) and not on the canary, then. They do that sometimes.

☆PhistucK

[Kent Brewster]

Are you testing on a trunk build? Would you mind trying Canary, if so? Mine says 80.0.3977.0 is the up-to-date version.

--Kent

[PhistucK]

I cannot try with the canary, it is blocked in my corporate computer for some (probably dumb) reason.

☆PhistucK

[Kent Brewster]

Okay, thanks. Is there anyone within the sound of my voice who can please try to reproduce PhistucK's results on Canary?

--Kent