Cross origin issues when calling `toPixels` in chrome extension.

[Dan Oved]

I want to build a chrome extension that can feed any image from a webpage through a tfjs network.  The extension would search the page for any image element then allow the images to be read for inference.

As the first part of this step, in a content script in the chrome extension, I need to convert the image to pixels with tf.fromPixels(image).  Occasionally, the image comes from a cross-origin url.

When calling tf.fromPixels(image) from within the content script, an error occurs:

"DOMException: Failed to execute 'texImage2D' on 'WebGL2RenderingContext': The image element contains cross-origin data, and may not be loaded."

Has anyone ever gotten around this?  Are there any examples where cross-image origins image reading works in a chrome extension?

FYI my manifest.json looks like:

{
  "manifest_version": 2,


  "name": "tfjs extension",
  "description": "Chrome Extension, TypeScript, Visual Studio Code",
  "version": "1.0",


  "content_scripts": [
      {
          "matches": ["<all_urls>"],
          "js": ["js/content_script.js"]
      }
  ]


  "permissions": [
    "activeTab"
  ]
}

[Nikhil Thorat]

Hi Dan,

CORS can be tricky, check out this post on CORS with image reads in WebGL: https://webglfundamentals.org/webgl/lessons/webgl-cors-permission.html

TLDR: you need to make sure the right HTTP headers are set (Access-Control-Allow-Origin) / possibly set the "crossOrigin" attribute on the <img> tag.

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/5d2f613e-2de3-4b81-8bfd-e95805997c62%40tensorflow.org.

[Dan Oved]

I see.  This would be for a third party site, of which the server that hosts the image is not accessible, so guess that wouldn't be possible.  

[Laurent Denoue]

can't you download the images yourself from the extension, then process them?

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/0e57ec35-6daf-48ef-88f0-c1e64ef8042d%40tensorflow.org.

[Nikhil Thorat]

Yeah, if you can host it yourself by downloading and reuploading you should be able to read the pixels.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/0e57ec35-6daf-48ef-88f0-c1e64ef8042d%40tensorflow.org.

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/CAK7XdaXZbvQFe%2B9bLLbU6G3qumXgqcx-ch1VvHUhnTmWezniBw%40mail.gmail.com.

[Laurent Denoue]

Can't we just:

- download the image using the extension

- draw it to a canvas

- pass the canvas to tf.fromPixels

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/0e57ec35-6daf-48ef-88f0-c1e64ef8042d%40tensorflow.org.

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

[Dan Oved]

I've tried something that creates another image element and "downloads" it by setting the src:

function reloadImage(image: HTMLImageElement) {
  const copiedImage = new Image();

  return new Promise((resolve, reject) => {
    copiedImage.onload = () => {
      resolve(copiedImage);
    }

    copiedImage.crossOrigin = 'anonymous';

    copiedImage.src = image.src;
  })
}


async function toTensor(element: HTMLImageElement) {
  const image = await reloadImage(element);

  const tensor = tf.fromPixels(image);

  return tensor;
}

and get the same error.

Then I tried also rendering to canvas and getting the image data:

function loadImageData(image: HTMLImageElement) {
  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');
  canvas.width = image.width;
  canvas.height = image.height;
  context.drawImage(image, 0, 0 );
  return context.getImageData(0, 0, image.width, image.height);
}


async function toTensor(element: HTMLImageElement) {
  const imageData = await loadImageData(element);

  const tensor = tf.fromPixels(element);

  return tensor;
}

And I get an error:

DOMException: Failed to execute 'getImageData' on 'CanvasRenderingContext2D': The canvas has been tainted by cross-origin data.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/0e57ec35-6daf-48ef-88f0-c1e64ef8042d%40tensorflow.org.

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/CAK7XdaXZbvQFe%2B9bLLbU6G3qumXgqcx-ch1VvHUhnTmWezniBw%40mail.gmail.com.

[Laurent Denoue]

yes, it makes sense that setting the same URL to a new IMG (even if painting after to a CANVAS) fails.

I think you need to really download the image to disk from the extension, then load it back into a new image.

Laurent

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/449c8989-4a7c-41d3-abdd-af78e1b5da49%40tensorflow.org.

[Dan Oved]

For the record, I'm trying to run this extension while browsing wikiart.org to run tfjs models on wiki art images.

https://www.wikiart.org/

The images on that side have headers Access-Control-Allow-Origin: * so I believe it's something to do with the chrome extension config.

https://uploads5.wikiart.org/temp/5884b62e-11a4-40e0-bbaa-b02a6ea9d6a7.jpg!PinterestLarge.jpg

https://uploads1.wikiart.org/images/conrad-marca-relli(1).jpg!PinterestLarge.jpg

https://uploads6.wikiart.org/images/jes-s-rafael-soto.jpg!PinterestLarge.jpg

[Dan Oved]

Ok I got it working by following this stackoverflow answer:

https://stackoverflow.com/questions/5511378/security-err-dom-exception-18-on-using-getimagedata-in-a-chrome-extension

When granting cross-origin permissions via the "permissions" in the manifest.json, this is granted to the background script.  So I have the content script send a message with the url to the background script, and it does the processing, sending a response back.  This works:

In the content script:

    chrome.runtime.sendMessage({url: element.src}, function(response) {
      console.log(response.result);
    });

In background page script:

async function loadImage(url: string): Promise<HTMLImageElement> {
  const image = new Image();

  return new Promise((resolve: (HTMLImageElement) => void, reject) => {
    image.onload = () => { resolve(image) }

    image.src = url;
  });
}


async function toTensor(url: string) {
  const image = await loadImage(url);

  const tensor = tf.fromPixels(image);

  return tensor;
}

chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.url) {
    const imageTensor = toTensor(request.url);

    sendResponse({result: 'loaded'});
  }
});

[Laurent Denoue]

👍

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/f98aec53-35c7-4e30-85d9-0ece887af7da%40tensorflow.org.