How to predict image by url (images on s3 bucket) using trained model

[John Pham]

Hi guys

Have anyone implemented a feature that supports predict image by URL use tensorflowjs yet?

In this case we use: S3 bucket, tensorflowjs and lambda function

[John Pham]

Please help on this :(

[Kevin Scott]

John,

I've been working on an image classifier along the lines of XCode's create-ml that might help you:

https://github.com/thekevinscott/ml-classifier-ui

The demo (in /example) has a search feature that pulls images via the Imgur API. It suffers from occasional CORS issues but if you're pulling from an S3 bucket that shouldn't be an issue for you.

It's still a work in progress (in particular, UI performance is not great - happy to accept PRs!) but maybe it can guide you in the right direction.

Kevin

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/a50c8c97-56b2-4f7c-90fc-1c9473602d78%40tensorflow.org.

[John Pham]

Hi @Kevin Scott

Thank you for the response. I just have a look and I think we have a little bit confusing here. Your example only works on browser right?

I want to have an example that converts an image buffer get from remote URL => predictable data for the tensorflow model. Because i want to do predict on nodejs backend server so we can't have HTML Image Element as on browser

[Nikhil Thorat]

Hi John,

Can you first download the image, get the bytes of the decoded image, and pass it to our tensor constructor?

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/be8533e1-5103-48ad-9654-012755e88fee%40tensorflow.org.

[Kevin Scott]

Yes, only in the browser. I spent a little time looking into Node.js image data and wrote up my findings here:

https://thekevinscott.com/dealing-with-mnist-image-data-in-tensorflowjs/#fetching-image-data-outside-of-the-dom

I (believe) the team is working on more advanced image manipulation tools, so you may want to wait to hear from them!

[Kevin Scott]

Ah, see the comment above mine - defer to Nik's advice!

[Iconoclasta Vesgo]

I'm using a package callled get-pixels! check it out... https://www.npmjs.com/package/get-pixels

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/90f59e8d-c979-4cee-b505-f8ea20311a4c%40tensorflow.org.

[John S]

Tiago, could you please share how you load an image into the model.

Im having problems getting the right data and malking a tensor.

Ive tried both going with using fromPixels and loading the image array into a tensor but no luck. A working workflow would be goo to understand.

thx

[Iconoclasta Vesgo]

Hi, so get-pixels does what canvas.getImageData does, it spits out pixel data in a flat array. I bang my head arround it for a while too.

What i've done is call getPixels with the url of the file, get the pixels.data from it, and remove the last channel, by slicing the array. Here's my implementation, it's not optimized but it works:

function getImage(arr) {

let img_size = 64 * 64 * 3

let pixels = arr.slice(0, img_size)

if(pixels.length !== 12288) //this is just a sanity check

console.log(pixels.length)

return pixels.map(cur => cur / 255) //normalize pixel data to 0/1

}

function pix(file) {

return new Promise((resolve, reject) => {

getPixels(file, function(err, pixels) {

if(err) {

console.log("Bad image path", file)

return reject()

}

// console.log(pixels.data.length)

return resolve(getImage(Array.from(pixels.data)))

})

})

}

//this is the fn that loops all my local image files

async function loadImages(){

let images = []

for (var i = 0; i < NUM_DATASET_ELEMENTS; i++) {

const image = path.join(IMAGE_URL, FILES[i])

let arr = await pix(image)

images.push(arr)

if(i % 500 === 0 && i !== 0){ //log the process...

let p = Math.round((i / NUM_DATASET_ELEMENTS) * 100)

process.stdout.write('\r' + p + '% done!')

}

}

return images

}

Hope it helps!

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/b788aaa9-d282-4a55-8752-1273c9f858dc%40tensorflow.org.

[Daniel Smilkov]

Since you are in nodejs, try the node-canvas package:

const {Image} = require('canvas');

const img = new Image()

img.src = ....

You can use the Image.src property to point to a URL. See this section for details.

Then you can pass the image to tf.fromPixels(img) to make a tensor.

Hope this helps!

Daniel

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/CAAirNBm6nFHNOhqZoLQ%3D%3D%2B%2BEQT9GT0KamY1DCfShqSiXCh9rQA%40mail.gmail.com.

[Iconoclasta Vesgo]

There's a mistake on my code, i guess... i'm trying now to go for the browser and using canvas and the slice of the get-pixels output isn't right!! you need to remove every 4th occurence that corresponds to the alpha channel...

data = outputFrom-get-pixels.filter((el, i) => {

let j = i + 1

if(j % 4 !== 0 || i === 0){

return el

}

})

[John S]

Thx Tiago and Daniel.
Gonna try this later.
I can’t get around how to present the data to the model.
I used canvas as you wrote but presenting the canvas object to fromPixels the canvas is 0x0?
If I instantiate the canvas object as canvas(244,224)
I get further but then the model complains it wants the a tensor as
[-1, 224,224,3] but is getting either [224,224,3] of [1,224,224,3]
Can’t get my hear around this -1 at the beginning.any clues?

[Daniel Smilkov]

John S, can you share some relevant code to help us debug? One thing comes to mind: after fromPixels() returns a [224, 224, 3], reshape it as [1, 224, 224, 3] since your model might expect the first dimension to the batch size.

Daniel

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/8e8088ef-da02-4dc0-9a98-04c5e554230d%40tensorflow.org.

[John S]

Sure. I’ll will when I get home. I’ve tried with resize but i cannot make to -1 as the model wants. Only 1 and that will cause a complain.

[Tiago Vasconcelos]

the -1 means whatever size you have... 

let's say you process 5 images. each image is of shape, [224, 224, 3] , or 224px X 224px X 3 channels (RGB)

When you feed to the model the 5 pics it's an array with 5 pics or a tensor, 4d tensor if i got it right, so [5, 224, 224, 3] or if you don't know how many pics you have, [-1, 224, 224, 3]

I output from get pixels as array, and push to another array, so you have an array of arrays, right?? I then output that as X = tf.tensor2d[ArrayofArrays], (let's say an array that holds 5 images). Then when i want to feed into the model, i reshape that as X.reshape([-1, 224, 224, 3])

I hope it's a bit of help to understand, i'm learning also!!

[John S]

Right. 

I tried it with getPixel but no luck, mind you Im doing a prediction here. The code right now is with canvas approach and using fromPixels()

Error :  Error: The shape of dict['Placeholder'] provided in model.execute(dict) must be [-1,299,299,3], but was [1,299,299,3]

Code:

'use strict';

const tf = require('@tensorflow/tfjs')

require('@tensorflow/tfjs-node')

const canvass = require('canvas')

//tf.setBackend('tensorflow')

const {loadFrozenModel} =require( '@tensorflow/tfjs-converter');

const fs=require("fs");

var fetch = require('node-fetch');

global.fetch = fetch;

async function tfmodel(){

const MODEL_URL = 'http://127.0.0.1:8082/tensorflowjs_model.pb';

const WEIGHTS_URL = 'http://127.0.0.1:8082/weights_manifest.json';

const model = await loadFrozenModel(MODEL_URL, WEIGHTS_URL);

//const image = await getPixel('./test_7.jpg');

const imgElement = await parse_canvas('./test_7.jpg');

console.log(imgElement)

// tf.fromPixels() returns a Tensor from an image element.

const img = tf.fromPixels(imgElement).toFloat();

const offset = tf.scalar(127.5);

// Normalize the image from [0, 255] to [-1, 1].

const normalized = img.sub(offset).div(offset);

const final = normalized.reshape([-1, 299, 299, 3]);

model.execute({Placeholder: final});

return model;

};

async function getImage(arr) {

let img_size = 299 * 299 * 3

let pixels = arr.slice(0, img_size)

if(pixels.length !== 12288) //this is just a sanity check

console.log(pixels.length)

return pixels.map(cur => cur / 255) //normalize pixel data to 0/1

}

function getPixel(file) {

return new Promise(resolve => {

let getPixels = require("get-pixels")

getPixels(file, function(err, pixels) {

if(err) {

console.log("Bad image path")

return

}

console.log(pixels.shape)

return resolve(getImage(Array.from(pixels.data)))

})

});

}

const newmodel=tfmodel();

function parse(file) {

let data = fs.readFileSync(file)

return Array.from(data)

}

async function parse_canvas(file) {

const canvImg = new canvass(299,299,3);

const img = fs.readFileSync(file);

canvImg.src = img;

return canvImg;

}

[Ping Yu]

The error you got is confusing, because the FrozenModel should support dynamic shape placeholder.

 can ayou dd this line before calling the predict method:

console.log(model.inputs[0]);

and paste the output here, thanks!

Ping

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/28bdf76e-a561-46c4-8978-6154dc69f9e6%40tensorflow.org.

[John S]

Here:

 { name: 'Placeholder',
  shape:
   [ Long { low: -1, high: -1, unsigned: false },
     Long { low: 299, high: 0, unsigned: false },
     Long { low: 299, high: 0, unsigned: false },
     Long { low: 3, high: 0, unsigned: false } ],
  dtype: 'float32' }

[John S]

One other thing:

The few examples of using canvas in node they use 

const image = new canvas.Image

That will generate an error  

Error: When running in node, pixels must be an HTMLCanvasElement like the one returned by the `canvas` npm package

So Im using as the code above, = new canvas(299,299), to get further but I wonder if its the right way?

[John S]

I have pretty much replicated this demo
https://github.com/tensorflow/tfjs-converter/tree/master/demo/mobilenet
Used the same method and even the model. So the problem has to be before fromPixels creates a tensor

And it still gives me the same shape error [-1,224,224,3] and gets

I narrow it down to something with the canvas into fromPixels()

Can anyone try to replicate the error with my code or better share a working example using canvas and fromPixels in node?

[Ping Yu]

I see, the shape of the model is in Long, which is why is failing the check (which are checking against javascript number).

I will add a fix to the converter, meanwhile, you can try to bypass using following code before the predict call:

model.input[0].shape = [-1, 299, 299, 3];

Let me know if that works for you, thanks!

Ping

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/6b228e19-44ca-4591-bc67-a4810f93b42a%40tensorflow.org.

[Ping Yu]

Please ignore my previous suggestion, it won't work, but I have already put up a fix PR. thanks.

Ping

[John S]

Allright Thx. Will keep an eye. Untill then we have a Python solution.