How to train Imagenet dataset with convolutional_network . HDF5 or CSV ?

[techm...@gmail.com]

1. I want to train my own dataset from Imagenet by using convolutional_network.
It is suggested to use the HDF5 datasets,but I don't know how to convert the images to HDF5 file.
Maybe CSV file is simple enough to understand ,but how color image data is stored in CSV?Including the class of sample(label),and its RGB is in 2D-space .

2. Is it possible by just changing one line in the conv.yaml file in "pylearn2/scripts/tutorials/convolutional_network" to load the new dataset.

!obj:pylearn2.train.Train {
dataset: &train !obj:pylearn2.datasets.mnist.MNIST {
which_set: 'train',
start: 0,
stop: %(train_stop)i
},

This is from the conv.yaml file , I think only I need to change is the "dataset: &train !obj:pylearn2.datasets.mnist.MNIST "line ,is it right ?

[Nicu Tofan]

I believe that the amount of work will be a bit larger.

As far as I can tell (grep -r -i imagenet) pylearn2 has no special support for ImageNet.

You are correct to assume that you would have to load a different dataset. But not only that line.

YAML files as used in Pylearn2 are simply constructors. The line that you pointed at simply means

from pylearn2.datasets.mnist import MNIST

dataset = MNIST(which_set='train', start=0, stop=train_stop)

That is, it creates an instance of the MINST class with those arguments.

For ImageNet you would have to create a class (maybe based on HDF5Dataset as suggested) that is capable of storing/handling the images. Maybe there are already some implementations from other users that I just don't know about.

Self promoting starts

Incidentally I'm working with ImageNet right now, so I have some resources to share. Unfortunate part is that they are very volatile at the moment and quite unpolished:

- a script for downloading images that are part of the ImageNet (you don't have to worry about this if you are a researcher

- a generic dataset for images of various shapes stored on the disc and a csv file; has utilities (called adjusters) for making the images the same size, among others

- this script for making the images square

Self promoting ends

--
You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[guangyu guo]

Thank you.

Have you ever succeeded to train your own image data by convolutional_network?

Do not have to be images from Imagenet,but it has to be your won ,stored in PC.
Of course  RGB color image, not grayscale , and the input has to be come from Conv2DSpace.

--
You received this message because you are subscribed to a topic in the Google Groups "pylearn-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylearn-users/MfWHpQU_rtA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylearn-user...@googlegroups.com.

[Nicu Tofan]

Yes, that's what I'm doing.

And that is what those resources help you do.

[guangyu guo]

Thank you for sharing.

Is there a simple way to train my won image data with Pylearn2?

It's difficult for me to create a class about image data.

I have red your self promoting.It's owesome to download images automatically.

[Nicu Tofan]

I would be inclined to say no, there is not, if you don't know have a pretty good understanding of how it works.

Even if my ImgDataset would be production-ready, there are still a number of things that you need to be aware (like partial_sum) in order to do something decent (i.e. not 32x32 pixels) with limited resources (K520 on Amazon g2.2xlarge, for example). Once I get some results I will publish a page somewhere describing the solution, but I don't expected to be short.

The reason that is often invoked is that the library is a machine learning research library

Other may prove me wrong, this is just my experience (and I'm not a researcher).

[techm...@gmail.com]

在 2015年6月15日星期一 UTC+9下午7:40:45，Nicu Tofan写道：

I will try to convert image data to a csv file and run it by Pylearn2.
There is already scripts to convert grayscale images to a csv file and train on Pylearn2.But I dont know if color image is OK?

The structure of a CSV file is like this:
each row represent for one sample
the first column represent for the label,and the rest of the row represent for the pixel.So it converted 2D-space pixels to a vector and read it in Conv2DSpace.
like this:
0,255,255,255,255,233,233,222,222,・・・,222
0,255,155,255,125,233,233,222,212,・・・,212
0,215,155,255,255,233,233,222,222,・・・,122
1,23,111,223,111,45,23,23,12,・・・11
1,23,11,23,111,45,23,23,12,・・・11
1,123,111,223,121,45,23,23,122,・・・11
・
・
・

So,if the images are RGB color,is it possible to whrite 3 channels of pixels in one line(maybe the order is R G B) and read it by Conv2DSpace and set the channel to 3?

[Nicu Tofan]

Sorry for the late reply.

I just want to beg you not to write pixels values in a .csv file. C'mon.

I wrote this class for this kind of usage. It was purposefully kept simple, without dependencies in pyl2extra, so you can just copy the whole file into your project and forget the whole pyl2extra thing.

The usage is very simple: have a csv file ready with targets op first column and file names on the second.

Example of a dataset used with a regression problem:

8.123,file1.png
2.123,file2.png

Example of a dataset used with a classification problem:

0,file1.png
1,file2.png

You have to assign textual classes yourself (0 - dog, 1 - cat), the class only deals with integers.

If you're loading the dataset for predictions, just keep the first row empty:

Example of regression:

,file1.png
,file2.png

Example of classification:

,file1.png
,file2.png

What the class does for you? The class is basically a DenseDesignMatrix and you should treat it like so. It only helps you read the images, scales them to have same size and converts them to square. The images are then converted to a single giant numpy array of shape 'b, 0, 1, 'c' - that is examples, rows, columns, channels.

yaml example:

dataset: &trndataset !obj:pyl2extra.datasets.images.Images {
    source: 'train.csv',
    image_size: 128
}

However, note that - for a class to be available inside yaml - it must be importable. So add a setup script modeled after this one, add the relevant path to PYTHONPATH or you're stuck on using Python code.

In a more general sense I believe this may be a way to introduce newcomers to the pylearn2 library so I'm also writing a tutorial based on it. This class leans on the simplicity side and I will probably write one that is more flexible (header, column names, extract a range of rows).

[guangyu guo]

I have already written all the pixels and labels of image data into a csv file and converted it to a pkl file.└(^へ^)┘

[guangyu guo]

I want to use your images.py.It's very easy to understand.

And here is a question:is it possible to include detail file path in the second column?

just like:

0,/home/username/image_data_path/file1.png

1,/home/username/image_data_path/file2.png

[Nicu Tofan]

Absolutely. Relative paths are relative to current working directory. Absolute paths are interpreted directly.

[guangyu guo]

ValueError occured ,I don know why:

ValueError: VectorSpace(dim=1, dtype=float64) with total dimension 1 can't format a batch into VectorSpace(dim=10, dtype=float64)because its total dimension is 10

help  please

[techm...@gmail.com]

Now I know it's about the batch_size,n_classes and Dataset_size
I have 8 classes(from 0 to 7),2590 of dataset which can be divided by the batch size 10.But the error still occures

ValueError: VectorSpace(dim=1, dtype=float64) with total dimension 1 can't format a batch into VectorSpace(dim=8, dtype=float64)because its total dimension is 8

[Nicu Tofan]

The tests for classification and regression are now passing.

The problem should be solved.

If you still have problems take note of the binary_target_dim of the Softmax.

Softmax(max_col_norm=1.9365, layer_name='y',

    binary_target_dim=1, n_classes=NUMBER_OF_CLASSES)

[guangyu guo]

Thank you very much.

I will try that later and report the result.

--
You received this message because you are subscribed to a topic in the Google Groups "pylearn-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylearn-users/MfWHpQU_rtA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylearn-user...@googlegroups.com.

[guangyu guo]

I need your help again.

I set the binary_target_dim : 8

But the error still occures :

ValueError: VectorSpace(dim=1, dtype=float64) with total dimension 1 can't format a batch into IndexSpace(dim=8, max_labels=8, dtype=int64)because its total dimension is 8

How can I change the dim of VectorSpace ?

--
You received this message because you are subscribed to a topic in the Google Groups "pylearn-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylearn-users/MfWHpQU_rtA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylearn-user...@googlegroups.com.

[Nicu Tofan]

That's just it: you don't set that to 8, you (almost always) set it to 1 because there is only one "kind" of class.

You set n_classes to 8.

With binary_target_dim>1, you can use one layer to simultaneously predict a bag of words

[guangyu guo]

I set the binary_target_dim to 1,the softmax part of conv.yaml is like this:

      Softmax(max_col_norm=1.9365, layer_name='y',

             binary_target_dim=1, n_classes=8,istdev: .05)

But it still need something to fix:

      NotImplementedError:VectorSpace(dim=1,dtype=float64)does't know how to format as IndexSpace(dim=1,max_labels=8,dtype=int64)

Thank you very much.

[guangyu guo]

I still can not figure it out.I was trying to write a cnn program by myself but failed.
Can you write a detaile instruction about how to modifie the conv.yaml file Please?

First,I ignored the 'algorithm.monitoring_dataset' ,  'termination'  and  'extensions'  part.

Then changed Conv2DSpace shape to [256,256] and channels to 3.

Third,in the 'models.mlp.Softmax' part,changed the n_class and set the binary_target_dim to 1.

Then the error occured :

NotImplementedError: VectorSpace(dim=1, dtype=float64) doesn't know how to format as IndexSpace(dim=1, max_labels=8, dtype=int64)

Is it about the float or int or max_labels?

Thankyou~~~