AutoColorCorrelogram indexing not working with PNG format

[Myrtha Fuog‏]

Hello,

Based of the simpleapplication which is provided on GitHub, I wanted to use the basic global features.

However, I noticed that indexing with an AutoColorCorrelogram Extractor would occasionally throw:

Exception in thread "Consumer-01" java.lang.ArrayIndexOutOfBoundsException: 3

at java.awt.image.ComponentSampleModel.getPixel(ComponentSampleModel.java:750)

at java.awt.image.Raster.getPixel(Raster.java:1519)

at net.semanticmetadata.lire.imageanalysis.features.global.AutoColorCorrelogram.hsvImage(AutoColorCorrelogram.java:230)

at net.semanticmetadata.lire.imageanalysis.features.global.AutoColorCorrelogram.extract(AutoColorCorrelogram.java:242)

at net.semanticmetadata.lire.builders.GlobalDocumentBuilder.extractGlobalFeature(GlobalDocumentBuilder.java:192)

at net.semanticmetadata.lire.builders.GlobalDocumentBuilder.getGlobalDescriptorFields(GlobalDocumentBuilder.java:210)

at net.semanticmetadata.lire.builders.GlobalDocumentBuilder.createDescriptorFields(GlobalDocumentBuilder.java:258)

at net.semanticmetadata.lire.indexers.parallel.ParallelIndexer$Consumer.run(ParallelIndexer.java:1186)

at java.lang.Thread.run(Thread.java:745)

This only happens for PNG files, though. The reason for this seems to be the additional 4th band/channel a PNG (Red/Blue/Green/Alpha) has, where other formats such as JPGs have only 3 (Red/Blue/Green). It looks like AutoColorCorrelogram.hsvImage() does not consider this possibility, always declaring:

// converting to HSV:
int[] hsv = new int[3];
convertRgbToHsv(r.getPixel(x, y, pixel), hsv);

My question: How can I still use AutoColorCorrelogram for PNGs?

Thanks for considering

Myrtha

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

my code for indexing:

public class Indexer {
    public static void main(String[] args) throws IOException {
        ParallelIndexer indexer = new ParallelIndexer(4, "index", args[0]);
        //indexer.addExtractor(CEDD.class);
        //indexer.addExtractor(FCTH.class);
        indexer.addExtractor(AutoColorCorrelogram.class); //not working with png format (more than 3 bands/channels)
        indexer.run();
        System.out.println("Finished indexing.");

    }
}

[Mathias Lux‏]

Hi!

You can use the static method 

ImageUtils.createWorkingCopy(BufferedImage)

on your image before indexing it. 

https://github.com/dermotte/LIRE/blob/master/src/main/java/net/semanticmetadata/lire/utils/ImageUtils.java#L165

A simple way to do so would be:

public static void main(String[] args) {
    ParallelIndexer indexer = new ParallelIndexer(4, "index", "testdata");
    indexer.setImagePreprocessor(new ImagePreprocessor() {
        @Override
        public BufferedImage process(BufferedImage image) {
            return ImageUtils.createWorkingCopy(image);

        }
    });
    //indexer.addExtractor(CEDD.class);
    //indexer.addExtractor(FCTH.class);
    indexer.addExtractor(AutoColorCorrelogram.class); //not working with png format (more than 3 bands/channels)
    indexer.run();
    System.out.println("Finished indexing.");
}

cheers,

Mathias

--
Sie erhalten diese Nachricht, weil Sie in Google Groups E-Mails von der Gruppe "lire-dev" abonniert haben.
Wenn Sie sich von dieser Gruppe abmelden und keine E-Mails mehr von dieser Gruppe erhalten möchten, senden Sie eine E-Mail an lire-dev+u...@googlegroups.com.
Weitere Optionen finden Sie unter https://groups.google.com/d/optout.

--

Priv.-Doz. Dr. Dipl.-Ing. Mathias Lux

Associate Professor at Klagenfurt University, Austria

http://tinyurl.com/mlux-itec ... contact and cv

LIRE image search engine - http://www.lire-project.net

[Myrtha Fuog‏]

Thanks a lot, that worked like a charm!

[Myrtha Fuog‏]

Hey again,

I have spent quite some time with a a similar problem: When indexing with CvSURF or CvSIFT, some images cause the following exception to occur:

Exception in thread "Thread-2" java.lang.UnsupportedOperationException: Provided data element number (1702400) should be multiple of the Mat channels count (3)

at org.opencv.core.Mat.put(Mat.java:2549)

at net.semanticmetadata.lire.imageanalysis.features.local.opencvfeatures.CvSiftExtractor.extract(CvSiftExtractor.java:133)

at net.semanticmetadata.lire.builders.AbstractLocalDocumentBuilder.extractLocalFeatures(AbstractLocalDocumentBuilder.java:83)

at net.semanticmetadata.lire.indexers.parallel.ParallelIndexer$ExtractorForLocalSample.run(ParallelIndexer.java:1046)

at java.lang.Thread.run(Thread.java:745)

This only happens for certain images (some pngs and a grey JPG) and when opencv is involved. Is this a known problem? Is there maybe any other preprocessing that needs to be done to images of a dataset for LIRE to work properly in general? 

Any hints would be greatly appreciated,

Myrtha

--------------------

public class Indexer {

    public static void main(String[] args) throws IOException {

        ParallelIndexer indexer = new ParallelIndexer(4, "index", args[0], 128, 500);

        indexer.setImagePreprocessor(ImageUtils::createWorkingCopy);

        //local features

        indexer.addExtractor(CvSiftExtractor.class);

        indexer.addExtractor(CvSurfExtractor.class);

        indexer.run();

        System.out.println("Finished indexing.");

    }

}

[Sangwhan Moon‏]

Hello,

I haven't tried this myself, but from the back of my head and a guess based on the error it seems like

indexer.setImagePreprocessor(ImageUtils::get8BitRGBImage);

could work. Haven't tested it though.

Best regards,

> --
> Sie erhalten diese Nachricht, weil Sie in Google Groups E-Mails von der Gruppe "lire-dev" abonniert haben.

> Wenn Sie sich von dieser Gruppe abmelden und keine E-Mails mehr von dieser Gruppe erhalten möchten, senden Sie eine E-Mail an lire-dev+u...@googlegroups.com <mailto:lire-dev+u...@googlegroups.com>.
> Weitere Optionen finden Sie unter https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

[Myrtha Fuog‏]

Hello and thanks for the hint.

While trying this out I realized that the preprocessor set withindexer.setImagePreprocessor(someMethod) does not get executed at all, when theParallelIndexer is run(), at least for local feature indexing (still working fine for global ones).

I'm currently trying to figure out why. 

Is this by design or am I missing something here? 

[Mathias Lux‏]

Hi!

This feature was contributed, so I guess it could have happened that the preprocessor is just run for the global extractors, not the local ones. I can see what I can do (debug, verify, fix). Thanks for pointing it out!

Cheers,

Mathias

Wenn Sie sich von dieser Gruppe abmelden und keine E-Mails mehr von dieser Gruppe erhalten möchten, senden Sie eine E-Mail an lire-dev+u...@googlegroups.com.
Weitere Optionen finden Sie unter https://groups.google.com/d/optout.

[Myrtha Fuog‏]

For the record and if anyone else might encounter the same problem:

Despite what I wrote earlier, the preprocessor does get executed when using the ParallelIndexer. However, it looks like this only happens when images are being indexed, but not before, during the process of codebook generation. Since this happens first, exceptions related to images needing preprocessing are thrown during the codebook generation process.

My solution to this was to abandon the methods from ImageUtils altogether and do the preprocessing separately, using the imagemagick command line tools. 

(see http://stackoverflow.com/questions/34305379/how-to-convert-gray-scale-png-image-to-rgb-from-comand-line-using-image-magick  on how to do this)