Bag of Words on Android with JavaCV

[Filip Ligač]

Is there a way to implement a Bag of Words model using JavaCV? I can't
find import files for definition types like "BOWKMeansTrainer",
"BOWImgDescriptorExtractor" and so on.

Could anyone tell me if JavaCV even has these types?

Thanks,
Filip

[Samuel Audet]

Yes, it's here:
http://code.google.com/p/javacv/source/search?q=BOWImgDescriptorExtractor

[Filip Ligač]

Thanks for the answer. I got that working. Now I'm trying to use BoW
in ObjectFinder on Android so the matching could be quicker. But I
don't really know what to add to BoW vocabulary. Just imageDescriptors
from ObjectFinder class? And how can I convert it from FloatBuffer[]
to CvMat?

Thank you for the answer.

On 22. Apr, 06:05 h., Samuel Audet <samuel.au...@gmail.com> wrote:
> Yes, it's here:http://code.google.com/p/javacv/source/search?q=BOWImgDescriptorExtra...

[Filip Ligač]

I suppose there is no sample in JavaCV for this model, right?

[Samuel Audet]

On 2012-04-25 17:22, Filip Ligač wrote:
> I suppose there is no sample in JavaCV for this model, right?

Not that I know of, but it shouldn't be hard to translate a C/C++ sample
to Java..

As for wrapping a FloatBuffer in a CvMat, we should be able to do it
this way:
CvMat mat = CvMat.createHeader(width, height, CV_32F, channels);
mat.data_fl(new FloatPointer(floatBuffer));
And then any changes in the FloatBuffer should be reflected in the CvMat
as well.

Or do you need to merge /multiple/ FloatBuffer into /one/ CvMat? We can
do that this way:
CvMat mat = CvMat.create(width, height, CV_32F, channels);
FloatBuffer matfb = mat.getFloatBuffer();
for (FloatBuffer fb : myFloatBufferArray) {
matfb.put(fb);

[Filip Ligač]

Thanks for the answer, Samule.

I'm trying to write this BoW vocabulary initialization but
"descriptors" value is always null after extractor.compute() function.

Here is C++ sample:
// detecting keypoints
SurfFeatureDetector detector(400);
//FastFeatureDetector detector(1,true);
vector<KeyPoint> keypoints;

// computing descriptors
//Ptr<DescriptorExtractor > extractor(new
SurfDescriptorExtractor());// extractor;
Ptr<DescriptorExtractor > extractor(
new OpponentColorDescriptorExtractor(
Ptr<DescriptorExtractor>(new SurfDescriptorExtractor())
)
);
Mat descriptors;
Mat training_descriptors(1,extractor->descriptorSize(),extractor-
>descriptorType());
Mat img;

cout << "------- build vocabulary ---------\n";

cout << "extract descriptors.."<<endl;
//int count = 0;
Rect clipping_rect = Rect(0,120,640,480-120);
Mat bg_ = imread("background.png")(clipping_rect), img_fg;
while (dirp = readdir( dp ))
{
// count++;
filepath = dir + "/" + dirp->d_name;

// If the file is a directory (or is in some way invalid) we'll skip
it
if (stat( filepath.c_str(), &filestat )) continue;
if (S_ISDIR( filestat.st_mode )) continue;

img = imread(filepath);
if (!img.data) {
continue;
}
img = img(clipping_rect);
img_fg = img - bg_;
detector.detect(img_fg, keypoints);
// {
// Mat out; //img_fg.copyTo(out);
// drawKeypoints(img, keypoints, out, Scalar(255));
// imshow("fg",img_fg);
// imshow("keypoints", out);
// waitKey(0);
// }
extractor->compute(img, keypoints, descriptors);

training_descriptors.push_back(descriptors);
cout << ".";
}
cout << endl;
closedir( dp );

cout << "Total descriptors: " << training_descriptors.rows << endl;

FileStorage fs("training_descriptors.yml", FileStorage::WRITE);
fs << "training_descriptors" << training_descriptors;
fs.release();

BOWKMeansTrainer bowtrainer(1000); //num clusters
bowtrainer.add(training_descriptors);
cout << "cluster BOW features" << endl;
Mat vocabulary = bowtrainer.cluster();

FileStorage fs1("vocabulary_color_1000.yml", FileStorage::WRITE);
fs1 << "vocabulary" << vocabulary;
fs1.release();

[Filip Ligač]

My code just for vocabulary creation:

// detecting keypoints
KeyPoint keypoints = new KeyPoint();

DescriptorExtractorPtr descriptorExtractor = new
DescriptorExtractorPtr();
DescriptorMatcherPtr descriptorMatcher = new
DescriptorMatcherPtr();
BOWextractor = new BOWImgDescriptorExtractor(descriptorExtractor,
descriptorMatcher);

BOWinitialised = true;

// computing descriptors
CvMat image;

BOWKMeansTrainer BOWtrainer = new BOWKMeansTrainer(10); //num
clusters
CvMemStorage storage = CvMemStorage.create();

for (int i = 0; i < allFiles.length; i++)
{
// count++;
IplImage iplImage =
cvLoadImage(Environment.getExternalStorageDirectory() + "/
HockeyCardDetector/cards/" + allFiles[i]);

FeatureDetectorPtr detector = FeatureDetector.create("SURF");
DescriptorExtractorPtr extractor =
DescriptorExtractor.create("SURF");
detector.get().detect(iplImage, keypoints, null);
CvMat descriptors = new CvMat(null);
extractor.get().compute(iplImage, keypoints, descriptors);

BOWtrainer.add(descriptors);
}

CvMat vocabulary = BOWtrainer.cluster();

CvFileStorage fileStorage =
CvFileStorage.open(Environment.getExternalStorageDirectory() + "/
HockeyCardDetector/vocabulary.xml", storage, CV_STORAGE_WRITE);
cvWrite(fileStorage, "Vocabulary", vocabulary,
cvAttrList(null,null));
cvReleaseFileStorage(fileStorage);
cvClearMemStorage(storage);

[Filip Ligač]

If you knew a better way to speed up SURF for comparing images apart
from BoW, I'd be glad. SURF is very slow on Android when you have to
compare about a hundred images.

[Filip Ligač]

It crashes on this line...I don't know why, does CvMat have to be
initialized?

BOWtrainer.add(trainingDescriptors);
...
CvMat vocabulary = BOWtrainer.cluster();

[Samuel Audet]

On 2012-04-29 10:41, Filip Ligač wrote:
> It crashes on this line...I don't know why, does CvMat have to be
> initialized?
>
> BOWtrainer.add(trainingDescriptors);
> ...
> CvMat vocabulary = BOWtrainer.cluster();

I don't see "trainingDescriptors" anywhere in your source code... but
looking at the C++ code, it looks like it should be filled with
"descriptors".

Samuel

[Filip Ligač]

Yeah, you're right I just modified the source code a little but still
don't know what might be the problem.

[Filip Ligač]

I mean it crashes on this line exactly. It looks like CvMat should be
initialized or something.

CvMat vocabulary = BOWtrainer.cluster();

[Filip Ligač]

If I add SURF descriptors to BOWtrainer in iteration through every
image, vocabulary is null and it crashes on cvWrite. But there are
values in descriptors variable, I checked it with
descriptors.get(rows, cols).

BOWtrainer.add(descriptors);

...
CvMat vocabulary = BOWtrainer.cluster();

CvMemStorage storage = CvMemStorage.create();

CvFileStorage fileStorage =
CvFileStorage.open(Environment.getExternalStorageDirectory() + "/

HockeyCardDetector/vocabulary.xml", null, CV_STORAGE_WRITE, null);
cvWrite(fileStorage, "Vocabulary", vocabulary, cvAttrList());
cvReleaseFileStorage(fileStorage);
cvClearMemStorage(storage);

[Samuel Audet]

If I were you, I would start with that C++ sample that is known to work,
and do the same thing in Java with JavaCV, then that should work without
problems

[Filip Ligač]

That's what I'm trying to do but I'm stuck at creating BOW vocabulary.

Here's the original C++ code:

// detecting keypoints
SurfFeatureDetector detector(400);
//FastFeatureDetector detector(1,true);
vector<KeyPoint> keypoints;

// computing descriptors

Ptr<DescriptorExtractor > extractor(
new OpponentColorDescriptorExtractor(
Ptr<DescriptorExtractor>(new SurfDescriptorExtractor())
)
);

Mat descriptors;
Mat training_descriptors(1,extractor->descriptorSize(),extractor-
>descriptorType());
Mat img;

while (dirp = readdir( dp ))
{

filepath = dir + "/" + dirp->d_name;

img = imread(filepath);
if (!img.data) {
continue;
}

detector.detect(img, keypoints);

extractor->compute(img, keypoints, descriptors);

training_descriptors.push_back(descriptors);
}

closedir( dp );

BOWKMeansTrainer bowtrainer(1000); //num clusters
bowtrainer.add(training_descriptors);

Mat vocabulary = bowtrainer.cluster();

FileStorage fs1("vocabulary_color_1000.yml", FileStorage::WRITE);
fs1 << "vocabulary" << vocabulary;
fs1.release();

My code:
SurfFeatureDetector detector = new SurfFeatureDetector();
SurfDescriptorExtractor extractor = new SurfDescriptorExtractor();

CvMat trainingDescriptors = CvMat.create(1, allFiles.length);
CvTermCriteria terminateCriterion = new
CvTermCriteria(CV_TERMCRIT_ITER, 100, 0.001);
terminateCriterion.epsilon(FLT_EPSILON);
BOWKMeansTrainer BOWtrainer = new BOWKMeansTrainer(10,
terminateCriterion, 3, 2);

for (int i = 0; i < allFiles.length; i++)
{

IplImage iplImage =
cvLoadImage(Environment.getExternalStorageDirectory() + "/

HockeyCardDetector/cards2/" + allFiles[i]);
String path = Environment.getExternalStorageDirectory() + "/
HockeyCardDetector/cards2/" + allFiles[i];

// detecting keypoints
KeyPoint keypoints = new KeyPoint();

//detector.get().detect(iplImage, keypoints, null);
detector.detect(iplImage, keypoints, null);
// computing descriptors

CvMat descriptors = new CvMat(null);

extractor.compute(iplImage, keypoints, descriptors);

BOWtrainer.add(descriptors);

}

CvMat vocabulary = BOWtrainer.cluster();

CvMemStorage storage = CvMemStorage.create();
CvFileStorage fileStorage =
CvFileStorage.open(Environment.getExternalStorageDirectory() + "/
HockeyCardDetector/vocabulary.xml", null, CV_STORAGE_WRITE, null);
cvWrite(fileStorage, "Vocabulary", vocabulary, cvAttrList());
cvReleaseFileStorage(fileStorage);
cvClearMemStorage(storage);

But still I have no idea where the problem can be.

[Filip Ligač]

Is it possible, that BOWKmeansTrainer.cluster() method is wrong? Or is
it too big for the file?

Because if I do this, it's written to the file correctly with millions
of descriptors:

cvWrite(fileStorage, "Vocabulary", BOWtrainer.getDescriptors(0),
cvAttrList());

[Filip Ligač]

It's just really strange because BOWtrainer.cluster() returns null
everytime.

[Filip Ligač]

I'm trying to get this example working -
http://www.morethantechnical.com/2011/08/25/a-simple-object-classifier-with-bag-of-words-using-opencv-2-3-w-code/
. I would be really thankful if you could give me a hint how to
translate it to JavaCV.

[Samuel Audet]

On 2012-04-30 00:44, Filip Ligač wrote:
> That's what I'm trying to do but I'm stuck at creating BOW vocabulary.

Ok, so let's start with this:

> // computing descriptors
> Ptr<DescriptorExtractor> extractor(
> new OpponentColorDescriptorExtractor(
> Ptr<DescriptorExtractor>(new SurfDescriptorExtractor())
> )
> );

Your Java code doesn't do the same thing:

> SurfDescriptorExtractor extractor = new SurfDescriptorExtractor();

So, fix that and also fix other differences with the original C++ code,
and things should work!

Samuel

[Filip Ligač]

Ok, and could you help me with translating that one? I tried something
like this but I wasn't successful.

DescriptorExtractorPtr extractor = new
OpponentColorDescriptorExtractor = new DescriptorExtractorPtr = new
SurfDescriptorExtractor();

[Filip Ligač]

This compiles OK but the problem with BOWKMeansTrainer.cluster()
returning null still persists.

DescriptorExtractor extractor = new
OpponentColorDescriptorExtractor(new DescriptorExtractor(new
SurfDescriptorExtractor()));

[Samuel Audet]

I guess something like this should work:

DescriptorExtractor surfExtractor = new SurfDescriptorExtractor();
DescriptorExtractorPtr surfExtractorPtr = new
DescriptorExtractorPtr().put(surfExtractor);
DescriptorExtractor extractor = new
OpponentColorDescriptorExtractor(surfExtractorPtr);

Yes, we need to keep Java references or else they get deallocated

[Filip Ligač]

Thanks for this code but I don't think that is a problem. When I check
number of descriptors with BOWtrainer.descripotorsCount() after adding
all of them, there is a correct number there.

I can just think of the thing that I put descriptors right in
BOWtrainer with BOWtrainer.add(descriptors) and in original code
there's a special Mat training_descriptors where all descriptors are
being added in every itteration through all images and after this the
whole matrix is passed to bowtrainer. But I haven't found
CvMat.push_back method in JavaCV.

Mat training_descriptors(1,extractor->descriptorSize(),extractor-
>descriptorType());
...
training_descriptors.push_back(descriptors);
...

bowtrainer.add(training_descriptors);
Mat vocabulary = bowtrainer.cluster();

[Samuel Audet]

On 2012-04-30 20:39, Filip Ligač wrote:
> Thanks for this code but I don't think that is a problem. When I check
> number of descriptors with BOWtrainer.descripotorsCount() after adding
> all of them, there is a correct number there.
>
> I can just think of the thing that I put descriptors right in
> BOWtrainer with BOWtrainer.add(descriptors) and in original code
> there's a special Mat training_descriptors where all descriptors are
> being added in every itteration through all images and after this the
> whole matrix is passed to bowtrainer. But I haven't found
> CvMat.push_back method in JavaCV.

You're talking about this bit, right?

Mat descriptors;
Mat
training_descriptors(1,extractor->descriptorSize(),extractor->descriptorType());
Mat img;
while (dirp = readdir( dp )) {
filepath = dir + "/" + dirp->d_name;
img = imread(filepath);
if (!img.data) {
continue;
}
detector.detect(img, keypoints);
extractor->compute(img, keypoints, descriptors);
training_descriptors.push_back(descriptors);
}

Something like this should work:

CvMat[] descriptors = new CvMat[allFiles.length];
int descriptors_length = 0;

for (int i = 0; i < allFiles.length; i++) {

IplImage img = cvLoadImage(allFiles[i]);
if (img == null) {
continue;
}
KeyPoint keypoints = new KeyPoint(null);
detector.detect(img, keypoints, null);
descriptors[i] = new CvMat(null);
extractor.compute(img, keypoints, descriptors);
descriptors_length += descriptors[i].length();
}

CvMat training_descriptors = CvMat.create(1, descriptors_length,
extractor->descriptorType());
int i = 0;
for (CvMat m : descriptors) {
double[] d = m.get();
training_descriptors.put(i, d);
i += d.length;

[Filip Ligač]

Thanks once again for the code, it seems like it works as for
computing descriptors but then when I do this:

BOWKMeansTrainer BOWtrainer = new BOWKMeansTrainer(10);

BOWtrainer.add(trainingDescriptors);

CvMat vocabulary = BOWtrainer.cluster();

It doesn't even crash but when I debug the code, it disconnects during
assigning BOWtrainer.cluster() to vocabulary.

[Samuel Audet]

On 2012-04-30 22:51, Filip Ligač wrote:
> Thanks once again for the code, it seems like it works as for
> computing descriptors but then when I do this:
>
> BOWKMeansTrainer BOWtrainer = new BOWKMeansTrainer(10);
> BOWtrainer.add(trainingDescriptors);
>
> CvMat vocabulary = BOWtrainer.cluster();
>
> It doesn't even crash but when I debug the code, it disconnects during
> assigning BOWtrainer.cluster() to vocabulary.

Again, can you try to do the same thing that the C++ sample does?

[Filip Ligač]

The original code is:

BOWKMeansTrainer bowtrainer(1000); //num clusters
bowtrainer.add(training_descriptors);

cout << "cluster BOW features" << endl;

Mat vocabulary = bowtrainer.cluster();

FileStorage fs1("vocabulary_color_1000.yml", FileStorage::WRITE);
fs1 << "vocabulary" << vocabulary;
fs1.release();

So I think that this is pretty much the same:

BOWKMeansTrainer BOWtrainer = new BOWKMeansTrainer(10);

BOWtrainer.add(training_descriptors);

CvMat vocabulary = BOWtrainer.cluster();

CvMemStorage storage = CvMemStorage.create();
CvFileStorage fileStorage =
CvFileStorage.open(Environment.getExternalStorageDirectory() + "/

HockeyCardDetector/vocabulary2.xml", null, CV_STORAGE_WRITE, null);
cvWrite(fileStorage, "Vocabulary", BOWtrainer.cluster(),
cvAttrList());
cvReleaseFileStorage(fileStorage);
cvClearMemStorage(storage);

[Filip Ligač]

Except for the number of clusters but it does the same thing with
1000.

I hope that the way I initialized it is the right one.

[Filip Ligač]

Samuel, I really appreciate all the help you've come up with, but
could you give me some pseudo JavaCV code how I can create BOW
vocabulary to compare one image with the descriptors in the vocabulary
to get the one that is the most similar to the first image? I got it
working by modifying Object_Finder but on Android it was really slow
because of comparing SURF descriptors one after another so I was
thinking of this way.

As far as I can tell, it will take me ages to get it working. With a
little sample, it would be much better.

[Samuel Audet]

On 2012-05-01 00:55, Filip Ligač wrote:
> As far as I can tell, it will take me ages to get it working. With a
> little sample, it would be much better.

Yes, it would be much better, but as I mentioned before, there does not
appear to be any.

It's possible that there is a bug in JavaCV that prevents this from
working properly. OpenCV's use of C++ isn't exactly the best I've seen,
so if you could help me looking for bugs, I could help you with whatever
you want to do!

Or you could code some function in C++ that uses OpenCV's C++ API
directly, and compile it with JavaCPP to call it from Java...

Or you could try the Java wrapper that comes with android-opencv...

There's a lot of options here

Samuel

[nourab]

Hi , 

I have the same issue  , I want to developp an android app that extract bags of words for any captured image 

Can you help me ? 

[Samuel Audet]

Hello,

On 10/30/2013 05:25 PM, nourab wrote:
> I have the same issue , I want to developp an android app that extract
> bags of words for any captured image
>
> Can you help me ?

Do you have some sample C++ code that is close to what you'd like to do?

Samuel

[nourab]

this is the cpp code and I try to convert it to java 

[Samuel Audet]

On 11/13/2013 07:31 AM, nourab wrote:
> SIFT sift = new SIFT();
> featureDetector =sift.getFeatureDetector();
>
> descriptorExtractor=sift.getDescriptorExtractor();
>
> descriptorMatcher= new BFMatcher();
> System.out.println("end");

You should also try to move "sift" to a member variable. It looks like
it may be getting garbage collected...

Samuel