Instance segmentation with OpenCV

[Larienas]

Hello,

  I need to extract the people images using the technique mentioned in the link "https://www.pyimagesearch.com/2018/11/26/instance-segmentation-with-opencv/"

  I was wondering can this be done in java as well?

With Regards,

Suresh Jeyachandran

[Samuel Audet]

Sure, of course. The Python API of OpenCV is a subset of its C++ API.

[larien...@gmail.com]

Hello Samuel,

I understand, that we can get an Example in Python, Is there an example for this in Java?. Could you please point me to that?

With Regards,
Lariena

[larien...@gmail.com]

Hello Samuel,

  I understand, that  we can get an example in Python, Is there an example for this in Java?. Could you please point me to that?. It would be great to know how to do operations in JAVA as shown below python code.

net = cv2.dnn.readNetFromTensorflow(weightsPath, configPath)

# construct a blob from the input image and then perform a forward

# pass of the Mask R-CNN, giving us (1) the bounding box  coordinates

# of the objects in the image along with (2) the pixel-wise segmentation

# for each specific object

blob = cv2.dnn.blobFromImage(image, swapRB=True, crop=False)

[larien...@gmail.com]

Hello Samuel,

 

  I found the API readNetFromTensorflow when I tried to use, it is throwing an exception as

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.opencv.dnn.Dnn.readNetFromTensorflow_1(Ljava/lang/String;)J

       at org.opencv.dnn.Dnn.readNetFromTensorflow_1(Native Method)

       at org.opencv.dnn.Dnn.readNetFromTensorflow(Dnn.java:654)

 

Can you help me how to correct it?

 

With Regards,

Suresh Jeyachandran

[Samuel Audet]

There's a couple of working samples with readNetFromCaffe() here:
https://github.com/bytedeco/javacv/blob/master/samples/CaffeGooglenet.java
https://github.com/bytedeco/javacv/blob/master/samples/DeepLearningFaceDetection.java

Jan-Philipp Buck says it works just fine with readNetFromONNX().

And Florian Bruggisser is using readNetFromDarkNet without any problems:
https://github.com/cansik/deep-vision-processing/blob/master/src/main/java/ch/bildspur/vision/YOLONetwork.java

So I'm pretty sure readNetFromTensorflow() works as well. Please provide
the code and the data necessary to reproduce this error.

On 4/21/20 6:22 AM, larien...@gmail.com wrote:
> Hello Samuel,
>
>
>
> I found the API readNetFromTensorflow when I tried to use, it is throwing an exception as
>
> Exception in thread "main" java.lang.UnsatisfiedLinkError: org.opencv.dnn.Dnn.readNetFromTensorflow_1(Ljava/lang/String;)J
>
> at org.opencv.dnn.Dnn.readNetFromTensorflow_1(Native Method)
>
> at org.opencv.dnn.Dnn.readNetFromTensorflow(Dnn.java:654)
>
>
>
> Can you help me how to correct it?
>
>
>
> With Regards,
>
> Suresh Jeyachandran
>
>
>
> From: larien...@gmail.com <larien...@gmail.com>
> Sent: lundi 20 avril 2020 22:57
> To: 'Samuel Audet' <samuel...@gmail.com>; jav...@googlegroups.com
> Subject: RE: [javacv] Instance segmentation with OpenCV
>
>
>
> Hello Samuel,
>
> I understand, that we can get an example in Python, Is there an example for this in Java?. Could you please point me to that?. It would be great to know how to do operations in JAVA as shown below python code.
>
> net = cv2.dnn.readNetFromTensorflow(weightsPath, configPath)
>
> # construct a blob from the input image and then perform a forward
>
> # pass of the Mask R-CNN, giving us (1) the bounding box coordinates
>
> # of the objects in the image along with (2) the pixel-wise segmentation
>
> # for each specific object
>
> blob = cv2.dnn.blobFromImage(image, swapRB=True, crop=False)
>
> With Regards,
>
> Lariena
>
> -----Original Message-----
>

> From: Samuel Audet <samuel...@gmail.com <mailto:samuel...@gmail.com> >
>
> Sent: lundi 20 avril 2020 03:12
>

> To: jav...@googlegroups.com <mailto:jav...@googlegroups.com> ; Larienas <larien...@gmail.com <mailto:larien...@gmail.com> >

[larien...@gmail.com]

Hello Florain,

 

  Thank you so much for introducing me to your work. I don’t know how to thank you.

 

  I tried a small example but I could not make it to work. I have pasted the my code below.

 

import static org.bytedeco.opencv.global.opencv_imgcodecs.IMREAD_COLOR;

 

import java.io.File;

import java.io.IOException;

import java.nio.file.Path;

import java.nio.file.Paths;

 

import org.bytedeco.opencv.opencv_core.Mat;

 

import ch.bildspur.vision.MaskRCNN;

import ch.bildspur.vision.result.ObjectDetectionResult;

import ch.bildspur.vision.result.ResultList;

 

public class MaskRCNNTest {

 

       public static void main(String[] args) {

             String configPath = "C:\\Users\\sures\\OpenCV\\mask_rcnn_inception_v2_coco_2018_01_28.pbtxt";

             String modelPath = "C:\\Users\\sures\\OpenCV\\mask_rcnn_inception_v2_coco_2018_01_28.pb";

             Path config = Paths.get(configPath);

             Path model = Paths.get(modelPath);

             MaskRCNN maskrcnn = new MaskRCNN(config, model);

             maskrcnn.setup();

             try {

                    Mat image = load(new File("data/group.jpg"), IMREAD_COLOR);

                    ResultList<ObjectDetectionResult> detections = maskrcnn.run(image);

                    if (detections == null) {

                           return;

                    }

                    float confidenceSum = 0;

                    for (ObjectDetectionResult detection : detections) {

                           System.out.println(detection.getClassName() + "\t[" + detection.getConfidence() + "]");

                           confidenceSum += detection.getConfidence();

                    }

 

                    System.out.println(

                                 "found " + detections.size() + " objects. avg conf: " + (confidenceSum / detections.size()));

             } catch (IOException e) {

                    e.printStackTrace();

             }

       }

 

}

 

Could you please let me know what correction I need to make?

 

With Regards,

Lariena

 

From: flo...@nexpose.ch <flo...@nexpose.ch>
Sent: mardi 21 avril 2020 09:25
To: larien...@gmail.com; jav...@googlegroups.com; Samuel Audet <samuel...@gmail.com>
Cc: Jan-Philipp Buck <coole....@gmail.com>
Subject: Re: [javacv] Instance segmentation with OpenCV

 

Loading the Tensorflow MaskRCNN files works, I just tested it here:

https://github.com/cansik/deep-vision-processing/blob/master/src/main/java/ch/bildspur/vision/MaskRCNN.java#L25

 

I used the following models: 

https://github.com/cansik/deep-vision-processing/releases/download/repository/mask_rcnn_inception_v2_coco_2018_01_28.pb

https://github.com/cansik/deep-vision-processing/releases/download/repository/mask_rcnn_inception_v2_coco_2018_01_28.pbtxt

 

Please be aware that for loading tensorflow models into opencv dnn, you have to create a new config file which is explained here:

https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API#generate-a-config-file

 

For SSD and R-CNN networks, there are even some pre-built.

 

best regards

Florian

[Samuel Audet]

You're not showing how you get the row values. How are you getting the values of the rows?

On 4/23/20 2:34 AM, larien...@gmail.com wrote:

Hello Florian and Samuel,

 

  I took inspiration looking what Florian have implemented and started to implement MaskRCNN but I am facing a problem hope you can help.

 

import static org.bytedeco.opencv.global.opencv_core.minMaxLoc;

import static org.bytedeco.opencv.global.opencv_dnn.blobFromImage;

import static org.bytedeco.opencv.global.opencv_dnn.readNetFromTensorflow;

import static org.bytedeco.opencv.global.opencv_imgcodecs.IMREAD_COLOR;

 

import java.io.File;

import java.io.IOException;

import java.nio.file.Path;

import java.nio.file.Paths;

 

import org.bytedeco.javacpp.DoublePointer;

import org.bytedeco.javacpp.FloatPointer;

import org.bytedeco.opencv.opencv_core.Mat;

import org.bytedeco.opencv.opencv_core.MatVector;

import org.bytedeco.opencv.opencv_core.Point;

import org.bytedeco.opencv.opencv_core.Rect;

import org.bytedeco.opencv.opencv_core.RectVector;

import org.bytedeco.opencv.opencv_core.StringVector;

import org.bytedeco.opencv.opencv_dnn.Net;

import org.bytedeco.opencv.opencv_text.FloatVector;

import org.bytedeco.opencv.opencv_text.IntVector;

 

import ch.bildspur.vision.result.ObjectDetectionResult;

import ch.bildspur.vision.result.ResultList;

 

public class MaskRCNNTest {

 

       public static void main(String[] args) throws IOException {

 

             String configPath = "mask_rcnn_inception_v2_coco_2018_01_28.pbtxt";

             String modelPath = "frozen_inference_graph.pb";

             Path config = Paths.get(configPath);

             Path model = Paths.get(modelPath);

 

             Net net = readNetFromTensorflow(model.toAbsolutePath().toString(), config.toAbsolutePath().toString());

 

             if (net.empty()) {

                    System.out.println("Can't load network!");

                    return;

             }

 

             Mat image = load(new File("example_01.jpg"), IMREAD_COLOR);

 

             Mat inputBlob = blobFromImage(image);

 

             // set input

             net.setInput(inputBlob);

 

             // create output layers

             StringVector outNames = new StringVector("detection_out_final", "detection_masks");

             MatVector outs = new MatVector(outNames.size());

 

             // run detection

             net.forward(outs, outNames);

 

             // evaluate result

             postprocess(image, outs);

       }

 

       private static void postprocess(Mat frame, MatVector outs) {

             IntVector classIds = new IntVector();

             FloatVector confidences = new FloatVector();

             RectVector boxes = new RectVector();

 

             for (int i = 0; i < outs.size(); ++i) {

                    Mat result = outs.get(i);

                    for (int j = 0; j < result.rows(); j++) {

 

                    }

              }

       }

 

In postprocess function when I get the result “Mat result = outs.get(i)" the row value is always -1. Am I doing anything wrong?

Can you please guide me?

 

With Regards,

Lariena

 

From: flo...@nexpose.ch <flo...@nexpose.ch>
Sent: mercredi 22 avril 2020 10:17
To: 'Samuel Audet' <samuel...@gmail.com>; larien...@gmail.com
Cc: 'Jan-Philipp Buck' <coole....@gmail.com>
Subject: RE: [javacv] Instance segmentation with OpenCV

 

Hi Lariena

 

I think you misunderstood my answer, it was just a proof that loading the model works. As you see in the source code, I have not yet implemented the algorithm and the result fetching.

 

Best regards

Florian

[Samuel Audet]

Awesome, that looks great! No need to port all examples, just a few
would be good. Thanks a lot!!

Samuel

On 5/5/20 4:46 AM, flo...@bildspur.ch wrote:
> Dear All
>
> I found a bit of time today to implement a working version. Please have a look here as a reference in JavaCV:
>
> https://github.com/cansik/deep-vision-processing/blob/master/src/main/java/ch/bildspur/vision/MaskRCNN.java
>
>
>
> @Samuel: I am still looking for time to port all my examples to the javacv examples
>
> Best regards
> Florian

[larien...@gmail.com]

Hi Florian,

 

  Thanks a lot 😊  Much appreciated.

 

   How do I extract just the detect images. I am trying to

 

   Mat cloneImage = originalImg.clone();

 Rect roi = new Rect(x, y, width, height);

 Mat cropped = new Mat(cloneImage, roi);

                                            

 //Mask

 Mat mask = detection.getMask();

 

I need to “bitwise and”  the mask and region of interest something like “cv2.bitwise_and(roi, roi, mask)”. I don’t know how to do it with Mat of type “org.bytedeco.opencv.opencv_core”.

 

Any guidance on this?

 

With Regards,

Lariena

 

From: flo...@bildspur.ch <flo...@bildspur.ch>
Sent: lundi 4 mai 2020 21:47
To: larien...@gmail.com; Samuel Audet <samuel...@gmail.com>
Cc: 'Jan-Philipp Buck' <coole....@gmail.com>; javacv <jav...@googlegroups.com>
Subject: Re: [javacv] Instance segmentation with OpenCV

 

Dear All

I found a bit of time today to implement a working version. Please have a look here as a reference in JavaCV:

https://github.com/cansik/deep-vision-processing/blob/master/src/main/java/ch/bildspur/vision/MaskRCNN.java

@Samuel: I am still looking for time to port all my examples to the javacv examples

Best regards
Florian

 

__________

 

Florian Bruggisser

Art Director

bildspur.ch | Hohlstrasse 256 CH-8004 Zürich

[Samuel Audet]

If you're looking for a way to apply an ROI, that would be with Mat.apply():
http://bytedeco.org/javacpp-presets/opencv/apidocs/org/bytedeco/opencv/opencv_core/Mat.html#apply-org.bytedeco.opencv.opencv_core.Rect-

On 5/12/20 6:56 AM, larien...@gmail.com wrote:
> Hi Florian,
>
>   Thanks a lot 😊  Much appreciated.
>
>    How do I extract just the detect images. I am trying to
>
>    Mat cloneImage = originalImg.clone();
>  Rect roi = new Rect(x, y, width, height);
>  Mat cropped = new Mat(cloneImage, roi);
>
>  //Mask

>  Mat mask = detection.getMask();/*

[larien...@gmail.com]

Hi Samuel,

I am able to get the ROI as before, What I am looking is to get the "bitwise and" of ROI and the mask. Something like "cv2.bitwise_and(roi, roi, mask)" for Mat type “org.bytedeco.opencv.opencv_core”.

With Regards,
Lariena


-----Original Message-----
From: Samuel Audet <samuel...@gmail.com>
Sent: mardi 12 mai 2020 02:19
To: larien...@gmail.com
Cc: flo...@bildspur.ch; 'Jan-Philipp Buck' <coole....@gmail.com>; 'javacv' <jav...@googlegroups.com>
Subject: Re: [javacv] Instance segmentation with OpenCV

[Samuel Audet]

There is that too in the C++ API:
http://bytedeco.org/javacpp-presets/opencv/apidocs/org/bytedeco/opencv/global/opencv_core.html#bitwise_and-org.bytedeco.opencv.opencv_core.Mat-org.bytedeco.opencv.opencv_core.Mat-org.bytedeco.opencv.opencv_core.Mat-org.bytedeco.opencv.opencv_core.Mat-

[larien...@gmail.com]

Hi Samuel and Florian,

My goal is just extract all the images detected by MaskRCNN from the original image.

//Clone the original image
Mat cloneImage = originalImg.clone();
// Extract the ROI

Rect roi = new Rect(x, y, width, height);

Mat cropped = cloneImage.apply(roi);
//get the detected Mask from MaskRCNN
Mat mask = detection.getMask();
// Convert them to Binary
threshold(mask, mask, 150, 255, THRESH_BINARY);

//Bitwise And to get the detected object
Mat result = new Mat(mask.rows(), mask.cols(), cropped.type());
bitwise_and(cropped, mask, result);

I get the error shown below. The cropped and mask Mat Size are same but the array size are different
Exception in thread "main" java.lang.RuntimeException: OpenCV(4.1.2) C:\projects\javacpp-presets\opencv\cppbuild\windows-x86_64\opencv-4.1.2\modules\core\src\arithm.cpp:229: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and type), nor 'array op scalar', nor 'scalar op array' in function 'cv::binary_op'

at org.bytedeco.opencv.global.opencv_core.bitwise_and(Native Method)
at com.opencv.maven.MaskRCNN.App.main(App.java:101)

Any guidance on this?

With Regards,
Lariena

[Larienas]

Hi Samuel and Florian,

  I fixed the issue by looking into https://answers.opencv.org/question/82671/bitwise_and-giving-sizes-of-input-arguments-do-not-match/

with Regards,

Lariena

[Samuel Audet]

Yes, we have to make sure all images/rois have the same size.