Issues with bottom vision

[John Lettang]

Hi all,

I have my pick and place working with the top camera alone now. However I am having some issues with bottom vision.

To start, I am not able to do the advanced bottom camera calibration. I am getting the error of too many misdetects and am being told to look at my fiducial/nozzel tip detection however both of those seem to be working fine. 

And of course that issue with the bottom camera calibration may be the issue I have with bottom vision where I am finding that I cannot find a good threshold and way to isolate the part with my current set up. I was looking through the instructions for both of these on the wiki but I couldn't quite figure it out. Namely the wiki talked about "probing" pixels you wanted and didn't want but I couldn't figure out how to do this and didn't see a dedicated wiki page to bottom vision.

Attached below should be some helpful screenshots and code that outline my issues.

I appreciate the help and let me know if there are any additional log files or anything that would be useful.

-John

<cv-pipeline>
   <stages>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ParameterNumeric" name="pThreshold" enabled="true" parameter-label="Threshold" parameter-description="Set the brightness threshold that isolates the shiny contacts of a part." stage-name="threshold" property-name="valueMax" effect-stage-name="threshold" preview-result="true" minimum-value="1.0" maximum-value="254.0" default-value="100.0" numeric-type="Integer"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ParameterNumeric" name="pDetail" enabled="true" parameter-label="Min. Detail Size" parameter-description="Minimal size of a detail that should be included in the detected shape." stage-name="filterContours" property-name="minArea" effect-stage-name="contours" preview-result="true" minimum-value="0.0" maximum-value="0.25" default-value="0.01" numeric-type="SquareMillimetersToPixels"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageCapture" name="0" enabled="true" default-light="true" settle-first="true" count="1"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageWriteDebug" name="deb0" enabled="true" prefix="bv_source_" suffix=".png"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.BlurGaussian" name="3" enabled="true" kernel-size="9" property-name="BlurGaussian"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskCircle" name="4" enabled="true" diameter="525" property-name="MaskCircle"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="5" enabled="true" conversion="Bgr2HsvFull"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskHsv" name="6" enabled="true" auto="false" fraction-to-mask="0.0" hue-min="60" hue-max="130" saturation-min="32" saturation-max="255" value-min="64" value-max="255" soft-edge="0" soft-factor="1.0" invert="false" binary-mask="false" property-name="MaskHsv"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskHsv" name="threshold" enabled="true" auto="false" fraction-to-mask="0.0" hue-min="0" hue-max="255" saturation-min="0" saturation-max="255" value-min="0" value-max="100" soft-edge="0" soft-factor="1.0" invert="false" binary-mask="false" property-name=""/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="7" enabled="true" conversion="Hsv2BgrFull"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="8" enabled="true" conversion="Bgr2Gray"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.FindContours" name="findCountours" enabled="true" retrieval-mode="List" approximation-method="None"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.FilterContours" name="filterContours" enabled="true" contours-stage-name="findCountours" min-area="0.01" max-area="900000.0" property-name="FilterContours"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskCircle" name="11" enabled="true" diameter="0" property-name=""/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.DrawContours" name="contours" enabled="true" contours-stage-name="filterContours" thickness="2" index="-1">
         <color r="255" g="255" b="255" a="255"/>
      </cv-stage>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MinAreaRect" name="results" enabled="true" threshold-min="100" threshold-max="255" expected-angle="0.0" left-edge="true" right-edge="true" top-edge="true" bottom-edge="true" diagnostics="false" property-name="MinAreaRect"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageRecall" name="14" enabled="true" image-stage-name="0"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.DrawRotatedRects" name="15" enabled="true" rotated-rects-stage-name="results" thickness="2" draw-rect-center="false" rect-center-radius="20" show-orientation="false"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageWriteDebug" name="deb1" enabled="true" prefix="bv_result_" suffix=".png"/>
   </stages>
</cv-pipeline>

[Zdenko Stanec]

Hi John,

First, you should have some kind of "shade" around the nozzle as without it you will sometimes get too many misdetects as the camera can pick up some random detects.

When you add this shade, recalibrate the machine and try this pipeline, if it is not OK, just try to adjust the "threshold".

<cv-pipeline>
   <stages>

      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageCapture" name="0" enabled="true" default-light="true" settle-first="true" count="1">

         <light class="java.lang.Boolean">false</light>
      </cv-stage>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageWriteDebug" name="13" enabled="true" prefix="bv_source_" suffix=".png"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.BlurGaussian" name="10" enabled="true" kernel-size="9" property-name="BlurGaussian"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskRectangle" name="4" enabled="true" width="450" height="450"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="1" enabled="true" conversion="Bgr2HsvFull"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.MaskHsv" name="2" enabled="true" auto="false" fraction-to-mask="0.0" hue-min="0" hue-max="5" saturation-min="50" saturation-max="100" value-min="200" value-max="250" soft-edge="0" soft-factor="1.0" invert="false" binary-mask="false" property-name="MaskHsv"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="3" enabled="true" conversion="Hsv2BgrFull"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="6" enabled="true" conversion="Bgr2Gray"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.Threshold" name="12" enabled="true" threshold="200" auto="false" invert="false"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.FindContours" name="5" enabled="true" retrieval-mode="List" approximation-method="None"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.FilterContours" name="9" enabled="true" contours-stage-name="5" min-area="100.0" max-area="630.0" property-name="FilterContours"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.DrawContours" name="7" enabled="true" contours-stage-name="9" thickness="2" index="-1">

         <color r="255" g="255" b="255" a="255"/>
      </cv-stage>

      <cv-stage class="org.openpnp.vision.pipeline.stages.MinAreaRect" name="results" enabled="true" threshold-min="1" threshold-max="255" expected-angle="0.0" left-edge="true" right-edge="true" top-edge="true" bottom-edge="true" diagnostics="false" property-name="MinAreaRect"/>

      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageRecall" name="14" enabled="true" image-stage-name="0"/>

      <cv-stage class="org.openpnp.vision.pipeline.stages.DrawRotatedRects" name="8" enabled="true" rotated-rects-stage-name="results" thickness="2" draw-rect-center="true" rect-center-radius="5" show-orientation="false">
         <color r="0" g="255" b="0" a="255"/>
      </cv-stage>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageWriteDebug" name="15" enabled="true" prefix="bv_result_" suffix=".png"/>
   </stages>
</cv-pipeline>

Zdenko,

[Clemens Koller]

Hi!

As Zdenko wrote already, I also strongly recommend to get a uniform background - maybe even blurred because
of depth of focus.
Ideally you a grey background around 50% intensity (i.e. 127DN out of 255DN), so the IC package itself
is darker than the grey background and the pins are brighter.
The pinx can be almost white (reflective), but avoid clipping / driving the camera pixels into saturation
as it distorts position information.

Additionally, you can use i.e. a circular mask to eliminate the area outside of the nozzle if your
component size permits.

Regards,

Clemens

On 7/12/22 20:00, John Lettang wrote:
> Hi all,
>
> I have my pick and place working with the top camera alone now. However I am having some issues with bottom vision.
>
> To start, I am not able to do the advanced bottom camera calibration. I am getting the error of too many misdetects and am being told to look at my fiducial/nozzel tip detection however both of those seem to be working fine.
>

> And of course that issue with the bottom camera calibration may be the issue I have with bottom vision where I am finding that I cannot find a good threshold and way to isolate the part with my current set up. I was looking through the instructions for both of these on the wiki <https://github.com/openpnp/openpnp/wiki/Calibration-Solutions#advanced-camera-calibration> but I couldn't quite figure it out. Namely the wiki <https://github.com/openpnp/openpnp/wiki/Bottom-Vision> talked about "probing" pixels you wanted and didn't want but I couldn't figure out how to do this and didn't see a dedicated wiki page to bottom vision.

> --
> You received this message because you are subscribed to the Google Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com <mailto:openpnp+u...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com <https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[John Lettang]

Hi,

so I already have a sort of shade on the nozzle which is just 3D printed. However, it is quite a bit more reflective than I thought so I will try to put some construction paper on it or something to make it less reflective.

However, I think my main issue even with the pipeline you gave me Zdenko is just that the nozzle tip is reflecting light as brightly as the pads themselves, and I can't think of a good way to make that less reflective. I can try using less light on the bottom or something but I don't know for this one. As you can see in the pictures below it is a pretty good threshold for the pads, it jus also gets the outside of the nozzle tip...

Also, if I am misunderstanding what you mean by "shade" I apologize. Perhaps a picture would be beneficial.

Best,

John

[Clemens Koller]

Hi, John!

I started with spray painting a disc as a background and also the nozzles, with non-perfect result.
Finally, I ended up using some green (blue might also work) coloured matte non-reflective but also non-structured plastic
shield behind the nozzle. A piece of blank green PCB material might also work nicely.
If you go with colour cameras, I would go for green as there are likely less green bits on components - for AOI reasons, I bet! ;-)

Ideally, the nozzle tip doesn't introduce additional geometry in the vision, but YMMV.
An automatic light adjustment to trim the background to a certain DN might improve consistency in the image recognition.

Clemens

On 7/12/22 22:34, John Lettang wrote:
> Hi,
>
> so I already have a sort of shade on the nozzle which is just 3D printed. However, it is quite a bit more reflective than I thought so I will try to put some construction paper on it or something to make it less reflective.
>
> However, I think my main issue even with the pipeline you gave me Zdenko is just that the nozzle tip is reflecting light as brightly as the pads themselves, and I can't think of a good way to make that less reflective. I can try using less light on the bottom or something but I don't know for this one. As you can see in the pictures below it is a pretty good threshold for the pads, it jus also gets the outside of the nozzle tip...
>
> Also, if I am misunderstanding what you mean by "shade" I apologize. Perhaps a picture would be beneficial.
>
> Best,
>
> John
>
> On Tuesday, July 12, 2022 at 2:17:12 PM UTC-6 Clemens Koller wrote:
>
> Hi!
>
> As Zdenko wrote already, I also strongly recommend to get a uniform background - maybe even blurred because
> of depth of focus.
> Ideally you a grey background around 50% intensity (i.e. 127DN out of 255DN), so the IC package itself
> is darker than the grey background and the pins are brighter.
> The pinx can be almost white (reflective), but avoid clipping / driving the camera pixels into saturation
> as it distorts position information.
>
> Additionally, you can use i.e. a circular mask to eliminate the area outside of the nozzle if your
> component size permits.
>
> Regards,
>
> Clemens
>
> On 7/12/22 20:00, John Lettang wrote:
> > Hi all,
> >
> > I have my pick and place working with the top camera alone now. However I am having some issues with bottom vision.
> >
> > To start, I am not able to do the advanced bottom camera calibration. I am getting the error of too many misdetects and am being told to look at my fiducial/nozzel tip detection however both of those seem to be working fine.
> >

> > And of course that issue with the bottom camera calibration may be the issue I have with bottom vision where I am finding that I cannot find a good threshold and way to isolate the part with my current set up. I was looking through the instructions for both of these on the wiki <https://github.com/openpnp/openpnp/wiki/Calibration-Solutions#advanced-camera-calibration <https://github.com/openpnp/openpnp/wiki/Calibration-Solutions#advanced-camera-calibration>> but I couldn't quite figure it out. Namely the wiki <https://github.com/openpnp/openpnp/wiki/Bottom-Vision <https://github.com/openpnp/openpnp/wiki/Bottom-Vision>> talked about "probing" pixels you wanted and didn't want but I couldn't figure out how to do this and didn't see a dedicated wiki page to bottom vision.

> > To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com <https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com> <https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/openpnp/04398d9c-33a3-4428-b60d-de2054cce563n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> You received this message because you are subscribed to the Google Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com <mailto:openpnp+u...@googlegroups.com>.

> To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/6178ed44-0823-44ec-adf0-11fa40c4be7fn%40googlegroups.com <https://groups.google.com/d/msgid/openpnp/6178ed44-0823-44ec-adf0-11fa40c4be7fn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Litterio Andrea Guainella]

Hi John,

I also had the same problem and solved with print a led diffuser with white PLA+.

As you can see this diffuser is very very simple so just place it on top camera mainly because I use stock liteplacer camera ( yesterday the new ELP arrived. I can't wait to mount them).

Alternatively you can try with oven cook paper (I tried with it before print diffuser for sure reason) cut squadre sheets (4/5) and union throught biadhesive at corner. After with scissor create hole with same dimension of eye of camera then try to vision part and adjust adding or removing sheets.

If you use ELP camera you should try (without print diffuser) to lower exposure value (without diffuser) in camera settings. (a best value range -7 ~ -10) as Zdenko suggest with this modification you should see a little shadow around nozzle.

LAG

[mark maker]

Hi John,

the others gave good tips.

I would opt for a bright green shade. Just using a heavy "art shop" matte color paper works very well. If possible you can mount it lower down, so it would obstruct the nozzle shaft and its potential reflections.

For those using green Juki nozzles, the two greens should match quite well.

Then perform OpenPnP (not device!) white balance by holding some brushed metal etc. in front of the camera.

https://github.com/openpnp/openpnp/wiki/Camera-White-Balance

Then you can hold the green paper or other material into the camera and enable the color histogram in the camera view (context menu/right mouse button).

It should give you a nice defined green bump peeking out from the red/blue baseline, probably sharper and higher than this one:

Now to your nozzle tip reflection:

This is really an amazing amount of frontal reflection. Are you using co-axial lighting? Super strong LEDs? The part is even seemingly glows with diffraction!

And why is the nozzle so sharp all the way up? What camera are you using? This looks like super-strong lighting, large-sensor/wide angle/closed aperture (>= f22) quality:

For reference. This is how it look on my machine (ELP camera):

_Mark

To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/8f11e3cf-3ee1-47ae-8271-192cd7836f84n%40googlegroups.com.

[John Lettang]

Hi all,

Thank you for all of the ideas. I managed to find some matte green construction paper so I am going to try to go ahead and put that on my shader and hopefully it will help a little bit.

As for the nozzle and lighting. I am going to attach a couple of pictures of my set up just so you all can actually see it. Truth be told, this liteplacer kit was assembled a while ago so I don't know the make or model of the camera, only that they are "USB 2.0 cameras". However, I would be more than happy to switch the camera out. If I may ask, what ELP camera do people use here?

As for the lighting, it is just an LED ring that has a 3D housing with one layer for diffraction. However, I might try the LED diffuser that LAG suggested. As far as I know the LEDs aren't unusually strong but either way when I added the green construction paper it helped tremendously. Only thing is my advanced bottom camera calibration now goes through but gives a "fish eye" effect. I will give it a couple more tries but it takes so long I wanted to mess with some other settings before going back to that one. But I think the white balance also helped a ton.

I will come back when I have a couple theories for what is happening with the advanced calibration.

-John


The picture of my nozzle and top camera is from before I put on the green construction paper, but that is my setup.

https://drive.google.com/drive/folders/17Zc49g7GDW4Hl_Ia5g_5uP2IVcvwVqm3?usp=sharing

[John Lettang]

Alright, I am back a lot later to say that I have in fact not figured out what is happening with the advanced bottom vision calibration. I would think that there would be some misdetects happening or something but whenever I am looking at the camera visual it usually has a pretty accurate read on the center of the nozzle. I have tried the calibration with three different nozzles, all of the yielding about the same result. Any help would be appreciated.

Best,

John

[tonyl...@gmail.com]

Please post the log and your resulting machine.xml

[John Lettang]

Here they are, I did also notice at the end that it looks like it got off at the ending few checks. It looked like it was because of lighting so I will see if that will help at all.

[tonyl...@gmail.com]

John,

Your residual errors still look very large - even ignoring the outliers they are at least 100 times larger than I see with my LitePlacer: 

Now, I am using a higher resolution camera format (1280x1024 cropped to 1024x1024) versus your 640x480 so that may explain some of the difference, maybe a factor of 2 or 3 but certainly not a factor of >100.  Some things to consider:

1. I assume your top camera calibration works ok so your basic machine mechanics must be working ok.  However, verify that your bottom camera and lens are securely mounted - if the camera or lens is moving around it will make a mess of things.
2. Post an image from your bottom camera (with advanced calibration turned-off) of your nozzle tip.  You need to have a good shade behind the nozzle for good bottom vision performance.
3. Check your camera settling - maybe the images you are capturing are from before your machine vibration has damped-out.
4. I recommend cropping the bottom camera to a square image (under the General Settings section on the Advanced Calibration tab) - it would be exceptionally rare to need to use those pixels that are outside the largest circle you can inscribe within the image (so no point in calibrating them and bogging-down all your pipelines with them).
5. Turn-off or dim any overhead lights in the room before you run advanced calibration on the bottom camera - the extra glare can sometimes produce false detections.

Tony

[John Lettang]

Hi Tony,

1. Hmm, I assumed that my top camera advanced calibration was fine but I just looked at the numbers and compared them to yours and they are still quite large error wise. I will post a screenshot of it below. Does this mean something might be off with the distance my system goes during each movement?

2. Image posted below, I have a dark green shade and there are lights above the setup but they are turned off during advanced calibration. The dark parts aren't typically a problem but I could try redoing the shade to see if that will help.

3. Looking over settings I am not quite sure if there is a recommended settle method and time. I am not seeing too much info on it on the wiki but I can trial and error with it a bit to see if that would help. Right now I am at the default "fixed time" "250 ms"

4. I will try cropping to see if that helps

5. I will also go ahead and try making the room more dark.

Also, would anyone have a good recommendation in case I should try different cameras?

Thanks,

John

[tonyl...@gmail.com]

John,

Yeah, something looks amiss there as well.  Just for reference, here are the results from my top camera calibration:

As you can see, my residual errors are a couple of orders of magnitude lower than yours.  I have my top camera set to 800x600 format (FWIW, I'm just using the stock LitePlacer cameras).

I think the first thing you should do is investigate your camera settling (turn on diagnostics and run some tests) and then redo backlash compensation.  I can't say that these are optimum, but here are the settling settings I'm using on my top camera: