Strip feeder hole detection problem

[Jacek M.]

Hello,

I have a problem with detection of holes in strip feeder. Top camera px/mm is calibrated (reticle.jpg) so 1.5mm on camera is exact 1.5mm hole on strip. I changed pipeline for holes detection a bit as per improved pipeline.jpg. When I edit pipeline or even try to detect holes on feeder, red circles are spot on on holes on strip (spot on circles.jpg). The cricles I believe are used to set up feeder (green) are offset though (offset.jpg). I'm wondering af anybody can offer any help on that? I tried a few different cameras, calibrated lens, calibrated px/mm and problem still exists.

[ma...@makr.zone]

Hi

there is a bug in the standard pipeline. I reported it here,

https://groups.google.com/g/openpnp/c/c9jclp9Dii0/m/81dBYeC_AwAJ

At the time Tony Luken was in the middle of developing other improvements to the ReferenceStripFeeder, so I hoped he would fix it, but it seems to have gone unnoticed.

This is what I reported (fixed pipeline pasted at the end):

The ReferenceStripFeeder  standard pipeline has a Canny edge detection in front of Hough Circles.

But Hough circles has its own Canny edge detection in-built, so this will create "the edges of the edges". What Hough is seeing is this (simulated with two Cannies in a row):

Sometimes it will therefore fuse parts of the outer and inner circles together and create this:

That's easily observable in the NullDriver Auto Setup.

The Canny should be removed.

Also, I don't think a Median Blur is a good thing before edge detection, it will amplify aliasing. Just leave the Gaussian blur in.

When I disable Canny and BlurMedian I get this:

Now I can successfully pick 0201. Well in the NullDriver... :-)

That's what I'd recommend. But as @tonlyluken is on it, I hope he can consider and integrate this:

<cv-pipeline>
   <stages>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageCapture" name="original" enabled="true" settle-first="true" count="1"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ConvertColor" name="gray" enabled="true" conversion="Bgr2Gray"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.BlurGaussian" name="predetect-1" enabled="true" kernel-size="5"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.DetectFixedCirclesHough" name="results" enabled="true" dp="1.0" param-1="25.0" param-2="20.0"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.ImageRecall" name="recalled" enabled="true" image-stage-name="original"/>
      <cv-stage class="org.openpnp.vision.pipeline.stages.DrawCircles" name="display" enabled="true" circles-stage-name="results" thickness="1">
         <color r="255" g="0" b="0" a="255"/>
      </cv-stage>
   </stages>
</cv-pipeline>

_Mark

Am 08.12.2020 um 01:50 schrieb Jacek M.:

Hello,

I have a problem with detection of holes in strip feeder. Top camera px/mm is calibrated (reticle.jpg) so 1.5mm on camera is exact 1.5mm hole on strip. I changed pipeline for holes detection a bit as per improved pipeline.jpg. When I edit pipeline or even try to detect holes on feeder, red circles are spot on on holes on strip (spot on circles.jpg). The cricles I believe are used to set up feeder (green) are offset though (offset.jpg). I'm wondering af anybody can offer any help on that? I tried a few different cameras, calibrated lens, calibrated px/mm and problem still exists.

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/00c27d4d-8d58-4b8f-8b18-ddf4247d36c8n%40googlegroups.com.

[Jacek M.]

I tried your pipeline, thanks, but it didn't fix the problem. I can see in your bug report that your red circles were off. On my machine red circles are spot on and problem is with green circles being off. I'm wondering if anybode else had similar issue?

[Richard Sim]

ok, trying to remember here - it was quite a few years ago that I wrote that code...

Jacek:

I think what you have is a combination of two things happening:

1) I think your tape height is set incorrectly. The size of the circles (for all colours) is the detected circle in the image, not the idealized 2mm circle size of tape holes. The green circles spacing is at the camera's idealized 4mm spacing for the tape - if that's off, it's because your strip height is incorrect. The height of paper tape and plastic tape is different (yes it's small, but it matters a lot!), so make sure you set it correctly for each strip.

2) You may have set up your strip backwards. In the first image the camera is between the two strips of tape, whereas it should be over a part at that point (IIRC). I may be wrong here, or you may have moved your camera, but I'm pretty sure if shouldn't look like that.

Mark:

The default pipeline is set up that way to handle as many real-world tape and lighting setups as possible. It's a place to start from that should at least get people up and running, but is unlikely to be ideal for everyones machine - or even every strip that a single person is using (white paper, black plastic, transparent, ...). The median blur is disabled by default, as like you say it's not good for most situations - but it was left in so it could easily be toggled on in place of the gaussian blur in cases where it did help (I can't remember what those are anymore though). The Canny -> Blur -> Hough is done very deliberately however, again to handle real-world tape, lighting, and camera FOV/perspective distortion. Paper tapes have relatively significant depth to them, which will very much confuse the circle detection, especially for circles not directly under the camera or in less-than-ideal lighting where you have shadows. It's very possible that the default pipeline can/should be tweaked further to improve the results (offsets are bad, yo), but it does require testing a whole ton of bad camera, lighting, lens, and tape types; the NullDriver is definitely none of those!

-Richard

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/171a565d-3408-485d-975c-331618cf53d7n%40googlegroups.com.

[Richard Sim]

Mark: I just took a quick refresher at the code, and one more note on the default pipeline configuration. One big thing that stands out is that the blur sizes are in pixel units, so machines with different camera resolutions will need to have those changed to get the expected results. The default pipeline was tuned to the recommended ELP 2.0MP camera's resolution for this reason. In the Canny -> Blur -> Hough case, the blur is done with a radius of 5px, and since the Canny will result in 1px edges, this gets blurred out more than enough to not cause Hough to have issues. Hough will never detect overlapping circles, so you should get back a single circle centred at each blurry circle, of approximately the right size (Hough is limited to detecting 1.35-1.65mm circles). Like I said in my original reply, it's made to work for as many bad setups as possible. I'd hope that if people put the effort into making a machine that results in clean images, that they put some of that effort into configuring the pipeline to take advantage of that quality!

And yes, I meant 1.5mm tape holes in my last post. :)

[Jacek M.]

Hi Richard,

I'm wondering what do you mean by tape height? Tape thickness which is about thickness of a part? I have searched everywhere but I'm not sure where do you set it. This though, led me to recalibrate px/mm in a way that object was at the heiht of PCB/feeders and it improved a lots. This leads me to a conclusion that I need better camera with better lens. Thanks for your help.

Regards,

Jacek.

[Richard Sim]

You should be able to specify the height/z via the first reference hole (but it may be broken? I don't have anything running to check unfortunately): https://github.com/openpnp/openpnp/wiki/ReferenceStripFeeder#z-and-pick-height

Ideally you'd have the top of the PCB and the top of all feeders at the exact same z height, as it makes things much easier.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/38fa6184-619a-487d-bda1-badd1256191dn%40googlegroups.com.

[Jacek M.]

Okay I understand now. I set it up after autodetection. Camera Z is fixed so I'm not sure why would it have any effect on hole size? Do you say that I have to find on what height tape is, enter it before autodetection and then autodetect?

[Richard Sim]

Sorry, I mis-remembered about the height stuff. The specified z/height of the strip won't affect the dimensions that are used, so correcting the strip height will not fix offsets/sizes. The only way to correct it is to do what you did and calibrate the camera's units per pixel at the height of the top of the strips. But it does reinforce how important it is to have the top of all your feeders and the top of your PCB all aligned quite accurately, as otherwise you'll see the same kind of offsets you saw with the strip feeder in other areas too like fiducial checks, and of course other feeders that use vision.

I'm trying to find this mentioned in the wiki, but coming up pretty short. The only reference I can find is buried in the last sentence of Head Offset here: https://github.com/openpnp/openpnp/wiki/Troubleshooting-Placement-Accuracy#head-offset

Note that this head offset is only valid at a single z-height (your PCB height) and it is recommended to have all pick locations/feeders/strip tapes on the same z-height.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/55980d4e-6b8d-46f1-aeb0-ed364e358f9en%40googlegroups.com.

[ma...@makr.zone]

Hi Richard

Your rationale behind Canny is perfectly valid, but the problem is that HoughCircles has a built-in Canny that you may have overlooked, because it is not very well documented. In fact you need to read the rather obscure comments about param1 to realize it:

https://docs.opencv.org/master/dd/d1a/group__imgproc__feature.html#ga47849c3be0d0406ad3ca45db65a25d2d

By adding a second Canny in front of HoughCircles you are doubling the edges of the sprocket hole, so the HoughCircles detection now has two potential targets.

> Hough will never detect overlapping circles

Not two overlapping circles but one off-center circle that takes some of its gradient maxima from one edge and other maxima from the doubled-up edge. The HoughCircles is probabilistic and tolerant to some degree (see param2). It seems it can easily derail from one edge and snap into the other. In fact I only found the issue because that is precisely what happened in my case (see my earlier post with the image showing it clearly).

--> vs.

In real camera images, the holes are often away from the camera center and will therefore be seen slightly from the side, which makes the sprocket hole look asymmetric (in 3D). Shadows and highlights may also vary due to lighting, that is not centered relative to the hole. As a consequence, there may actually be multiple asymmetric edges in play. The double Canny will make it even more likely that some of these doubled-up asymmetric edges can blend into each other and shift the detected circle. 

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/CAEP8J%3DMG2UHVYbo%3DLqSBB90%3DHGYJ8VAY4LtKXJmP5NusW_JeWQ%40mail.gmail.com.

[ma...@makr.zone]

> I can see in your bug report that your red circles were off. On my machine red circles are spot on and problem is with green circles being off.

The circles all come from the same detection. The ones forming the longest row are highlighted in red. Those are the only ones that matter for setup, if I remember correctly. You can forget about the other colors (in fact you could probably disable the stage drawing them).

If you want to setup multiple feeders side by side you need to optically cover the ones you don't want detected. Somebody should probably fix that to only include rows near the camera center.

Looking at your original pictures more closely, I noticed you don't have the double Canny stage at all. So my initial remark does not apply. Sorry about the wasted cycle.

The detection does not actually look that bad. Once you cover up the unwanted feeder and roughly center the camera above the sprocket holes, I'm not sure you have a problem there at all.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/171a565d-3408-485d-975c-331618cf53d7n%40googlegroups.com.

[ma...@makr.zone]

Personally, I'm not convinced it's this issue, see my other reply.

But, he probably means that you calibrated your camera for a specific mm/pixels value at PCB surface height. And maybe the feeder is at a different height from your PCB. This will change the effective mm/pixels scale because the feeder/tape surface/sprocket hole edge is closer to/farther away from the camera. The ReferenceStripFeeder Vision assumes and depends on correct camera image scale to some degree.

You can enable millimeter ticks on the cross-hairs, so you can see if the row of sprocket holes matches every 2nd tick.

If you have a wide lens and wildly different feeder/PCB heights there may be a point where you simply can't make it work. Either make sure to have very similar heights across PCB and all (visual) feeders or use a much longer lens and higher-up camera mount.

Like I said, I don't believe that's the case for your examples. But maybe others will read this in the future and it might help them.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/38fa6184-619a-487d-bda1-badd1256191dn%40googlegroups.com.

[ma...@makr.zone]

Oh, just saw that Richard already answered to that effect.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/2ab6f5dd-7478-5649-cd9c-d0dcec703b38%40makr.zone.

[Jacek M.]

I actually have top of PCB and top of feeders aligned quite closely. But I tried to cut corners and put PCB I use for camera px/mm calibration on top of feeders, so it was probably about 3mm away from feeders. Anyway, I bought camera with 6mm lens and hopefully it will be much more accurate.

[Jacek M.]

I think it only takes into account these which are next to the part you clicked in wizard, there's no need to hide other feeders.

[Jacek M.]

Going forward I'm going to use camera with 6 mm lens to avoid such problems, including fisheye.