Top vision pipeline?

948 views
Skip to first unread message

SMdude

unread,
Dec 29, 2016, 6:34:11 AM12/29/16
to OpenPnP
Hi Jason and fellow openpnpers,

I'm trying to get my top camera to do fiducial recognition but it is being difficult. I have double checked the size calibration. I have also fiddled around with the fiducial pad size settings. It will identify holes in tape, no problem, just doesn't like my fiducials.
I have had a look in the opencv vision provider directory and it looks like I need to adjust the pipeline a little to get better definition on the fid circle.
Where can this be done?
Can this then be applied to different fids? Eg if I have different colour boards, those fids will need different settings.

Cheers

Jason von Nieda

unread,
Dec 29, 2016, 10:44:42 AM12/29/16
to OpenPnP
SMdude,

The FID locator has not been converted to the pipeline system yet, but I can tell you that in every single instance of someone having problems with fids it was not the vision code that was the issue. The most common issues are units per pixel problems and lighting. Can you send me some debug images so I can have a look? I can probably tell you what is wrong from those. 


Jason


--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/ca0170eb-c825-4541-89b1-b499b9a203dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

SMdude

unread,
Dec 30, 2016, 2:49:57 AM12/30/16
to OpenPnP
Hi Jason

I think I have figured out my problem.
Looking again at the pipeline images, I noticed that the fiducial dots were not that well defined. So added extra light, and it twigged. My cameras have auto exposure and white balance set as default. I also noticed that it had detected some nearby through hole pads that were much brighter compared to the fid.

The elp cameras I have, as default settings, have auto white balance enabled and auto exposure.
When initially testing out my cameras, I used iSpy. In this program I can set the auto exposure etc, except when I disconnect the camera, the camera settings default back to auto exposure and auto white balance, low light compensation.

Can these controls be added to the "camera specific" section?
I have looked about the net and can't find any programs for changing the settings of the camera itself.

I have attached images of the camera controls available in iSpy.

Cheers
cam exposure.JPG
Camera settings.JPG

Jason von Nieda

unread,
Dec 30, 2016, 10:16:50 AM12/30/16
to OpenPnP
Hi SMdude,

Unfortunately, controlling exposure and white balance in a platform independent way is "hard". This is something I've been working on for a long time. We currently use OpenCV for the majority of our camera capture and OpenCV doesn't handle these properties in most cases.

I just took another look, and it looks like it might work in Linux if you are using the V4L capture driver, and maybe in some other rare cases. I will go ahead and add controls for it, and we can see if it works, but I don't have high hopes.

Someone started working on this a while back in https://github.com/openpnp/openpnp/issues/328 but it was not finished. It's easy enough to finish, so I'll do that and maybe it will be helpful for some folks.

Jason




--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

Jason von Nieda

unread,
Dec 30, 2016, 1:20:56 PM12/30/16
to OpenPnP
Okay, you can now set OpenCV camera properties. See https://github.com/openpnp/openpnp/blob/develop/CHANGES.md#2016-12-30 for more information.

This may or may not work on your system and camera, but it's worth a try. Look for the exposure property and try different values to see if it has any effect. Make sure you hit Apply after any changes.

Jason

SMdude

unread,
Dec 30, 2016, 3:20:06 PM12/30/16
to OpenPnP
Hi Jason,

Superb!

I changed the setting for autoexposure and set it to "set after open", checked it, closed openpnp, disconnected cameras.
Hooked up cameras in order, opened openpnp, check exposure operation and the settings still were working :D

Its probably still best at this stage to first experiment with camera settings in iSpy or whatever as you have sliders for all of the settings and the settings change as you adjust.
Then just take your preferred values and put them in openpnp.
Would it be possible when you select the property, to show what the current setting is? <it's not that important though as it really will be a one off setup..

I will try later this morning to get it to recognise the fids, but I think it should be good now.

Thank you so much!

Cheers

Jason von Nieda

unread,
Dec 30, 2016, 3:47:58 PM12/30/16
to OpenPnP
Glad to hear it helped!

Populating the current value is a little complex due to how the configuration system works, but I added a button so you can pull the current value. Accomplishes the same thing, but this was much easier than actually showing the value when you change the property. There is a little refresh button next to the value field now. Give it a try and see what you think.

Jason


--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

SMdude

unread,
Dec 30, 2016, 10:06:22 PM12/30/16
to OpenPnP
HI Jason,

I have found for the bottom vision camera that the settings don't work/apply to the camera. When I hit apply, the camera does reset, just the settings do not stick/change.
Double checked the top and that one is working fine.

I was able to get it to identify my fids, but then it was too bright to identify the feeder tape. I will re-do my tests and attach the debugging images for you later on today.

Cheers!

Jason von Nieda

unread,
Dec 30, 2016, 10:45:06 PM12/30/16
to OpenPnP
Please send me your machine.xml
--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

SMdude

unread,
Dec 30, 2016, 11:37:48 PM12/30/16
to OpenPnP
Hi Jason,

Machine.xml attached.

Thanks a bunch!

Cheers
machine.xml

Jason von Nieda

unread,
Dec 30, 2016, 11:40:34 PM12/30/16
to OpenPnP
SMdude,

Are they the exact same model of camera?

Also, I don't know if you intended this or not, but you have the bottom vision camera exposure effectively disabled, since it's not set for before or after open. It should be set to one or both.

Jason


--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

SMdude

unread,
Dec 31, 2016, 12:09:19 AM12/31/16
to OpenPnP
Hi Jason, yes exact same camera for top and bottom.
I first checked apply after starting, that did nothing for the bottom camera.

Ok, just did a full restart and it is now applying my settings, however I did have to uncheck the before and after settings on auto exposure. Perhaps that was the problem.

Jason von Nieda

unread,
Dec 31, 2016, 12:11:07 AM12/31/16
to OpenPnP
Ah, yes, could be. It would depend on the order that things are set. You can change the order by rearranging the lines in machine.xml. I'll have to think about how to handle that in the UI.

Jason


--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

SMdude

unread,
Dec 31, 2016, 12:51:55 AM12/31/16
to OpenPnP
HI Jason,

When I hook up my cameras I always do it in the same order and so far they have always ended up in the correct function.
It would be nice however if each camera had a unique ID that could be stored in the camera settings, or if we could write an id to the camera firmware, but all things considered, it may not be worth the effort.

I have attached some pipeline debug images.
These were taken with the lighting off, in fact, everything was off just to make sure it was not getting interference from somewhere else.
Exposure settings were adjusted. I get the same results unless the exposure is set to make the image really bright, then paper tapes are too bright to be recognised.
It identifies the holes in tape really well, under normal lighting, and exposure conditions.

Cheers and happy new year!
1483162827942_2_result_8314089800874083473.png
1483162827942_0_template_6950692391742296475.png
1483162827942_1_camera_3750277772449486633.png

Jason von Nieda

unread,
Dec 31, 2016, 10:12:39 AM12/31/16
to OpenPnP
The fid image is very underexposed. Definitely need lighting. You want the fid itself to show up as a bright dot. Consider diffuse white light. For the tapes, just work on contrast for the holes. Openpnp is only looking at the holes in the tape, so as long as there is good contrast between the holes and the tape it should work.

FYI: most people use the elp came in auto exposure mode no problem, some people use them even without light. I chose the elp cameras because they have decent auto exposure.

Pardon brevity, traveling.

Jason

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

SMdude

unread,
Dec 31, 2016, 11:12:36 PM12/31/16
to OpenPnP
HI Jason,

Yes that image was taken with all motors and lighting, including my workshop fluros off, just to be sure there was no electrical noise effecting the imaging.
I have done some more tests and collected some more images, this time with my head lighting on, and exposure set as high as possible and still able to pick up the holes in tape.
I also added a makeshift diffuser(sheet of paper) which then made one of the fids off to the side on the next board shine, which it could see, but when it was centred over the fid, it no longer stood out due to no reflection.

So it looks like I might need to find a ring light that mounts in close to the camera to make the fids reflect back to the camera as well as a diffuser.
However, ideally, we don't want light reflecting directly back to the lens, the light should just illuminate the features??
What do commercial machines use for a lighting setup on the head?

I didn't get a raw image with the diffuser fitted.
There is a piece of paper covering the fiducial on the second board in the "with diffuser" images as it was picking up this one and not the one I wanted.

Anyway, enjoy your new years celebrations!

Cheers

no diffuser 1.png
no diffuser 2.png
no diffuser 3.png
raw image no diffuser.JPG
with diffuser 1.png
with diffuser 2.png
with diffuser 3.png

SMdude

unread,
Dec 31, 2016, 11:27:59 PM12/31/16
to OpenPnP
Oh and here's a pic of the setup just to show the actual lighting.
I can rotate the angle of the lighting and have tried with many different positions.

The camera is actually square to the bed. I spent a fair bit of time getting it "right" whilst trying to avoid at all costs making a much nicer mounting bracket!
And the goose that did the layout on those boards put the fids on the wrong side!

Cheers
lighting.jpg

Cri S

unread,
Jan 1, 2017, 2:49:26 AM1/1/17
to OpenPnP
You should check fiducial with gold/silver/... coating, not hal. Commercial machines have the same problems. For this transparent solder mask was created that resolve the problem with hal surface finish. Annother solution is ir illumination and ir filter removed. Both solution require that the fiducial is under solder mask.
The other solution on commercial machine to this problem is to invert colors and increase gamma.

SMdude

unread,
Jan 1, 2017, 5:32:20 AM1/1/17
to OpenPnP
Hi Cri,

I don't have any boards with gold fids unfortunately.
Looking at this example on youtube, I don't see a lot of difference as to what I had while I had auto-exposure set.
https://www.youtube.com/watch?v=aeW66c9oN5Y
I have now set all the camera settings back to default. Argh!



Cri S

unread,
Jan 1, 2017, 8:09:31 AM1/1/17
to OpenPnP
If you make a simple mapping (threshold),
Less then 80 and higher then 180 is white,
Rest is black. Maybe you must adjust the values,
take some shoots and check with gimp or Photoshop. On gimp, gray conversion is not luminance, you must choose the other item, otherwise the resulting gray image is different.
This then gives you good fiducial recognition and carrier tape hole recognition too.

Glen English

unread,
Jan 1, 2017, 5:05:35 PM1/1/17
to OpenPnP
Maybe some nearest-neighbour based hysteresis rather than a hard threshold.

that's what the Yammy does. Agreed Cri S on inverting and gamma modification. Yammy does same.

SMdude

unread,
Jan 1, 2017, 5:15:40 PM1/1/17
to OpenPnP
Hi Cri,

I will have a play later on and see.

Jason, I have been thinking, as the fiducial is a package and a part, would it not be possible to then use the part alignment function, diverted to the top camera instead, then the pipeline could be edited to suite each individual fiducial package, for different colour solder mask, board finish etc.
Could a check box be added to this screen to instead collect the image from the top camera?

Ultimately, if openpnp is intended to be used on different machines, including commercial machines, the top camera pipeline is going to need to be flexible to be able to be adjusted to different cameras, lighting etc.

Anyway, I thought I would put that out there to see what you think. I won't be attempting to setup/use it in the next few days so sleep on it!

Cheers
select camera.JPG

SMdude

unread,
Jan 1, 2017, 5:17:47 PM1/1/17
to OpenPnP
Oh, also, on the above screenshot, should I have enabled selected for the fid? I'm sure I have tried it enabled/disabled many a time, but wondering how it should be set.

Jason von Nieda

unread,
Jan 1, 2017, 5:25:13 PM1/1/17
to OpenPnP
Enabled doesn't matter for fids. The setting in the screenshot is for bottom vision only.

As I mentioned earlier in the thread, the fid checker has not been converted to the pipeline system yet, but it will be. Once it is it will be possible to set it up however you like. In addition, we will switch to circle detection by default which works better than the template matching we do now.

My intention is that all vision ops will be converted to the pipeline system eventually as this allows for the most flexibility. Not sure if we will do different pipelines per board but it might be worth considering.

Jason

On Sun, Jan 1, 2017 at 2:17 PM SMdude <spiteri...@gmail.com> wrote:
Oh, also, on the above screenshot, should I have enabled selected for the fid? I'm sure I have tried it enabled/disabled many a time, but wondering how it should be set.

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

Krzysztof Rudnicki

unread,
Jan 13, 2017, 1:02:38 PM1/13/17
to OpenPnP
Hi, 
I'm trying to use OpenPnP to my new machine DIY and I have a problems with fiducial detect.
Would be great, if the pipeline system will be availabe to fiducial dataction.
How do you think, when you could to add to this software?

Krzysztof

Jason von Nieda

unread,
Jan 13, 2017, 1:05:03 PM1/13/17
to OpenPnP
You can look here ( https://waffle.io/openpnp/openpnp ) to see what the currently development pipeline is. This feature is currently #5 on the list.

Until then, see https://github.com/openpnp/openpnp/wiki/Fiducials#troubleshooting. Most fiducial issues are just configuration problems.

Jason


SMdude

unread,
Jan 13, 2017, 4:56:00 PM1/13/17
to OpenPnP
Hi Krzysztof,

What you can do if everything is set up correctly(detects tape?) and it still isn't detecting the fids is instead of asking openpnp to automatically identify the fids is to:
Press the button next to the find fid button ans "set the boards location and rotation using 2 placements"
Then follow the prompts, select the fids you want to use and line up the camera on them.
This is the next best thing and it works!

Cheers

Marek T.

unread,
Jun 28, 2017, 10:46:15 AM6/28/17
to OpenPnP
Hi Jason,

Understand that subject of the fiducials pipeline editor has actualy stucked at some deadpoint? Not any activity from @visualecho1 (#425) since 3 months. We wait for his wake up or you do something with this?

I'm asking because I have following problem with fiducials:
- detection doesn't want work at ELP 1Mpx resolution at all despite of good quality visialisation for human eye. the screenshots may show soon.
- when set camera to VGA it's started to work. But a bit strange. I'm making experiments on some 2x2 panel where each board has 4 fidushales (it means in every corner, don't ask why some client made it so). And on some panels it works more-less ok, but on another the system "mixes" fiducials from two neighboring boards. It means, when we work on Right-Down board, system jumps properly to Left-Down first fiducial and makes first scan then jumps to the fidu which belongs to the board from the left side and scan it thinking that is still at the same fidu (so calculated fidu is worth nothing). Next the system jumps to the Right-Upper fidu and says it's out of 1% etc. Looks like limiting the range of scanning (MaskCirgcle) could solve it. Or maybe some other advise? Attaching two sets of fiducials recognition. Hope not problem that zipped, but there is many pics in completes and don't want to make to much mess here.
fiducialsOK.7z
fiducials-error.7z

Jason von Nieda

unread,
Jun 28, 2017, 11:00:18 AM6/28/17
to OpenPnP
Yes, that PR has stalled out and the codebase has changed so much since then it's probably going to be hard to merge. I have another branch where this is mostly working, it probably just needs another 2-3 hours to get it finished. I have some free time this week, so I'll see if I can get it finished.

Jason


--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Message has been deleted

Marek T.

unread,
Jun 28, 2017, 11:16:36 AM6/28/17
to OpenPnP
Good news! :-)

Jason von Nieda

unread,
Jun 29, 2017, 10:30:15 PM6/29/17
to OpenPnP
First version of this has been released to the test build: http://openpnp.org/test-downloads/

For more details see my comment on the issue: https://github.com/openpnp/openpnp/issues/329#issuecomment-312158337

Please let me know if this helps and if you have any problems with it. If it all looks good I'll merge it into the main release in the next day or so.

Jason


Marek T.

unread,
Jul 1, 2017, 4:16:39 AM7/1/17
to OpenPnP
Hi Jason,
I'll test it on Monday for sure.
Reply all
Reply to author
Forward
0 new messages