Background Removal for Bottom Vision, Green Juki Nozzle Tips

[ma...@makr.zone]

Hi everybody,

a new testing version is available, adding calibrated background removal for Bottom Vision, when using green Juki nozzle tips (and other color-keyed nozzle tips).

No more manual MaskHSV tweaking in the Pipeline Editor needed!

As I do not have Juki nozzle tips, I need your help in testing this!

See the full description and instructions here:

https://github.com/openpnp/openpnp/pull/1383

Thanks, Jan, for the test images I used.

Testing version:

https://openpnp.org/test-downloads/

_Mark

[ma...@makr.zone]

No takers?

[Ian Arkver]

I finally got my machine through all the I&S stages last weekend. Thanks to you and Tony for all the work on those and the auto-cal features.

I'll give this a whirl too, but probably not until this weekend. I've Juki nozzles (well, clones, but they look the same).

All the best,
Ian

[Zdenko Stanec]

I would try but I have red bottom LED so I'm out 😁😁

[Duncan Ellison]

Thanks for this new feature _Mark, I did take a quick look but it was clear that I didn't fully understand how it was supposed to work.  I'm away until after the w/e, but will try again next week.

Just didn't want you to think you did this for nothing :-0

[theworl...@gmail.com]

Hi _Mark,

I just tried it and apparently don't fully understand how to implement it. I tried to implement the 'BrightnessAndKeyColor' option. While it shows colors and claims there are no issues when I click on the 'Show Problems' button, it doesn't change anything. I suspect I have the wrong/old pipeline since it doesn't use HSV. But I didn't see a new one to copy in from the instructions, nor did it reset to a usable one when I clicked 'Reset' on the NozzleTip screen.

I do see under Machine Setup -> Vision -> Bottom Vision a pipeline that looks closer to your instructions but doesn't seem to use the HSV settings from there.

[mark maker]

Hi theworld,

Yes, it should probably be mentioned more prominently that a MaskHSV stage in the pipeline is needed (the default pipeline always has one).

I propose you update to the newest testing version that also exposes the other important parameters:

https://github.com/openpnp/openpnp/pull/1384

and then follow the instructions to use the default pipeline:

https://groups.google.com/g/openpnp/c/V4codmahils/m/yoZWKVNnAgAJ

It will then use both the MaskHSV stage and the new Threshold and Minimum Detail parameters.

Btw. your image with the green shade looks very nice. Ignore the message about the background being too dark, it only happens because you have a green shade (I will try to fix that judgement).

_Mark

--
You received this message because you are subscribed to a topic in the Google Groups "OpenPnP" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openpnp/hjc3HpOUzzk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/d9a22e31-19e0-46a3-ba00-899289380ffdn%40googlegroups.com.

[Saito Ryosuke]

Hi mark

I am testing the functionality implemented in your PR#1383.

I am getting the latest test version and testing it.

https://github.com/openpnp/openpnp/pull/1383
I'm trying to test the functionality implemented in this PullRequest.



https://youtu.be/ztuNUodxr-w
In this video, I have selected "BrightnessAndKeyColor" as the Background Calibration Method.
You can see that the removal of color areas by MaskHSV works very well.

Very good function!

At first, I could not imagine what to do after selecting "BrightnessAndKeyColor" for "Background Calibration" - "Method".

It seems that background calibration was done by following the steps below, but please point out any unintended operations.

1. go to "Machine Setup" - "Nozzle Tips" - "Background Calibration" - change "Method" from "None" to "BrighenessAndKeyColor".
2. press "Apply" button at the bottom right.

3. press "Calibration" - "Calibration" - "Calibrate" to perform calibration of the nozzle.
4. the "Background Calibration" is completed.

//--------------------------------------------------------------------------------------------

I found problem in this version.
MaskCircle" is not working in "Vision" - "Bottom Vision Settings" - "Pipeline".

When I say "does not work", I mean that the image will be cropped with a *fixed diameter* 

even if you change the "diameter" parameter.



https://youtu.be/EMVyqUA4B60
At first, I was trying something different from the "Bottom Vision Nozzle Tip Background Knockout Calibration" test.
Naturally, I selected "None" for the Background Calibration Method, but I noticed that the Pipeline, 

which I had set in the past and was very stable, was no longer working. ("MaskRectangle" is working)

However, in the "Machine Setup - Nozzle Tips - Calibration" pipeline, 

the "MaskCircle" seems to provide the function to "crop" the diameter as specified.

thanks

Ryosuke

2022年2月20日(日) 1:30 ma...@makr.zone <ma...@makr.zone>:

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/c5b6222d-660f-409a-b416-dddccba9c6ben%40googlegroups.com.

[Saito Ryosuke]

oh I found #1384

I will try this PR first.

https://github.com/openpnp/openpnp/pull/1384

2022年2月26日(土) 2:27 Saito Ryosuke <cron...@gmail.com>:

[mark maker]

Hi Saito,

> I am testing the functionality implemented in your PR#1383.

Appreciate!

> At first, I could not imagine what to do after selecting "BrightnessAndKeyColor" for "Background Calibration" - "Method".
> It seems that background calibration was done by following the steps below.

>1. go to "Machine Setup" - "Nozzle Tips" - "Background Calibration" - change "Method" from "None" to "BrighenessAndKeyColor".
>2. press "Apply" button at the bottom right.
>3. press "Calibration" - "Calibration" - "Calibrate" to perform calibration of the nozzle.
>4. the "Background Calibration" is completed.

Yes that is right. I thought it was clear from this:

https://github.com/openpnp/openpnp/wiki/Nozzle-Tip-Background-Calibration#calibration-operating-principle

I have now added a note and link where the Method is described:

https://github.com/openpnp/openpnp/wiki/Nozzle-Tip-Background-Calibration#selecting-the-method

> MaskCircle" is not working in "Vision" - "Bottom Vision Settings" - "Pipeline". When I say "does not work", I mean that the image will be cropped with a *fixed diameter* even if you change the "diameter" parameter.

That is intentional, please read the Wiki:

https://github.com/openpnp/openpnp/wiki/Nozzle-Tip-Background-Calibration#calibration-pipeline-control

Unfortunately, there is no way to provide visual feedback in the pipeline editor, i.e. I cannot gray out the property, for instance (or I don't know how).

_Mark

You received this message because you are subscribed to a topic in the Google Groups "OpenPnP" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openpnp/hjc3HpOUzzk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/CAH6K8ZXoRLrUj6HcZCyN29--jpmpdfFai%2B7E7VDpobpYU_hxnQ%40mail.gmail.com.

[Saito Ryosuke]

Hi mark

Thanks for letting me try this nice feature. I'm very excited about this new feature!

> Yes that is right. I thought it was clear from this:

>https://github.com/openpnp/openpnp/wiki/Nozzle-Tip-Background-Calibration#calibration-operating-principle

Yes, I read that document.
I could read it as an explanation of the principle. Obviously, I think this information is necessary, too.

In addition, I think it would be easier for people like me, who have not yet explored the OpenPnP cave, 

to step through it if there were more specific step-by-step instructions for operation. :-)

>That is intentional, please read the Wiki:

>https://github.com/openpnp/openpnp/wiki/Nozzle-Tip-Background-Calibration#calibration-pipeline-control

I see! It was indeed written.

I was able to find the field it was affecting.
There seemed to be a large distance between each pipeline and the "Max. Part Diameter" value associated with the nozzle tip. I will have to read the Wiki carefully.

//----------------

Finally got around to trying out the features of PR#1384!
So far, I'm confident that I can free up the time I've been spending on pipelines.

Once I was able to notice the relationship between "MaskCircle" and "Max. Part Diameter" values, mark paved the way for the rest. This is a great feature!

This is a very important step to get to the future.

But I have created so many bottom visions in my life.
I am going to redefine them steadily.

thank you!

Saito

2022年2月26日(土) 3:04 mark maker <ma...@makr.zone>:

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/36fed45a-ac18-c49e-9aff-354d529e4f6c%40makr.zone.

[mark maker]

Hi Saito

It is valuable to get the user perspective, thanks.

The user experience is not yet complete. The two latest things (background removal, parameteric pipelines) are only building blocks towards a larger concept to make bottom vision easier. These building blocks are ready to use now, yes, but they require an adaptation of existing setups, which might be unnecessary complex, compared to a fresh setup, i.e. where the default pipeline can be assumed. Until I have all the building blocks in place, the user experience will be fragmented.

Once I have all the building blocks, the goal is to make users forget about pipelines, at least in bottom vision and fiducial vision. So all the difficulties with controlled/uncontrolled stages etc. should go away.

All the settings should be controlled from outside the pipelines, and you will have to set them up in a more logical order. The "Max. Part Diameter" for instance is a property of the nozzle tip, so when you set up the nozzle tip, you have to think about it, not when you set up bottom vision. It is a physical property that can easily be determined by the user, without any knowledge in computer vision, or knowledge about how to navigate and use the Pipeline Editor. 

These physical properties can sometimes have multiple purpose,  "Max. Part Diameter", for instance, does not only control the Mask of bottom vision, it also controls the sensitive area of Auto-Focus in part height sensing. More usages can be added.

Rather than writing a step-by-step list in the Wiki, maybe I can teach Issues & Solutions to ask questions like "What is the diameter of your largest part ever picked with nozzle tip NT3?". So you will be asked to provide these answers in a logical sequence, when setting up the machine elements. Issues & Solutions can also keep track of what you already configured and what not.

Having said that, for expert use, and to cover very exotic cases, that might fool the automatisms, everything must still be open for manual change. The Wiki will give guidance there, but it is true that everything needs to be read very carefully, and a lot of personal reasoning, background knowledge, and perseverance is required 😁

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/CAH6K8ZXFxiTNJW4%3DNPREksP-P1DDrB0u6vdD78z8sh8nU78xGg%40mail.gmail.com.

[Ian Arkver]

Hi Mark,

Something has broken. I rebased onto your latest test branch (3facba19) and fired up the machine. Visual homing now fails. It seems something odd has happened to the down camera pipelines/settings. I wonder if you've changed something in AbstractBroadcastingCamera.

I've pushed my .openpnp2 dir to GitHub in this repo:

https://github.com/ian-arkver/openpnp-settings

The version at the head of "test" was from before the last pull. The ones openpnp2 now creates are on the test_vision_bg_sub branch. The diffs are here:
https://github.com/ian-arkver/openpnp-settings/compare/test_vision_bg_sub?expand=1

I'm not sure what the virtual-camera-matrix is, but it seems significant that the numbers are wildly different. I also don't know why openpnp makes these changes without me doing *anything*, other than opening and closing the program. I guess this is some sort of settings migration step.

The log trace shows nothing of interest - just it homing and trying to find the home fiducial. There are no messages when it's changing all the settings on reading the old ones.

Here's a screenshot of how the down camera now looks to OpenPnP (up camera is fine, btw. it's just got nothing to look at here):

For reference, here's what's actually being captured and streamed by the camera. Yes the blue tape is vitally important ;-)

I'll try going backwards to see if I can find out where this regression was introduced.

Regards,
IanJ

[Ian Arkver]

Hi again,

So... odder and odder. I went backwards through git history versions of OpenPnP with my HEAD (test) version of the settings and all behaved the same (i.e. broken).

 I then rolled back one commit in the settings to the version before I re-ran the down camera calibration, and all is fine again.

I then went back to your latest test commit and it continues to be OK, if I use this version of .openpnp2 dir from before that calibration run.

So, it was actually that calibration run that broke things! Luckily I keep these settings backups in git.

I'll try rerunning the downcam cal again and see if that breaks it again.

Regards,
Ian

Happy OpenPnP pic for reference...

[tonyl...@gmail.com]

Something went terribly wrong with that calibration run of your down camera.  Please post the log file from that run.  It would also be instructive to look at the diagnostics displayed on the Advanced Calibration tab.

Tony

[mark maker]

Hi Ian,

Please, for this problem open a new thread, or continue the Advanced Camera Calibration thread. Thanks.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/3b4243f4-10ed-4612-94ae-d2b02804c112n%40googlegroups.com.

[Ian Arkver]

Back on topic - I fluked a nozzle calibration which ran to completion and got the HSV results. I think it's done it's thing well, but I'm not sure what the longer term benefit will be on other vision tasks. I've not done any of those, so can't quantify it.

Quick video of the "Show problems" attached.

Regards,
Ian