Isses with detecting outline ( ESP32 with print on it) how to invert image

[Fabian Kropp]

Hi,

i got the image capture for bottom vision pretty relyable now.

Thing is i want to capture the body of the part, not the pins.

The litte i understand, it figures the bright parts as outline. Which in my cae would be the print on the backside instead of the body outline.

I would like to invert the attached imag in teh vision pipeline to force detection of the actual part outline.

Vision pipeline is stock pipeline with a low threshhold.

any hints on how to quickly achieve this ? the documatation does not cover that.

thanky

Fabian

[mark maker]

Huh? Haven't we discussed all this? See Instructions here, and the linked discussion...

https://github.com/openpnp/openpnp/pull/1546

Especially this step by step that I wrote for you, and specifically about the diameter!

https://groups.google.com/g/openpnp/c/ucHeolNaPuk/m/LjVC3aX4AwAJ

Please report back this time.

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openpnp/0b9dbbe8-9743-419f-97b7-6492ed3d514en%40googlegroups.com.

[Fabian Kropp]

I had to swap out camera and nozzle / head setup.

I can’t get a bigger screen nor was I able to move the camera closer for now. 

I am currently stuck with the lumenpnp.

Any recommendations for an affordable kit that has the geometry right for such parts and is stiff enough ? (My lumenpnp is reinforced with water proof multiplex material - siebdruckplatte) 

I thought a quick win with the work you did would be converting the image, hence my question here.

Maybe I missed something when setting up bottom vision again for that part :-( 

Will double check later. 

But it worked fine for me until the nozzle head started loosing the tips when there was no vacuum , that was what I had reported back in the other thread 

Thx

Fabian

[Toby Dickenson]

Hi,

I am a lumenpnp user, and have just read that thread that Mark linked. I also place large modules with a tricky bottom view using multi-shot vision.

Lumenpnp changed the bottom camera from v2 to v3, moving the camera down and the focal plane up, increasing the bottom camera field of view. I think the lumenpnp team wanted to support large modules without using the openpnp multi-shot feature. The v4 has a wide-angle camera which supports even larger modules. I think that is all counterproductive. I have modified my v3 to be more like the v2: I moved the camera up to get more detail in bottom vision for small parts, and rely on multi-shot vision for anything large. So there is definitely some contradictory guidance around relating to the lumenpnp cameras. But it should be fine for what you are trying to achieve.

I dont see anything in the screenshots that you shared previously that would make your part challenging to the current stock vision pipeline, combined with Mark's guidance in that old thread.

Back in June Mark wrote "Needless to say, it would be better to fix the shade, i.e., make it larger or move it closer to the camera". This is necessary so that openpnp can automatically mask out the nozzle and background. What results do you have in your nozzle tip background calibration?

I see you have made the shade wider, with an extra layer above. If this is still not adequate, you might have more luck by moving it towards the camera. My camera has this vertical shade attached to the nozzle. It blocks some specular reflections of the white walls off the nozzle, and also provides a uniform background for the bottom camera with multishot vision on these large parts.

To view this discussion visit https://groups.google.com/d/msgid/openpnp/e9e4d81a-0d23-4676-b483-791f29547ee9n%40googlegroups.com.

[Fabian Kropp]

Thank you for the answer.

As I said I cannot change that right now.

I need the machine, it is used for production.

I for a reason asked for the inverted image.

I will try to get all of that fixed (new position / different lens etc.) and then get back to the proper fix.

For now I would want a quick and dirty hack by inverting the black and white image I got.

Therefore the separate thread.

Looks like I will have to place that by hand then. 

I am unable to deduct the possibility of image manipulation in the pipeline by myself. 

I should have posted the question from a different account then I probably would have gotten an answer.

Sad regards

Fabian

[Toby Dickenson]

The black and white image comes from a Threshold stage in the vision pipeline. You will find an "invert" option on that Threshold stage. Please let us know if this works for you.

Toby

To view this discussion visit https://groups.google.com/d/msgid/openpnp/9c17c6fd-53bc-4c51-bf4c-5c7a39b00e9bn%40googlegroups.com.

[mark maker]

> I can’t get a bigger screen nor was I able to move the camera closer for now.

Yes but you can also influence the span with the Max Part Diameter. Did this not work?

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/e9e4d81a-0d23-4676-b483-791f29547ee9n%40googlegroups.com.

[mark maker]

No manual tweaking should be required. Instead just use the "body" vision settings. And then restrict the area around the corners by using a smaller Max Part Diameter on the Nozzle tip. This will mask the surroundings away.

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/CAH35urfRyHG09Sf6BGZ9uKH80sEr%2BCJZ49z0Vjyu8yXxgdby5w%40mail.gmail.com.

[Fabian Kropp]

Hi mark,

even with a small max part diameter this seems not to work. 

I presume the issue still being here between my keyboard and the chair .....

See photos of the confirmation and result 

[mark maker]

Please send a native camera image, and your machine.xml

To view this discussion visit https://groups.google.com/d/msgid/openpnp/8fcdaa36-0f7e-4c9f-869f-78ed07dca277n%40googlegroups.com.

[Fabian Kropp]

PFA the requested items.

[Fabian Kropp]

I played a bit more with it.

It had worked before, because my screen had been much bigger - which I now cannot do any more.

Hence there must be something wrong (presumably on my side) which prevents the masking / proper identification of the body.

btw.: i had set pick tolerance to 4mm - which was stupid, cause that is taken into account of the mask if I read the code correctly

[Fabian Kropp]

With higher threshhold i get the pads and the darwing on the backside recognized, but that is really of, at least at one corner 

[mark maker]

Oh, I also need the packages and parts xml. 

Sorry.

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/730b03e4-26e1-4a8e-b290-55804d44a760n%40googlegroups.com.

[Fabian Kropp]

here u go, thanks for checking.

[mark maker]

Hi Fabian,

many problems in deed. I hope you read this carefully, took me about two hours to figure all out (including fixing a bug in OpenPnP, see below 😇):

The camera has huge color noise. This makes it very difficult to mask a key color. Brighter lighting would likely solve it.

You can fight the noise by increasing the Minimum Detail Size on the nozzle tip background calibration:

But this will be at the expense of precision. Not okay for finer parts!

The green shade is also not saturated enough, it reaches only saturation 66 out of 255. 

NOTE: unfortunately, there was also a bug in the diagnostics. When the key color is both too dark and not saturated enough, it will not find a single pixel that satisfies the criteria. When that happens, the HSV Minimum/Maximum determination fails. This will ironically lead to diagnostics saying it is okay when in fact it is not. I will fix that. Note, the bugfix will not fix the underlying problem, which is lack of saturation, so even after I fix that, you will still need to address it, only OpenPnP is then correctly telling you to.

Better lighting will likely help with saturation too. Or use a more vividly colored shade. If it is a paper, make sure it has no "fuzz" which makes it milky. Also check camera White Balance (follow the Wiki).

https://github.com/openpnp/openpnp/wiki/Camera-White-Balance

If you can't fix the shade saturation problem physically: Exit OpenPnP, open machine.xml, set worst acceptable saturation to say 31:

    background-worst-saturation="31"

That's a hack, you should really fix the underlying problem.

With that wou should get something like this in background calibration. In addition, I also had to increase the Tolerance to 16:

In my simulation with your still image I was able to make it work, using the stock pipeline. You need to select the Whole Part Body vision on the package:

That way I finally got a result. Note the accuracy is not very good, that's due to the high value Minimum Detail Size (blurring):

Unrelated: Why is your camera rotated around Z by 44°? Residual errors are also huge.

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/84509dde-61c7-44df-b642-5d7920ee6c7an%40googlegroups.com.

[mark maker]

Hi Fabian, all,

the diagnostics fix is now in the OpenPnP test version. So for a background that is both too dark and not saturated enough, you should get this diagnostic:

See the PR:

https://github.com/openpnp/openpnp/pull/1709

Download:

Test Downloads – OpenPnP

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/67860a3d-97c4-4af8-8fdd-64582bcd8456%40makr.zone.

[Fabian Kropp]

Thank you for your effort.

I will crank up the light, maybe that helps with saturation ( it is at 30% currently)

Still need to find something I can make the white balance with properly. Should be able to provide proper test results this time by tomorrow.

45° turned camera, is an issue that was clearly located in front of the keyboard / drill press for the mounting holes need to be offset by 45° to make the camera align.

Is there an issue with it? 

[mark maker]

> 45° turned camera, is an issue that was clearly located in front of the keyboard / drill press for the mounting holes need to be offset by 45° to make the camera align.
> Is there an issue with it?

No, that should be fine. You could crop it to all valid pixels, in order to avoid some vision ops being disrupted. Note, the usual vision ops are fine, as they often use a circular mask anyway.

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/ff3aae51-f86a-498c-85e1-1bd4ec172ecan%40googlegroups.com.

[Fabian Kropp]

Hi,

I am apparently too stupid to get the background right with the other nozzles ( that do overlap the tip) 

Would it help to color the nozzles ?

That the key color is not consistent enough i presume is a result of the nozzle not beeing in the same color, right?

 ( the shade is 3d printed .... whereas the old one was wrinkled paper, that was for sure not uniform ).

Btw this is not the original v4 shade of the lumenPNp, that would be much smaller and have inconsistent surface.

[Fabian Kropp]

But the ESP on the N75 nozzle works like a charm :-)

[mark maker]

Hi Fabian,

Your green shade now looks much better!

Now if you changed the background-worst-saturation= in your machine.xml you must exit OpenPnP, then revert it back to the default 63. Otherwise OpenPnPs recommendations will not be good for the nicely saturated shade!

Now for some theory:

There are two scenarios to be distinguished:

1. Normal bottom vision, using the bright contacts of parts.
   
2. Whole body bottom vision, using the whole part outline, like your ESP32 module
   

In scenario 1 having a black nozzle tip is not a problem. But it must be really matte black, yours is a bit shiny. If okay, it will be darker than the contacts, so not a problem. Technically, the green shade will be converted into black too. The reason we make it green is just to distinguish it safely, even if there is much ambient light etc. that would perhaps be hard to "swallow" with a black matte shade. But technically we could use a black shade too.

In scenario 2 we absolute need a vivid color shade, a black one would not work, because it cannot be distinguished from a body which (unlike the bright contacts) can have many colors including black. In this scenario the green is not converted into black, instead it is directly used as a mask to detect the outline of the part to be aligned. Because of this, you have two valid sub-scenarios: 

1. You have very large parts (lie your ESP32) that surely cover the whole nozzle tip. So in bottom vision the color of the nozzle tip is of no concern, only the shade matters. As for the background calibration, a sufficiently dark and matte nozzle tip should not bother it.
   
2. You have nozzle tips of roughly the same color as the shade. The color must not be exactly the same, OpenPnP can detect the hue span, as long as both are more or less in the same hue sector (I think it may span about a third of the color wheel). Again it is important that both colors are vivid = high saturation!
   

_Mark

To view this discussion visit https://groups.google.com/d/msgid/openpnp/654d36b0-68cb-443a-8d65-c2d89e6efccdn%40googlegroups.com.