Writing tests for ReferencePushPullFeeder

[domini...@gmail.com]

Hi all,

I'm currently writing tests for ReferencePushPullFeeder as it currently is un-tested and I'm scared to make changes without knowing that I haven't broken existing code.

I have one question which I'm hoping to get an answer for, and this is this:

```

// FIXME why do we need to move the camera here? Without this, the call to `performVisionOperations` from `obtainCalibratedVisionOffset` will be
// using the current head position (0,0) which is obviously wrong for vision calibration.
machine.getDefaultHead().getDefaultCamera().moveTo(pickLocation);

```

The current version of the file is here: https://github.com/hydra/openpnp/blob/5fc59b4f508740c370acbe26e52130ec3b7821c2/src/test/java/org/openpnp/machine/reference/feeder/ReferencePushPullFeederTest.java#L174-L176

My WIP branch is here (in case I force-push over that commit: https://github.com/hydra/openpnp/commits/add-reference-push-pull-feeder-tests/

Latest version of the test is here: https://github.com/hydra/openpnp/blob/add-reference-push-pull-feeder-tests/src/test/java/org/openpnp/machine/reference/feeder/ReferencePushPullFeederTest.java

Any assistance greatly appreciated, and if anyone that's familiar with this code is in discord, please @mention me from the #development channel.

Cheers!

Dominic

[mark maker]

Hi,

I appreciate the effort! 

Just be aware that writing a useful test here is very difficult. When I say useful, I mean towards an integration test, that test a facility in its entirety and interaction, rather than based on mocked-up isolated single function unit tests. The latter I frankly find 95% useless, because they test the assumptions of the developer against the assumptions of the (same) developer, and nothing more. Worse, some developer then let themselves go in typical "risk compensation" and start using trial&error against the test, rather than thinking their design through, and being forced to decompose and simplify it, when they can't. I'm not saying you are such a developer 😇, but the next guy introducing a regression, might be. It's just my 35 year professional experience talking.

For a useful test, you practically need to simulate the physicality of the feeder for it to be conclusive. That means simulating a moving tape in the ImageCamera, i.e., it only being moved by an actuator, just when in the right position. Wow! 😎

For inspiration: I've already added many such "physical" simulations, including imperfections like vibrations, offsets, run-out, homing errors, focal blur, lens distortion, etc. and vision based pick and place location confirmation, that can be activated, see the SimulationModeMachine. 

https://github.com/openpnp/openpnp/pulls?q=is%3Apr+SimulationModeMachine+is%3Aclosed

If you want to realistically test this, you need to manually change the machine.xml to change the class of the machine to SimulationModeMachine. There is a machine.xml here, that could help:

https://github.com/hydra/openpnp/blob/5fc59b4f508740c370acbe26e52130ec3b7821c2/src/test/resources/config/SampleJobTest/machine.xml#L2

The SampleJobTest is the ultimate integration test that way. You could actually integrate yours there. Note, this is not my invention, it goes back to Jason the founder of OpenPnP. I just added many aspectual tests on top, and found those the most conclusive.

Back to your test and question:

I looked at the source and if configured right, the feeder should position here:

https://github.com/openpnp/openpnp/blob/5dd12c73972c74a5f3fa1810fc9167ce4919cec4/src/main/java/org/openpnp/machine/reference/feeder/ReferencePushPullFeeder.java#L2471

and or here (for OCR/QR):

https://github.com/openpnp/openpnp/blob/5dd12c73972c74a5f3fa1810fc9167ce4919cec4/src/main/java/org/openpnp/machine/reference/feeder/ReferencePushPullFeeder.java#L2520

I guess you are mocking up the feeder, without really setting it up. i.e. the sprocket hole locations 1 & 2 are still zero. 

Note, for the said useful tests, the machine and the feeders are usually setup interactively, in the SimulationModeMachine (see the machine.xml mentioned above, and accompanying other .xml files in the folder). This gives you the most realistic setup and also btw. implicitly tests the XML store/restore of the config, including upgrade handlers, to not regress. Integration is king!

Btw., I hope you read the other answer about the problems of a Tape Spec. 😬

https://github.com/openpnp/openpnp/issues/1635#issuecomment-2143916355

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/ffc44f4c-2591-42d2-824a-d9b439e4ba73n%40googlegroups.com.

[domini...@gmail.com]

Hi Mark,

My plan for simulating the actual pull, was to replace the camera's image with a new one, if the movement of the actuator matched the expected movement, and then have the feeder continue with locating the part.

I'll look into the SimulationModeMachine tests, I'm still navigating my way around the codebase and discovering as I go.

I currently have a passing test for `feed()` which includes detection of the tape, and doesn't throw an exception, in just going this the code coverage went from 0% to 85%.

Adding assertions and verifying operation of the actuator's movement is next for me, since that is what I need to test, and keep working when I work out how to add the code for 2mm component pitch tapes.

My WIP branch is updated, if you care to look at it now the tests pass.

https://github.com/hydra/openpnp/blob/add-reference-push-pull-feeder-tests/src/test/java/org/openpnp/machine/reference/feeder/ReferencePushPullFeederTest.java

Note that I also added tape specification here:

https://github.com/hydra/openpnp/blob/add-reference-push-pull-feeder-tests/src/main/java/org/openpnp/model/Tape.java

and corresponding unit tests:

https://github.com/hydra/openpnp/blob/add-reference-push-pull-feeder-tests/src/test/java/org/openpnp/model/TapeTest.java

*and* used it in the tests, where it is very useful indeed.

I'll review the other links and your other comments ASAP, lots to digest.

My feeling is that integration is NOT king, but very very important.  I prefer lots of unit tests and a few functional tests, and some integration tests.

* functional tests are slow, hard to write and have many dependencies.

* integration tests should just a few components working together, these are what you keep running when you're refactoring.

* unit tests are quick to run, quick to fail, can test many edge cases which require far to much test-setup and time to run as part of integration tests, you often throw many of them away when refactoring, often you use mocks to validate temporal ordering. (e.g. make Z safe /then/ move X/Y).

Given a 'test triangle' / 'test pyramid', ( https://martinfowler.com/articles/practical-test-pyramid.html ) I feel that Open PnP's triangle is currently inverted/top-heavy, however I'm still an OpenPnP n00b so take my thoughts with a pinch of salt.

Dominic

[mark maker]

Re testing: Nice theories, as always. I'll follow your practical efforts. 😎

> in just going this the code coverage went from 0% to 85%.

Now, I don't know how much tongue-in-cheek you said that 😉, but it's exactly such non-sense indicators that make my hair stand on end. Your test currently probes a miniscule aspect of the code, yet it indicates you got it covered by 85%.

> Can test many edge cases

Yes, in theory that's correct. But judging from my experience with employees over the years, I have yet to see a developer that spontaneously writes edge cases into their real-world tests, or is routine critical thinker enough to even find the relevant ones. It's called "confirmation bias", the stubborn human reluctance to try and find disconfirming evidence. Maybe when I can afford hiring a second developer to learn the same tech, study the same problem, understand the same design, etc. and then to implement such tests... and when I can offer them a bounty for tests that break the implementor's code, yes then, maybe, it will start to happen. Maybe big tech companies can afford that. Everybody else should remain realistic.

For more complex real-world cases, writing such conclusive tests is very difficult if not impossible. In the end, I prefer my developers using their rare and expensive brain time on the solutions. Decompose them until you really, really understand them and can stand behind them with professional pride. Best test criteria there is.

Having said that, I'm not against testing per se. Just against believing it to the silver bullet. And against test-driven development, and subsequent overconfident "implementation by trial & error". It is the same fallacy as teaching to the test. If we can agree on that, then this will work.

You can allay some of these fears by also including tests with simulated (random) imperfection and expected failure. 😉

To the problem at hand:

Note the ReferencePushPullFeeder is completely anchored and oriented by the two sprocket holes. Coordinates and orientation (east, west, north, south or any angle of rotation around Z) are determined by them. This affects cloning and transforming all coordinates and motion way-points to other feeders, the relative definition and orientation of the OCR/QR region, obviously the calibration, and likely other stuff I don't even remember.

With the 2mm pitch, the feeder will toggle between two states of where the sprocket holes are. In fact, to be as versatile as the feeder was before, any multiple given by the sprocket hole pitch divided by the part pitch should be supported. We might one day see 1mm pitch tapes, and the current ReferencePushPullFeeder can already do that, so you would have four states. The state will have to be persisted in the feeder.xml, and also be manually settable, when you mount an offline feeder or reload the tape. I don't know whether you can just use the present feed count. How a user would be able to see in which state a feeder is, physically, is another question.

This statefulness must be compensated for the said anchoring. Everything mentioned above, must work the same way, i.e. as if the sprocket holes were still in their neutral/initial position. So you will have to include feeder cloning, OCR/QR detection etc. into your tests, if you want to claim the test is conclusive. A very steep road to the real 85% 😅.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/dd25a4b6-b3e9-4fd3-8727-5a63c505888bn%40googlegroups.com.

[domini...@gmail.com]

I don't rely on test coverage, I use it as a simple measuring stick to see how much of the is executed (not tested!) by the tests. I then review files and tests to see where there might be value in adding tests.  It's also useful as a tool to find out if making a modification somewhere has the possibility of making a test fail or not, since if the of code or code path is not executed then there's no possibility a test will fail.

I have very different feelings on the pros and cons of writing tests, I don't really want to get into that in this email thread. However I will say my favorite programming environment is a pair-programming test-driven one, where you start with a functional test, and then implement vertical slices of code, expanding test cases and thinking about how it can go wrong, and write tests to cover these possibilities which yes, requires some critical thinking.  Tests allow newcomers to the project understand it, as good tests document the expected behavior of the system.

There's no silver bullet to software development, even with tests bugs and failures can still occur, but a well tested project, in my experience, has fewer bugs, fewer failures, and when failures do occur and changes are made you are more confident that the rest of the system still works.

I too have lots of experience with software development, I started in 1984, that's 40 years now (scary!).

Anyway, back to the problem in hand!

Yes, I'm still experimenting and learning the code, writing the tests has forced me to learn a lot about the internal of OpenPnP so if that's the only thing that comes from attempting to write a test then it's still worthwhile.  I haven't yet decided or even worked out a solution to the problem with the current ReferencePushPullFeeder.

Note that if the drag pin is moved slowly enough over the hole, then it will fall into the hole as it passes over the top of it, if an initial feed of 2mm doesn't expose a component, then a second offset feed will.

Other options I'm considering:

* not making any changes to the ReferencePushPullFeeder at all, since it's features don't really fit the CHMT 36/48 feeders that well, especially the OCR stuff in it.

* updating the ReferenceDragFeeder.

* making a new CHMT Drag Feeder specific system which has features that I want, with a UI that makes sense to the CHMT36/48 machines.  e.g. you define a feeder bank, tell it which side of the machine it is, and how many of each width feeders the bank has (they're milled out of aluminum, and never change), where the first and last drag holes are and it magically populates all the feeders and configures them.

I dunno, right now I just need something that works for me.

Dominic

[mark maker]

Also started 1984. Own company from 1993. Also scary. 😂

You answer with the right arguments and differentiation, that's good to hear.

Yes you could modernize the drag feeder instead, it specifically has a "0402 problem" only allowing it with "west" side feeders, or something. But it currently has a completely different vision approach (template image, no sprocket hole semantics).

You will still have to create a feeder per part, unless you want to reprogram the whole of OpenPnP. But like other feeders (ReferencePushPullFeeder, BlindsFeeder), you can auto-arrange them in a row (one-click setup) and/or auto-sync settings between them.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/85111b58-1a9c-4e0a-b2b5-532aeeb76727n%40googlegroups.com.

[vespaman]

A special chmt feeder, sounds like something I would have loved coming into OpenPnP, the setup as is, with the reference push  pull feeder is rather time consuming, doing all 48 feeders, and lots of thing can go wrong.

But now, having it all setup, the way I want it, I would not change since I use both sprocket hole vision, Y correction/adjustment on drag feed, as well as OCR (QR) on all feeders. But then again, I don't really have much problems with the 4mm feed on 0402 making the components jump.

Hm, I also started working in 1984. But of course I did a lot of hardware "development" before then, to my mother's despair. (e.g. ripping up carpets to place capacitive sensors under them, to detect humans getting close to my room. I'm sure what I thought was nice cabling by the age of 10 was not up to rest of family's standard :) )

 - Micael

[domini...@gmail.com]

A special chmt feeder, sounds like something I would have loved coming into OpenPnP, the setup as is, with the reference push  pull feeder is rather time consuming, doing all 48 feeders, and lots of thing can go wrong.

But now, having it all setup, the way I want it, I would not change since I use both sprocket hole vision, Y correction/adjustment on drag feed, as well as OCR (QR) on all feeders. But then again, I don't really have much problems with the 4mm feed on 0402 making the components jump.

yes, the sprocket hole and offset detection definitely useful.

Where and how are you placing the OCR labels on the feeders?  A photo would be very interesting, QR codes probably a good approach here.  Can you share some details of how you generate your QR code labels and how you get OpenPnP to read them?

 

Hm, I also started working in 1984. But of course I did a lot of hardware "development" before then, to my mother's despair. (e.g. ripping up carpets to place capacitive sensors under them, to detect humans getting close to my room. I'm sure what I thought was nice cabling by the age of 10 was not up to rest of family's standard :) )

heh,  I remember creating burglar alarms on solder-less breadboards to keep my brothers out of my room!  My dad brought surplus strain gauges home from his old work and they works good under carpets too :D  Funny!

I recall this book from 1982...

https://archive.org/details/30solderlessbreadboardprojectsbook1

Dominic

[vespaman]

A special chmt feeder, sounds like something I would have loved coming into OpenPnP, the setup as is, with the reference push  pull feeder is rather time consuming, doing all 48 feeders, and lots of thing can go wrong.

But now, having it all setup, the way I want it, I would not change since I use both sprocket hole vision, Y correction/adjustment on drag feed, as well as OCR (QR) on all feeders. But then again, I don't really have much problems with the 4mm feed on 0402 making the components jump.

yes, the sprocket hole and offset detection definitely useful.

Where and how are you placing the OCR labels on the feeders?  A photo would be very interesting, QR codes probably a good approach here.  Can you share some details of how you generate your QR code labels and how you get OpenPnP to read them?

OpenPnP has this already in the reference push pull feeder. Just set up the OCR region and print standard OCR labels of the component name. I also have the component name as text so both me and OpenPnP agree on what is loaded on each feeder. this video shows where I have put them, this sequence you can also see the sprocket hole vision on each feed on the left feeder. I think there's a earlier discussion regarding the OCR label holders. Let me see if I can find it! (Basically I am using Brother label printer with 9mm matte tape, on 3d printed "flags".

 

 - Micael

[vespaman]

No, did not find any relevant post regarding this, just very old stuff. So here's some pics;

While there are components that has jumped in this picture, this is not anything that normally happens during a full days run, but typically when I mess around with the machine, or change the spool etc.

(I think it has happened that component has jumped out of its pocket during a normal run, but this is rare. And there are other issues that are more frequent in my machine)

The "flags" are different for 12/16/24mm; (Also the labels has to be printed with the QR code on the left for the right hand feeders (otherwise the camera will not reach.

The OCR are (on my machine) in the same picture as the sprocket hole, so no extra move has to take place during startup check.

 - Micael

[domini...@gmail.com]

Hi Micael,

Those 'flags' are awesome, I need them for my machine, stat!

Do you have the cad models for those? I'd love to print up a batch.

Thanks,

Dominic

[Eladio Martinez]

Greetings,

Is there a STL file available to download for the OCR label holders?

--

You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/742f1507-09d9-4a97-8cb0-a146990ac5dfn%40googlegroups.com.

[vespaman]

Hi Guys,

Sorry for slow response..

I have now uploaded the OCR holders to thingiverse.

Also, I uploaded a starting point for a LED diffuser for the down cam, that I use with Jan's LED ring (own tweaking might be required on this one!).

Hope they will work good for you!

 - Micael

[domini...@gmail.com]

Hi Micael,

Great! Thanks for uploading them, much appreciated!  I'm sure those designs will be very handy.

Yes, the light right I made also needs some diffusion, I made one out of paper, but something more permanent would be nice so I'll look into your one.

Dominic