Vision Tree Refactoring

[Davydavedavington]

Hi Everyone,

I haven't been very active here as I'm still in the process of building my machine. I'm still waiting on parts so I've been coding to fill the time.

I've developed some of this locally, sort of as a proof of concept until it went a bit further. I'm far from finishing everything and I'm sure people would balk at a giant PR anyways so I thought I'd test the waters here (Perhaps someone else is also in the middle of doing this?). Maybe I'm not the right person for this either but here is my take on refactoring the bottom vision structure:

This is building on Mark's concept so most of this is redundant but I wanted to be clear on how I was going about things (https://groups.google.com/g/openpnp/c/7DeSdX4cFUE/m/5nCodBX3AQAJ)

Summary

• Basically what Mark proposed, added a "Part Analysis" tree within Machine Setup > Vision to hold configurable instances of PartAlignment derived classes.
  
• Mutating the interface of PartAlignment a bit:
  • Add a "compatible cameras" tab in the machine setup PartAlignment config - this would be used by the job processor instead of hardcoding 1 bottom camera (room for improvements here for camera selection method, etc).
  • Begin moving the primary burden of configuration from the part to the package and the  vision pipeline out package/part area entirely. When editing a package, the PartAlignment provides wizard to configure parameters in the package and a preview of the part in the case of my QFP class (This would replace the Vision tab shown below but I currently just added one called Analysis):
  • 
    
  • When editing a part, PartAlignment classes retain the method for providing a wizard but should be reserved for absolutely necessary things.
• Created LeadGroup model (properties: name, x, y, angle, leadRows, leadColumns, width, length, pitch, skippedLeads) to describe groups of leads in a more easily manipulatable way in package creation.
  • Not replacing Footprint - used for representing the part leads in vision rather than the PCB footprint - Unless I'm misunderstanding the purpose of Footprint
  • Does not care about lead shape (Or maybe should have rectangle/circle because of BGAs?)
  • Named so that PartAlignment wizards can easily grab the correct lead groups internally
  • Better alternative to individual pads because of verbose and bloated xml
• Added array of LeadGroup to Package model
  
• Providing more Part Alignment classes
  • Goal is only 1 pipeline per alignment instance (rather than generating multiple pipelines for parts and PartSettings instances)
  • ReferenceBottomVision - Slightly refactored to preserve all of the per part pipelines and pre-rotate settings while presenting the same unchanged interface on the Part -> "ReferenceBottomVision" tab
    • Make the tab disappear when not the selected PartAlignment in a  package
    • Need a "tidy up" method for when switching to and from this in the package so it doesn't leave a lot of dangling pipelines in the xml
  • QFPPartAlignment - Assumes 4 lead groups named R1,R2,R3,R4 along the 4 sides of the body and shows a preview of the expected part
    • Can actually be used to represent the majority standard parts - may need a better umbrella name
    • Checking "Total dimensions" toggles placing leads inside chip outline for things like QFNs
  • ChipPartAlignment - Assumes 2 lead groups of only 1 lead each and places the lead groups inside the body border
  • SimplePartAlignment - Makes no assumptions about leads and simply recognizes the part as a rectangle as normal using the single pipeline - Would be the "new" default for people who have their pipelines dialed in for part outline instead of leads
  • LaserPartAlignment - Who knows? I don't
• Migration strategy would consist of:
  • Assign all existing packages to an instance of ReferenceBottomVision and make sure settings are kept to ensure smooth transition
  • SimplePartAlignment would become the default for new packages

I realize the addition of lead analysis info is somewhat pointless for alignment, but when dealing with large SOTs and QFPs: I love the damaged lead rejection on my commercial machines. To me this seems even more valuable to lay the framework for diy machines since people are manually loading ICs into 3d printed trays. I would've used the Footprint model but I got the impression that people were exporting these directly from their PCB design software so its the PCB footprint not the component. Anyways, I could split all the extra PartAlignment classes and LeadGroup model into a separate PR too. The functionality just happened to get intertwined in my PartAlignment derivative classes while hashing out the UI.

Note: I kind of broke from the Bottom Vision terminology because honestly... its  a confusing name in this context. Bottom vision referring to the geometry of the machine when all a new user is looking for is how to configure how their parts are recognized (You can see I split out nozzle tips in the tree along with Board marks, that's a whole other thing though). One of my long term plans is needing a "Side camera" for part alignment which also throws a wrench into the usage and is what prompted my adventure into this.

Thoughts anyone? Idk I've been rewriting this for an hour, so hopefully it makes sense. I'm open to changes as I'd really like this feature to be usable when I'm finished. I've read the other thread and have some ideas on how to address the concerns raised there but not in this initial refactoring

David

[ma...@makr.zone]

Hi David

> I've been rewriting this for an hour, so hopefully it makes sense.

Only an hour? I usually takes me much longer for such emails. ;-)

But yes, most of this does make sense.  I kind of filled in some gaps with what we discussed in the other thread.

Some thoughts (numbers for easy reference):

1. I assume you saw in the other thread, that "moving the primary burden of configuration from the part to the package" (that I also proposed) is not what Jason envisions, he'd rather entirely remove the Package and just keep Parts. I guess there needs to be a solution that addresses both visions. Let's call this Aspect A.
2. Furthermore there is the idea by Jason and others that Jobs should freeze a copy of all the settings so they remain reproducible, when the main library is later changed with other jobs. Let's call this Aspect B.
3. I was thinking about a universal "Part Settings" object and Wizard that can be attached to whatever you like: Part, Package, Part shadow copy inside a Job, Package shadow copy inside a Job. 
4. When such a "Part Settings" objects is not attached, it can be inherited from the more general object. And/or it can be frozen as a shadow copy in the job.
   
5. Then make the GUI configurable (Aspect A): "I only want to manage Parts", "I only want to manage Packages", "I want to manage Packages mostly, but sometimes make overrides on Parts".
6. In the former two cases the surplus tab would vanish from the GUI and simplify things greatly. As these tabs are now very similar (both just list attached "Part Settings") this would be easy.
   
7. The third/mixed case would require some buttons to sync one setting to the other. Like "Take this Part setting and make it the Package default", "Remove these Part specializations and revert to the Package default" (the latter ideally with multi-selection).
8. Then make the GUI configurable (Aspect B):  "I want everything kept consistent and central across all Jobs", "I want to freeze setting inside the Job and if I tweaks things, it should only apply inside that Job".
9. Again buttons to sync: "Make these Job Part/Package setting the new library default", "Abandon these Job specific settings and revert to the current library default" (again ideally with multi-selection).
10. Your "LeadGroup model" is cool, but I do think it should generate into the current Package Footprint, not add a second feature.
    
11. Like you said, the E-CAD-imported footprints do usually refer to solder pads (cream layer) and not pins etc. so the shape is not exactly right for computer vision, but I made some experiments with blurred Template Image matching and this seems to work. So adding your generators as an alternative, but otherwise keeping these two things the same would be the right way, IMHO.
    
12. But I would make the generator independent from the vision class. So different generators can be combined with different vision classes.
13. One example is multi-shot partial vision when the part is larger than the camera view. So the nozzle has to move above the camera and expose multiple corners of the part. Your generator would be great to define the expected partial geometries. It should therefore not be bound to just one vision class.
    
14. Another solution is auto-learning part images. You could capture your first real part on the nozzle and ask the user if everything is alright and properly aligned. From then on you can use template image vision matching and add a score threshold to detect bent pins etc. This is IMHO the only practical way to do it, as synthetic templates deviate a lot from a real image, one bent pin will likely drown in the mismatch noise.
    
15. But auto-learning part images won't easily work with multi-shot, so the generator would still be handy.

Wanted to write more, but gotta go ;-)

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/777f93cc-5e2e-4ef8-9b06-f7a653a8da2an%40googlegroups.com.

[Davydavedavington]

Well technically I wrote the line about rewriting for an hour... at the 1 hour mark lol.

Thank you for the list, I cut down my post quite a bit to reduce the initial scope for a PR but I had been marinating on some of these:

1. I find that vision... troubling and would be an deal breaker for a lot of people coming from commercial systems (Universal, Mycronic, Europlacer) who wouldn't put up with it. The merging of parts and packages in the backend kills a valuable timesaving tool to only fix a tiny problem and introduces many more. I imagine if someone put it to a community vote they would find that this is by far the less popular architecture. This is straight up not an option for someone wanting to drive a high capacity - high output machine. The only way this would be acceptable is if the Part -> Package relationship was hidden and forced 1 to 1 for the user if they asked for it.
2. I'm not a big fan of this since it makes the job far less portable and you can't move a program to a physically distinct machine without a ton of extra funkiness to handle. You then need a dozen band aid features to deal with how changing multiple parts is now difficult and you have to do every single change for every single job. 
   
   Regardless, I think a simple middle ground would be some changes to the Configuration class so that packages and parts are loaded from both the local machine xml files and whatever job is loaded. Packages from the job could namespace the ID with something like JOB::RES-0603-1K-5% and be highlighted in a different color so you can differentiate it from the parts.xml one. A typical scenario I would imagine would be having an annoying part or package, selecting it in the table, pressing the "Copy to local" / "Copy to local and replace in job" and then working with that one (And screwing over the next guy who has the part loaded in the machine lol). Presumably when the job is unloaded, feeders and packages would have to fall back to the local version
   1. A more generic solution that might have been posted before would allow all sorts of crazy databases and stuff would be to have multiple DatabaseProviders that namespace their part/package IDs and then hands off their part and package lists to the Configuration class - and so we just have a JobDatabaseProvider that hides all of the job freezing. There are a few problems for when a database dissappears... but there should be good error handling for broken relationships either way because you have to let people delete things.
3. Part settings kind of sounds like you're describing a mini Package that you can shuffle around, why not just use Package? It sounds like you're renaming it to hide it from Jason ;)
4. Polymorphic packages, now we're talking :D
5. Agreed, though this feels like getting into a "Workspace" PR territory.
6. See below
7. See below
8. See below
   
9. I think I understand your concept for the above points but imagining implementing it feels like it would generate a tremendous amount of tangled business logic and testing from all the edge cases to keep a minority of users happy. Its far easier to hide a regular Part -> Package relationship behind a UI that Jason is happy with than to complicate what is normally a very simple data relationship. Either way, I don't think the changes I proposed at the top would get in the way of any of this, its just adding a wizard and package-centric part alignment options. The getPartConfigurationWizard(Package pkg) method would still be there for a PartSetting based PartAlignment + wizards.
10. Agreed, this house isn't big enough for 2 graphical representations of a part, but Footprint isn't the right name at all for lead matching at all so iunno how to reconcile it.
11. My inner "Don't over engineer it, its good enough" being wants to agree with you but I would say its a gross oversimplification that will lead to confused users who import footprints that don't match their parts. You can't bundle a fresh install of software with a database of footprints and call them kinda-good-enough-leads. It'd work for chip resistors and small stuff but the grand majority of footprints vary widely from their component measurements even in the datasheets (DPAKs, QFPs, and QFNs).
12. I'll move the PreviewPanel out into the GUI support area, but the entire generator UI is just a small wizard with helper methods that binds fields directly to LeadGroup fields. I suppose you mean creating something like a PackageFamily or PackageClass?
13. You read my mind lol, I deleted that example because I thought people would say that is way too far down the road functionality wise lol - a BigBadPart class can just use the same wizard but I guess in the backend would stitch 2 images together like in the preview here:
     
    
14. Personally I've found image template matching more difficult to troubleshoot because there's only so much you can do before you're forced to just bring the pass score down in a production environment - then its not really doing its job.  But I'm glad PartAlignment is already a separate object so people can develop both and we don't have to choose lol.
    Also I didn't want to muddy the topic initially with this but LeadGroups were also partly a choice for allowing a more robust lead recognition algorithm based on taking an ideal line along the lead group and then measuring the spacing and quantity of intersecting detected edges (this is after the initial alignment).
15. Indeed

Phew, this topic is exhausting lol. I can see why people don't want to touch this barrel of worms lol

David

[Jason von Nieda]

Hi David and Mark,

IMHO this thread is *already* too complex three posts in :) There are way too many changes being discussed here than we could ever reconcile into a set of changes that should all happen in a single PR.

I strongly suggest we split this up into separate threads - or even better, choose a smaller feature to focus on first, flesh it out, and get it implemented before trying to change everything at once.

I don't think the email list is a good venue for discussing massive refactoring like this. It gets very difficult to answer individual points and they all get mixed together. Perhaps it would be better to have a living document - a Wiki page perhaps - to serve as a design document and people submit PRs to it to come up with a final design? Another option would be to use the new Discourse site, create a topic, and keep the first post updated with a set of features and use the discussion to edit that post as needed. Just something to consider...

To me it sounds like the PartAlignment changes being proposed stand entirely on their own without any changes to the relationship between Part and Package - so perhaps we can leave that much larger discussion out of this one and just focus on the PartAlignment changes as an initial feature.

So, with that all in mind:

- Footprint is a misnomer and it should go away and/or be changed to exactly what is being described here. I never meant for it to have anything to do with PCB lands. It is intended to represent the parts/packages/chips/components/etc, not the PCB. So, yes, please replace/supplant Footprint.

- It's not clear to me where the ReferenceQFPAlignment, ReferenceChipAlignment, etc. fit in the vision system. Are these essentially subclasses of BottomVision? i.e. are they responsible for performing the vision operation, moving the head around, taking pictures, etc. or are they just for describing the part in such a way that ReferenceBottomVision can align it properly?

- LeadGroups are interesting, but as Mark said I think these should essentially be generators into Footprint, after we come up with a new name for it. Unless I am misunderstanding the goal here, it seems like LeadGroups would fill the same purpose in a different way as the Package Editor I was working on a while back: https://twitter.com/openpnp/status/1202804241020149760. In other words, the goal is to make it easy for the user to generate a package definition that can be used for advanced vision operation and visualization. Whether that is via LeadGroups, drag and drop, CAD import, cloud sharing, etc. shouldn't matter.

So, before we get too much further into this, can we limit the scope a bit?

Thanks,

Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/c7ea3316-2f96-467d-b68e-6280039e5448n%40googlegroups.com.

[ma...@makr.zone]

The writing of this response overlapped with Jason's mail*, so it does not yet respect that fully. I do agree with Jason that we should try and separate the topics as far as possible. At the same time I do see that it may be not so easy, the dependencies are complex.

*) Earlier writing:

> Part settings kind of sounds like you're describing a mini Package that you can shuffle around, why not just use Package? It sounds like you're renaming it to hide it from Jason

No, that's not what I mean. This is about inheritance/polymorphism, principles that I like very much (I'm not a fashion victim and no, I don't think OOP is "out", just because it was not the silver bullet they hyped it to be). Part Setting could be a "Base class" of Part and Package inasmuch as both can carry settings that ultimately affect the part. We could endlessly battle about where which settings should be located and whether two levels are needed at all. Or we could just let go and say everybody can choose a setup to their liking. Even the mix is possible, with an optional override logic.  In my professional life I have ample very positive experience with that settings inheritance pattern in much more complex hierarchies. After some getting used to, users love its power.

> it feels like it would generate a tremendous amount of tangled business logic

I don't think so. Essentially I have implemented as much in the ReferencePushPullFeeder class, where you can mark one feeder as a template and based on its Package (there is even a second level to further group Packages called "Tape Specification") it will then serve as the blueprint for the other feeders. You can also re-clone the settings, after you change the template. This is a less formal embodiment of essentially the same pattern.

https://makr.zone/openpnp-new-referencepushpullfeeder/393/

This way (and thanks to OCR) you can create a new feeder with one click:

https://youtu.be/5QcJ2ziIJ14?t=295

Further into the video you see the cloning in effect.

https://youtu.be/5QcJ2ziIJ14?t=474

The envisioned relationships and cloning operations between Packages/Parts and Library/Jobs would be very similar. At least to my standards this is in no way "a tremendous amount of tangled business logic" ;-)

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/c7ea3316-2f96-467d-b68e-6280039e5448n%40googlegroups.com.

[ma...@makr.zone]

Sorry the second video timestamp was wrong:

Should have been

"Further into the video you see the cloning in effect."

https://youtu.be/5QcJ2ziIJ14?t=404

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/d62c2a04-fe00-a5b6-7f18-85ee58117b4c%40makr.zone.

[Davydavedavington]

@Jason:

Absolutely agree lol, I pruned back everything I could in the first post to try to follow your post in the other thread about gradually introducing changes

Scope-wise, I really just want to concern myself with the PartAlignment interface/ui stuff in the first post, it shouldn't hurt anyone and would make it easier for people to experiment with different implementations. Even the LeadGroup part isn't necessary but it was a cool shiny thing.

>> Footprint is a misnomer and it should go away and/or be changed to exactly what is being described here. I never meant for it to have anything to do with PCB lands. It is intended to represent the parts/packages/chips/components/etc, not the PCB. So, yes, please replace/supplant Footprint.

Cool cool, though I can't think of a proper name to encapsulate LeadGroups or Leads lol. Is "Outline" too obscure? Taken from SOIC, SOT, etc

>> It's not clear to me where the ReferenceQFPAlignment, ReferenceChipAlignment, etc. fit in the vision system. Are these essentially subclasses of BottomVision? i.e. are they responsible for performing the vision operation, moving the head around, taking pictures, etc. or are they just for describing the part in such a way that ReferenceBottomVision can align it properly?

Subclasses yes or other derivatives of PartAlignment. Some components types are better suited for different alignment methods, different cameras, etc. From the code it looks like that was already the intention to extend it this way (minus the different camera part). I don't really see this as a big change but since its changing some methods and interfaces I called it a refactoring.

>> LeadGroups are interesting, but as Mark said I think these should essentially be generators into Footprint, after we come up with a new name for it. Unless I am misunderstanding the goal here, it seems like LeadGroups would fill the same purpose in a different way as the Package Editor I was working on a while back: https://twitter.com/openpnp/status/1202804241020149760. In other words, the goal is to make it easy for the user to generate a package definition that can be used for advanced vision operation and visualization. Whether that is via LeadGroups, drag and drop, CAD import, cloud sharing, etc. shouldn't matter.

Absolutely

@Mark:

Without writing too much more I do see the benefits of your points and look forward to seeing how it would unfold in a separate PR :)

David

[Jason von Nieda]

Inline:

On Sun, Apr 4, 2021 at 3:26 PM Davydavedavington <m...@davidgaul.com> wrote:

@Jason:

Absolutely agree lol, I pruned back everything I could in the first post to try to follow your post in the other thread about gradually introducing changes

Scope-wise, I really just want to concern myself with the PartAlignment interface/ui stuff in the first post, it shouldn't hurt anyone and would make it easier for people to experiment with different implementations. Even the LeadGroup part isn't necessary but it was a cool shiny thing.

>> Footprint is a misnomer and it should go away and/or be changed to exactly what is being described here. I never meant for it to have anything to do with PCB lands. It is intended to represent the parts/packages/chips/components/etc, not the PCB. So, yes, please replace/supplant Footprint.

Cool cool, though I can't think of a proper name to encapsulate LeadGroups or Leads lol. Is "Outline" too obscure? Taken from SOIC, SOT, etc

I'm seeing "Package Outline" and "Mechanical Data" in data sheets. I think Outline would be good.

>> It's not clear to me where the ReferenceQFPAlignment, ReferenceChipAlignment, etc. fit in the vision system. Are these essentially subclasses of BottomVision? i.e. are they responsible for performing the vision operation, moving the head around, taking pictures, etc. or are they just for describing the part in such a way that ReferenceBottomVision can align it properly?

Subclasses yes or other derivatives of PartAlignment. Some components types are better suited for different alignment methods, different cameras, etc. From the code it looks like that was already the intention to extend it this way (minus the different camera part). I don't really see this as a big change but since its changing some methods and interfaces I called it a refactoring.

I may just have to wait till you are further along to understand this well, but is the idea that these would be very light subclasses? Mostly I just don't want to see a bunch of code in ReferenceBottomVision duplicated.

In a perfect world, I think ReferenceBottomVision (or something similar but new) would just identify leads and try to match them to a given Package's leads. Maybe subclasses would provide "smarter" matching, but the vision operations would be the same, I think.

Thanks,

Jason

[ma...@makr.zone]

> but is the idea that these would be very light subclasses? Mostly I just don't want to see a bunch of code in ReferenceBottomVision duplicated.
In a perfect world, I think ReferenceBottomVision (or something similar but new) would just identify leads and try to match them to a given Package's leads. Maybe subclasses would provide "smarter" matching, but the vision operations would be the same, I think.

I do see a larger variety but that's not a problem IMHO, as we can do both: light subclasses and completely new implementations of the interface. Also there can be new abstract base classes inserted above ReferenceBottomVision to cover common settings & code, such as the pre-rotate option and multi-pass centering.

These are the variants on my radar:

1. MinAreaRect bounding box alignment (same as today). Size Check could be auto-learned.
   Pros: no outline data entry/import required.
   
2. Outline matching with E-CAD imported or manually entered/generated contact outlines.
   Pros: completely semantic, i.e. independent from camera, lighting etc., outline data could be community shared.
   
3. Outline matching with multi-capture for parts larger than camera view (subclass of 2): nozzle moves part above camera to expose configurable corners/edges (perhaps a 3 x 3 matrix of checkboxes). 2 and 3 could also be rolled into one class, i.e. the center exposure would simply be the single selected checkbox.
   Pros: same as 2 plus large part support.
   
4. Template image auto-learning. When a part/package is first encountered, the job is paused, the user is shown the capture in the camera view, and led through a process to interactively check/adjust the alignment, OK the pin 1 orientation, and OK that no bent pins are present. Perhaps as a new Machine Controls tab with the right selection controls.  The image is then learned as a template. Subsequent alignments will use the template to match position/rotation, the final match score is also used to detect bent pins, pin 1 markers (even informal body markings can be used to detect orientation) etc.
   Pros: no outline data entry/import required, 360° orientation support, robust "good part" check.
   

There would be three different pipelines involved.

It might even be feasible in the future to do some clamp or laser/LIDAR alignment that involves no vision at all. The interface should be completely agnostic there.

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/CA%2BQw0jyebAn%2BR%2BtU1ym-f5H5kpuA9Nh379W%3D%3Dqs3M8fD928f9Q%40mail.gmail.com.