eeschema: allow Kicad to talk to a database using EXTERNAL plugins (dll, so etc)

[Andre Iwers]

Hi There!

I constantly run into issues where I need to add all sorts of metadata to my sometimes 100+ parts schematics which are saved in an external inventory system. Initially doing it by hand I grew tired of it and decided to whip up a slightly less laborious way using a plugin approach which is quite flexible in terms of datasources. Also updating those is even worse.

I was interested if this is something Kicad would be interested in taking over for the broader public. I hooked it into the 'Symbol Properties' or 'Symbol Fields Table' dialog for ease of use and touched that existing code only marginally.

Quick Overview:

When I have selected a schematic symbol from the standard library, I'd like to be able to open the 'Symbol Properties' or 'Symbol Fields Table' to fetch part information from an external source.

Idea: Eeschema would simply provide a framework which connects to one or more plugins and the plugins themselves would be provided by the community as .dll or .so which then are saved somewhere on the computer and simply loaded when needed. The plugin can be programmed in any language such as python, java, c# etc. and would only need to honour the defined C wrapper interface. Plugins which don't adhere to that are simply ignored.

I have created a first draft of that framework to solve a laborious task I face once in a while and am wondering if this is something that is generally useful for the broader public. I also programmed a plugin for Inventree which is used here as part database.

In general: the framework is intended to use multiple plugins at the same time so that a user could potentially query for example Mouser, Digikey, Navision and Inventree at the same time.

The workflow:

A brand new project would be to:

• select a symbol from the standard kicad library
• open plugin search and search for the specific part
• transfer all metadata automatically into your schematic

Existing project:

• Open fields editor and bulk apply metadata using the above mentioned approach.

[Jon Evans]

Hi Andre,

I think it would be best to have two different discussions here, first about the idea and concept, and second about your proposed implementation.

I wanted to lead with this because I think there is a tendency in discussions to mix the two together unnecessarily.

This is a real need and something that we've already started working on related things for, although in a slightly different direction than you describe.  There are two different but related concepts here: 

1) Storing part metadata in a private database -- I have started working on this feature (#7436).  The scope of this one is that the database is SQL-like and ODBC-compatible.

2) Retrieving live part metadata from a public data source (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to already-placed symbols (or newly-placed).

(1) can obviously be built into KiCad and so it will be.  (2) is more challenging -- most of these data providers would be unhappy with KiCad providing an easy way for users to scrape their data, as it could cost them a lot of money.  There have been discussions with various data vendors about an integration approach, but in general this is something that needs to be negotiated per-vendor.

Because of this, I agree that the right approach is to build a framework rather than an implementation, which leaves e.g. Digi-Key open to providing their own KiCad data plugin if they wish to make their data available to KiCad users in this way.

This leads me to the implementation -- I'm not sure that creating a new binary plugin interface is the approach we should take for this.  We have been investing in the Python plugin ecosystem elsewhere, and this seems like an ideal fit for Python: people creating datasource plugins can distribute a single plugin that runs on any platform KiCad is supported in, and there are fewer stability and security concerns with loading an arbitrary Python plugin than with an arbitrary compiled binary.

For the workflow you describe: this makes sense in the use-case of adding metadata from an external source to parts that were already placed.  The Database Libraries workflow will be oriented more about placing symbols with all their metadata from the start.  Perhaps it would be possible to have this proposed metadata plugins interface work either at initial placement or afterwards?

Best,

-Jon

--
You received this message because you are subscribed to the Google Groups "KiCad Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to devlist+u...@kicad.org.
To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/CABQVKmKZAQtXZ-gdQOJiHGH_vJmwfU5Ho%3DCAaMtM_YD2PbZteQ%40mail.gmail.com.

[Seth Hillbrand]

Just to add on to what Jon has already said:

KiCad is a cross-platform application, so any solution needs to similarly be cross-platform.  If the target for the external plugin is to allow users and companies to develop their own interfaces that can be used in KiCad, then this needs to be done in a scripting language.

There are a couple reasons for this.  The big one is that MacOS cannot load a binary plugin from outside the signed application without triggering GateKeeper and all of the badness that comes with that. 

We can build/distribute binary plugins (like the 3d plugins) that are included in our codebase because these get signed with the rest of the application.  So, if the target is to allow ODBC interaction, we can probably support a binary plugin interface that explicitly does that. I think that this is (#7436)

Seth

To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/CA%2BqGbCBOVz_8CJLG5spNgg%3D%3D6RF4s390abE_LW8wQeeE5-A%3DdQ%40mail.gmail.com.

--

Seth Hillbrand

Lead Developer

+1-530-302-5483‬

Long Beach, CA

www.kipro-pcb.com    in...@kipro-pcb.com

[Andre Iwers]

Hi Jon & Seth, 

Thanks for sharing your thoughts!

Some thoughts here from my side:

1. Storing part metadata in a private database -- I have started working on this feature (#7436).  The scope of this one is that the database is SQL-like and ODBC-compatible.

1. • I am not sure if this relying on ODBC is flexible enough. In my case it would not be. I mostly work with either Navision or Inventree which stock data is managed by other people. In my case I rely heavily on their stock/metadata and I don't have direct access to the source. It's read-only. Having an ODBC compatible interface would mean that every software you'd need to work with must support that, or am I wrong here?
   • My approach tried to avoid exactly that. I have a driver which does all the translation work between kicad and the inventory system and the user can design that plugin as needed. Kicad would only impIement that once and that's it; there is no need to touch the source again when changing the plugins itself.
   • I admit that a compiled library might not be ideal as kicad is cross-platform, however, one could use a simple inventory_plugin.py file or archive which is digested by kicad and then use a python interpreter.

1. Retrieving live part metadata from a public data source (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to already-placed symbols (or newly-placed).

1. • The above approach could be used here too. Digikey provides API access and everyone can get their own API key.

Please let me know what you think.

Cheers,

Andre

[Jon Evans]

The ODBC interface is designed to support pulling component libraries from the database, not adding metadata to parts that are from other libraries.  So it's a slightly different (but related) use case.  At the moment there are no plans to support anything other than ODBC.

The use case of adding metadata to existing parts that come from other libraries is what you seem to be heading towards.  This one I would recommend we look at solving with a cross-platform plugin system of some sort.

-Jon

To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/2838e643-a03c-4cdb-8853-3228bfbd19d0n%40kicad.org.

[Andre Iwers]

I assume that this will be some sort of library format and users would be able to select this in here right?

I would be more than happy to use a non-compiled cross-platform plugin system. What's important to me, and what makes my and my colleagues life easier, is that it is well integrated and offers a unified interface; similar to what I showed here: https://gitlab.com/kicad/code/kicad/-/issues/12027

Apart from this, I am more than happy to use kicad's internal tools to create symbols and footprints and place those using the currently available facilities. In fact, currently our meta data also contains the right footprint string which reduced accidental placement of the wrong footprints a fair bit; it's also a QM thing.

[Jon Evans]

Yes, the database library would be a type of library format (exact UX is not yet determined).

-Jon

[Symdeb]

There were several conversations in the past on  Kicad Info, Inventree Github and Kicad gitlab. 

Here is a functional specification proposal for an API instead of ODBC. 

This built on  top the proposed search for parts feature on gitlab , as follows

1. Searching for parts from any resource . The API implementation needs to be provided, not in scope. Specification adds request/response proposal
   
2. Change (version) management between CAD and remote systems for footprints, symbols, models. Specification adds request/response proposals
   
3. Part to symbol/footprint/model mapping between remote system and the CAD. Specification adds request/response proposals ad version control logic.
   

"remote system" term is used here because it could be a PDM, PLM , home brewed python server script to a local or remote DB etc..

References:

https://github.com/inventree/InvenTree/discussions/4133

https://docs.google.com/document/d/1qXKEa9Djz1VnEzH7jK0rJ1p9Vu82tq4p_m2zBuNMG6k/edit#heading=h.fyuo1bpfgubj

https://gitlab.com/kicad/code/kicad/-/issues/12027

[Jon Evans]

There are a lot of different concepts and ideas here being thrown around.

First of all, to reiterate what I said on the linked issue:  No REST API will be accepted "instead of" ODBC.  The existing ODBC system will continue to exist as-is to support users for whom it is a better solution than REST.

Second, the scope of any project being developed by someone who hasn't worked on KiCad before really should be limited, as it will take time for someone to learn the codebase and build a good working relationship with the rest of the team.  To that end, I do not think it is in scope to consider any functionality not already supported by the KiCad library system.  So, change management / version control are _not_ something I would worry about at this point when defining the API interface with KiCad.

If someone wants to build a change management / version control system outside of KiCad (as part of whatever web service is exposing this proposed REST API) that is fine.

If you or anyone else is serious about pursuing this, I would recommend the next step is to continue to build out the API proposal document, based on APIs required to build a KiCad symbol library plugin.  Once we are in agreement on that, a proof-of-concept could be built for testing and review.

-Jon

[Symdeb]

The word  "instead" was probably a bit poorly chosen. Surely, retaining ODBC makes sense to make since it's already there and people are using that.

No objection on plugins or any variant of solution. As any proposal , the document was just for feedback, discussion, consideration. Nothing  set in stone.

It was kept as generic as possible w/o restricting the type of implementation.

[Symdeb]

One  comment on this:

most of these data providers would be unhappy with KiCad providing an easy way for users to scrape their data, as it could cost them a lot of money

Actually, Mouser , Digikey, Element14, among a few other already provide this since a long time for free but with a limit on maximum number of calls  or records retrieved per day,  based on the user's registered account. Though this quota is  sufficient say ~100 searches per day.  Recently  LCSC stopped their free service and only allows customers with a  VERY VERY LARGE order  amount history. (think CM's), Other retailers require  paid subscriptions.  These services often  involve OAuth2. and that probably is beyond scope for KicAD and even many users as it would need to manage call backs, client secrets, access token, refresh tokens etc...That is a complex affair.

Therefor I documented in the proposals that an "intermediate" service running locally or on a remote server implemented by the user (not kicad)  to handle the transformation 

between the call to the API and the service's data and also in there manage authentication (call backs, client credentials, access token and refresh tokens etc..). This part could e provided by external open source projects and out of scope for KiCad. KiCad just makes the API call  to that "intermediate" service based on the KicaD API spec.

[Andre Iwers]

Just to keep this discussion alive. There is a prototype now which allows KiCad to talk to an external REST API.

https://gitlab.com/kicad/code/kicad/-/merge_requests/1679