Improvements to the http library

[Rosy]

Hello everyone

I currently use KiCad privately and would like to use it in our company in the future.

It is important that there is a good solution to manage the libraries centrally. The http library is ideal because it enables a clean separation of frontend and backend.

Unfortunately, there are still a few things that I think need to be improved a bit.

I have currently created a merge request with which it is now possible to provide the http sublibraries with a description, which I consider to be an essential function and hope that it will be accepted promptly.


I have also recorded the following issues regarding the http library.

https://gitlab.com/kicad/code/kicad/-/issues/17584

https://gitlab.com/kicad/code/kicad/-/issues/17570

https://gitlab.com/kicad/code/kicad/-/issues/17569

For issues 17584/17570 Andre Iwers and I have submitted the merge request
1910/1912/1914
A possible solution has been developed, which is currently being discussed in MR 1910, as opinions differ on fields that are only displayed in the Choose Symbol dialog.
What is your opinion?
Otherwise, I would be happy to work out another solution.

Does anyone have some tips for where and how I could tackle problem 17569?

Thank you for your feedback.

[Rosy]

In the meantime I have familiarized myself with the existing solution and the code. It is now clear to me what the problem with 17569 is.

I would be happy to invest time in revising the http library so that the various problems can be solved, but I would like to know who I should talk to so that the work is not in vain but is then accepted?

[Jon Evans]

Hi Rosy,

> I would be happy to invest time in revising the http library so that the various problems can be solved, but I would like to know who I should talk to so that the work is not in vain but is then accepted?

This list is a good place, please say more about what you propose to
do before you start work.

I have been away from KiCad this past week and haven't had the chance
to look at the MRs yet. Maybe other developers will also chime in
here.

-Jon

> --
> You received this message because you are subscribed to the Google Groups "KiCad Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to devlist+u...@kicad.org.
> To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/a801fafd-930f-4b75-b718-2e706a06a0efn%40kicad.org.

[Rosy]

Hello everyone

I found a solution for issue  https://gitlab.com/kicad/code/kicad/-/issues/17584 that neither changes the behavior of the library nor affects the performance. To do this, I recorded the two merge requests:
master: https://gitlab.com/kicad/code/kicad/-/merge_requests/1914
8.0: https://gitlab.com/kicad/code/kicad/-/merge_requests/1925

I have also developed a solution for issue  https://gitlab.com/kicad/code/kicad/-/issues/17569 , which occasionally checks before a symbol is loaded whether the cache has already been created and, if not, created first.
To do this, I recorded the two merge requests:
master: https://gitlab.com/kicad/code/kicad/-/merge_requests/1927
8.0: https://gitlab.com/kicad/code/kicad/-/merge_requests/1926


I think it's important that dynamic libraries like the http library can also add descriptions to the sublibraries. To do this, I recorded the following merge requests:
master: https://gitlab.com/kicad/code/kicad/-/merge_requests/1921
8.0: https://gitlab.com/kicad/code/kicad/-/merge_requests/1919

I've had various thoughts about the issue https://gitlab.com/kicad/code/kicad/-/issues/17570 and my other ideas.
One of the problems with the existing solution is that certain requests return an array, making it almost impossible to expand.
I could imagine shouting a v2 of the http library. Attached I have a document that describes how I imagine this.

I would be happy to receive your feedback.

On Monday, April 22, 2024 at 5:09:39 PM UTC+2 Rosy wrote:

My goal is that the library remains compatible with existing APIs but that the various issues can be solved. I think it is important that the same functions as the standard functions are available for the http libraries. The issues mentioned show where this is currently not possible.
I also find it an important function that a description can be given for the categories, as is the case with the standard libraries. I haven't entered an issu for this but have already added an MR.

Something that I noticed and that I would certainly like to change is that at the moment the parts are stored in different caches with different loading states.

There should only be one part cache.

I currently have the idea that another key called “fields” can be returned by the API when the root request is made. Only if this is sent by the API will the new features be activated, then the field list can be preloaded via a new endpoint. This enables short field keys even for fields with long names.

I would also like to introduce modify fields that allow only those parts that have undergone updates to be reloaded.

The communication could look something like this:

# root
{
"categories" : "",
"parts" : "",
"fields" : ""  //activate new fields  features 
}

# fields.json
{
"reference":{
"show" : true,
"default_show_name" : false
},
"value":{
"default_show" : true,
"default_show_name" : false
},
"footprint":{
"default_show" : false,
"default_show_name" : false
},
"datasheet":{
"default_show" : false,
"default_show_name" : false
},
"description":{
"default_show" : false,
"default_show_name" : false
},
"keywords" : {},
"mfn" :{ //must be unique
"name" : "Manufacturer Number", //must be unique
"show_on_chooser" : true, //only needed if not false
"default_show" : false, //only needed if not false
"default_show_name" : false //only needed if not false
},
"mfurl" : { //must be unique
"name" : "Manufacturer URL", //must be unique
"show_on_chooser" : false, //only needed if not false
"default_show" : false, //only needed if not false
"default_show_name" : false //only needed if not false
}
}

# categories.json
[
{
"id" : "1", //must be unique
"name" : "Resistor", //must be unique
"description" : "Thick Film SMD Resistors",
"modify" : 1713798563 //datetime seconds since 1.1.1970
},
{
"id" : "2", //must be unique
"name" : "Capacitor", //must be unique
"description" : "SMD MLCC Capacitors",
"modify" : 1713798563 //datetime seconds since 1.1.1970
}
]

# /parts/category/{category_id}.json
{
"parts":[
{
"id" : "1", //must be unique
"name" : "MF-ab123", //must be unique
"modify" : 1713798563 //datetime seconds since 1.1.1970
"value" : "12k",
"description" : "Tick Film SMD Resistor 12kΩ, 1%",
"keywords" : "12kOhm",
"mfn": "ab123"
},
{
"id" : "2", //must be unique
"name" : "MF-ab102", //must be unique
"modify" : 1713798563 //datetime seconds since 1.1.1970
"value" : "1k",
"description" : "Tick Film SMD Resistor 1kΩ, 1%",
"keywords" : "1kOhm",
"mfn": "ab102"
}
]
}

parts/{part_id}.json
{
"id" : "2",
    "symbolIdStr": "passive:R",
    "exclude_from_bom": "False",
    "exclude_from_board": "False",
    "exclude_from_sim": "True",
"fields":{
"reference":{
"value" : "R", //can not be empty or a number
"show" : true, //only needed if not default
"show_name" : false //only needed if not default
},
"value":{ //only needed if not default for show or show_name
//value ignored if is preloaded
"show" : true, //only needed if not default
"show_name" : false //only needed if not default
},
"footprint":{ //only needed if exists or not default for show or show_name
"value" : "resistor:R", //only needed if exists
"show" : false, //only needed if not default
"show_name" : false //only needed if not default
},
"datasheet":{ //only needed if exists or not default for show or show_name
"value" : "$(DATA)/part.pdf", //only needed if exists
"show" : false, //only needed if not default
"show_name" : false //only needed if not default
},
"description":{ //only needed if not default for show or show_name
//value ignored if is preloaded
"show" : false,
"show_name" : false
}, "mfn" :{ //only needed if not default for show or show_name
//value ignored if is preloaded
"show" : false, //only needed if not default
"show_name" : false //only needed if not default
},
"mfurl" :{
"value" : "http:mfn.com",
"show" : false, //only needed if not default
"show_name" : false //only needed if not default
}
},
"order" : {
"mfn", "mfurl"   //define the order in witch the filds are added to the symbol
}
}

[Seth Hillbrand]

Hi Rosy-

Thanks for the interesting ideas on a version 2.

One thing that I would like to focus on is utilizing standard mechanisms where possible.  So, for example, in the section on caching, I would strongly prefer to utilize HTTP-based header calls instead of sending the caching data request to the web application.  Here, we could utilize If-Unmodified-Since in a GET request and properly parse the response in KiCad (HTTP412) to see whether we could utilize locally cached data.  Similarly, using transfer encoding instead of chunking the data in the backend will make the implementation more universal and speed for people.

The reason for this preference is to allow the use of standard tools to optimize content delivery without our needing to code additional logic to support it.

Seth

Seth Hillbrand

Lead Developer

+1-530-302-5483‬

Long Beach, CA

www.kipro-pcb.com    in...@kipro-pcb.com

To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/519f23dc-97bf-4ce5-8c5b-be4197c3b938n%40kicad.org.

[Rosy]

For me it would be important that the data that is changed on the server is recognized and adopted by KiCad.
Unfortunately, English is not my native language and I may not understand exactly what you mean. Could you give me a small example of how I should understand this or a source where I can read more about it?

[Seth Hillbrand]

Hi Rosy-

These are standard HTTP headers that are supported by all major web server and caching software.

If-Unmodified-Since:  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Since

Transfer Encoding: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Seth

Seth Hillbrand

Lead Developer

+1-530-302-5483‬

Long Beach, CA

www.kipro-pcb.com    in...@kipro-pcb.com

[Clemens Koller]

Hi there!

Using KiCad in a professinal environment drew my attention.
So I hope this is not fully OT here:
In my case I want or need to interface it to a MariaDB (MySQL) database where
the library components were generated automatically and are "married" to an ERP
system for purchasing and warehouse management on the database side.

So instead of querying libraries via http, is there or will there be an
SQL backend, too?

Unfortunately, I cannot spend much time with KiCad now. But migrating
KiCad to the existing component libraries is one of the next big steps to
get traction.

Regards,

Clemens

> --
> You received this message because you are subscribed to the Google Groups "KiCad Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to devlist+u...@kicad.org <mailto:devlist+u...@kicad.org>.
> To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/2b4f7c81-5186-4a9b-bb35-4da6f47568c6n%40kicad.org <https://groups.google.com/a/kicad.org/d/msgid/devlist/2b4f7c81-5186-4a9b-bb35-4da6f47568c6n%40kicad.org?utm_medium=email&utm_source=footer>.

[Seth Hillbrand]

There already is.

You can use ODBC connections to manage your libraries (https://docs.kicad.org/8.0/en/eeschema/eeschema.html#database-libraries)

Seth Hillbrand

Lead Developer

+1-530-302-5483‬

Long Beach, CA

www.kipro-pcb.com    in...@kipro-pcb.com

To unsubscribe from this group and stop receiving emails from it, send an email to devlist+u...@kicad.org.
To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/40215ede-ef90-4026-bbee-13f136461e49%40gmx.net.

[Rosy]

Hi Seth

Thanks for the information.

I think it would be possible to work with the "If-Modified-Since" method. However, this would mean that if a change occurred, the entire component master would be reloaded and not just the changes since the last call. The goal with the parameter is that the server only sends the changes, allowing quick work in the symbol chooser.

Transfer encoding: chunked is certainly an exciting method but only concerns the transfer between the client and the server. My idea is that when the first page has arrived, the next page is loaded asynchronously and during this time the JSON file is parsed and the parts are cached.

I have attached the latest version of my document.
The primary change is that instead of the key "chooser": bool in the fields, I inserted a key "type": int, which makes this a little more modular and oh an example with the idea that volatile fields may be supported in the future.

Generally when I talk about the cache, I mean the parts objects within Kicad.
Possibly an inherited cash register from LIB_SYMBOL supplemented with ID and timestamp.

For me it is generally important that working in Kicad is quick.

And since it is a real-time database that regularly contains new/different data, I would like to develop a good caching concept. In particular, with the idea of including volatile fields such as stock count, price or similar in the future.

Rosy

[Jon Evans]

> In particular, with the idea of including volatile fields such as stock count, price or similar in the future.

As discussed on previous gitlab MRs, volatile data should not be mixed up with part fields.

Fields are specifically non-volatile.

If you want to make a proposal for some system to get this volatile data somehow, that is fine, but it needs to be kept separate from fields.

-Jon

To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/c60cfe3b-fbff-4806-af87-749ac984acc9n%40kicad.org.

[Rosy]

Hi Jon

Thank you for your feedback, I am aware of the discussion and that is exactly why this is my suggestion, that this is an absolutely separate type and has nothing to do with the common fields. Even the server query is handled separately as this is definitely not used by every API.

My idea, which has not yet been thought of, is that the LIB_SYMBOL has a separate list of volatile fields that are only for the symbol chooser and may also be displayed differently, for example with italics or something like that.

I definitely wouldn't implement the whole part with the volatile fields in the first step; it's just important to me that it's already taken into account in the solution.

By the way, the current status of my documentation is below https://gitlab.com/RosyDev/kicad-dev-docs/-/blob/http-lib-v2/content/apis-and-binding/http-libraries/http-lib-v2-00.adoc?ref_type= heads visible.

greetings
Rosy

[Jon Evans]

> My idea, which has not yet been thought of, is that the LIB_SYMBOL has a separate list of volatile fields that are only for the symbol chooser and may also be displayed differently, for example with italics or something like that.

If volatile data is implemented, it should be supported in more places than just the symbol chooser.  People will want to query updated volatile data for parts already in the design, not just look up parts in the symbol chooser.

In general I think it would be best to completely separate these two ideas.  If you have some improvements in mind for the HTTP libraries, this should be done independently from a proposal for a volatile data API.  Doing it this way would mean that the volatile data API could be more broadly useful: for example, people may want to use it with parts that are not placed from a HTTP library.

-Jon

[Rosy]

You're right, I would definitely like to do this separately. I just wanted to keep the interface so dynamic that a future step in this direction would not be obstructed. Which is why I made a type for fields and not just a flag.

What is your view on caching?

Thank you for all the artistic contributions.

[Jon Evans]

> What is your view on caching?

I agree with Seth that HTTP standards should be used where possible.  I also think it would be ideal if there is no need for a "version 2" HTTP library, and instead we can make incremental improvements that are backwards-compatible.

> However, this would mean that if a change occurred, the entire component master would be reloaded and not just the changes since the last call. The goal with the parameter is that the server only sends the changes, allowing quick work in the symbol chooser.

If sending the entire library is a performance problem, this should be addressed with paging behavior rather than sending partial data in my opinion.  Optional "limit" and "after" parameters can be added to the query specification to allow the client to perform paging.  This is a backwards-compatible change; if either side does not support it, the whole library is transferred and the only issue is that the transfer size is larger.

-Jon

[Rosy]

I would also rather expand the existing v1 than create a new version. The main problem with version 1 is that categories and parts are transmitted as an array in the root. This means no extensions are possible. What I could also imagine is that we don't need a complete version 2, but rather the server uses a key in the validator to show that it supports the new features. e.g. a key called “version” with the value “1.1”.

I too am of the opinion that the normal http caching and compression methods should be used in algae. Paging is exactly what I wanted to introduce in that the server, if it supports it, splits the response into several pages and KiCad signals this by supplying a value for the next page. So the server would manage the paging and KiCad would only have to load page by page until the last page was reached.

The idea is also to have a timestamp as a parameter. This signals to the server that only the changes from this point onwards need to be transmitted.

[Andrew Lutsenko]

Hello,

There is a sql backend. It's not yet mature or easily user configurable but it's usable:

https://docs.kicad.org/8.0/en/eeschema/eeschema_advanced.html#database-libraries

On Fri, Apr 26, 2024 at 7:04 AM 'Clemens Koller' via KiCad Developers <dev...@kicad.org> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to devlist+u...@kicad.org.
To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/40215ede-ef90-4026-bbee-13f136461e49%40gmx.net.

[Rosy]

https://gitlab.com/RosyDev/kicad-dev-docs/-/blob/http-lib-v2/content/apis-and-binding/http-libraries/v1plus/http-lib-v1plus-00.adoc

In this document I have described how I imagine optimizing the http library while maintaining full compatibility with version 1, solving the problem with the inconsistent data between symbol chooser and schematics and still creating the possibility of having multiple columns in the symbol chooser will be displayed. The fields are adopted identically into the circuit diagram. No volatile fields are included.
Unfortunately, this version 1 has two major limitations that hinder the expansion and can only be changed with loss of compatibility:
1. The two requests *categories* and *parts* have an array in the root instead of an object.
2. Strings for Boolean values are used.

I would be happy to expand/adapt version 1 as described in the document if you agree.

I will be happy to talk to you again about version 2, which can address further problems such as caching, volatile fields, etc.

[Rosy]

In the meantime I have created an MR https://gitlab.com/kicad/code/kicad/-/merge_requests/1948, which contains various performance improvements, fixes for memory leaks, and a second version of the http library.

The detailed description of the second version can be found here:

https://gitlab.com/RosyDev/kicad-dev-docs/-/blob/http-lib-v2/content/apis-and-binding/http-libraries/v2/http-lib-v2-00.adoc? ref_type=heads

I have also prepared MR's for documentation. (I have marked these as drafts as long as the MR of my solution has not yet been adopted)

I have also tested the library with large amounts of data, but I would still recommend working with smaller project-based databases. I am wondering whether there is a way to include the project name in the query?

Is this value available in any way within the plug-in?

I am also wondering whether there is a way to provide the progress window with information when loading the library?

Can anyone help me with this?