Boards manager package for evolving cores?

[Roger Clark]

Hi IDE team,

I've started to look at a board manager package for the STM32 core (github.com/rogerclarkmelbourne/arduino_stm32) 

However I've come across a bit of an issue because the core is updated frequently e.g. many times per week.

I had presumed that I can have one package  zip file, for the core, which I can update every day, and that I can change the version number in the package json file e.g.

(See version numbers in bold - based on date)

"platforms": [ {
      "name":"STM32f103",
      "architecture":"stm32f103",
      "version":"20150623.0.0",
      "category":"STM32",
      "url":"http://localhost:8080/arduino/stm32duino_f103_20150623.zip",
      "archiveFileName":"stm32duino_f103_20150624.zip",
      "checksum":"SHA-256:cf3c6dc67a521c8a9d57c7be5fce8ab18c73b6cb74d73388481e64efe3314e8d",
      "size":"15288301",
      "help":{
        "online":"http://www.stm32duino.com"
      },

   etc etc etc

However I found that if I change the package json file the next day to

"platforms": [ {
      "name":"STM32f103",
      "architecture":"stm32f103",
      "version":"20150624.0.0",
      "category":"STM32",
      "url":"http://localhost:8080/arduino/stm32duino_f103_20150624.zip",
      "archiveFileName":"stm32duino_f103_20150624.zip",
      "checksum":"SHA-256:cf3c6dc67a521c8a9d57c7be5fce8ab18c73b6cb74d73388481e64efe3314e8d",
      "size":"15288301",
      "help":{
        "online":"http://www.stm32duino.com"
      },

Then when I restart the IDE, it removes all the boards that I had installed yesterday, and I have to go to the Boards Manager and install boards for 2015 06 24 (ie todays version)

I know I can add a new item to the platform's array, but this will be that I will have a larger and larger package json file.

Is there any way to prevent the IDE from removing the boards that have been installed, if their version number is no longer listed in the package file ?

Thanks

Roger Clark

[Victor Aprea]

Roger,

I think it's the other way around. Multiple zip files, one package JSON file (containing definitions that reference all your zip files)... I'm still trying to figure it out too, so please don't consider me an authority.

Cheers,
Vic

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Roger Clark]

Hi Victor,

I did a bit more testing after I posted and the current situation seems to be as follows.

If I change the version number at all in the "platforms" section of the package file. e.g. from

"version":"20150624.0.0"

to

"version":"20150624.0.1"

and I go to the Boards Manage, it removes the boards in question from the Board menu.

But it doesn't seem to remove the files themselves

If I add another section into the platform's array, that is a new version of the same core... e.g. see below the platform section with 2 versions

I get the drop down to select the version.

Note. If the "archiveFileName" is the same for both versions (even if the version number is different), you do not get the dropdown menu, but you do get a button to "update"

Anyway, as it stands.. Every time to make a change to the core that someone may need e.g. I am fixing a bug in the SPI lib today where the default bit order is LSBFIRST not MSBFIRST - then I'd need to add a while new section to the json file, I'd also need to  keep every zip file, regardless of how out of date it is. Well I guess I could just delete the old zips, but this would give an error if someone tried to install an old version.

Ideally for me, the IDE would not hide the version that it has installed, just because its no longer in the json file, in which case I'd only need one entry in the json file and I'd only need to have the latest zip on the server

 "platforms": [ {
      "name":"STM32f103",
      "architecture":"stm32f103",
      "version":"20150624.0.0",
      "category":"STM32",
      "url":"http://localhost:8080/arduino/stm32duino_f103_20150624.zip",
      "archiveFileName":"stm32duino_f103_20150624.zip",
      "checksum":"SHA-256:cf3c6dc67a521c8a9d57c7be5fce8ab18c73b6cb74d73388481e64efe3314e8d",
      "size":"15288301",
      "help":{
        "online":"http://www.stm32duino.com"
      },

    "boards": [
            {"name": "Maple Mini"},
            {"name": "Maple Rev3"},
            {"name": "Maple RET3"},
            {"name": "Generic STM32F103C series"},
            {"name": "Generic STM32F103R series"},
            {"name": "Generic STM32F103V series"},
            {"name": "Generic STM32F103Z series"},
            {"name": "STM Nucleo F103RB"},
            {"name": "Microduino Core STM32"}
          ],
        "toolsDependencies": [
            {
              "packager": "arduino",
              "name": "arm-none-eabi-gcc",
              "version": "4.8.3-2014q1"
            }
          ]
    },

{
      "name":"STM32f103",
      "architecture":"stm32f103",

      "version":"20150625.0.0",
      "category":"STM32",
      "url":"http://localhost:8080/arduino/stm32duino_f103_20150625.zip",
      "archiveFileName":"stm32duino_f103_20150625.zip",

      "checksum":"SHA-256:cf3c6dc67a521c8a9d57c7be5fce8ab18c73b6cb74d73388481e64efe3314e8d",
      "size":"15288301",
      "help":{
        "online":"http://www.stm32duino.com"
      },

    "boards": [
            {"name": "Maple Mini"},
            {"name": "Maple Rev3"},
            {"name": "Maple RET3"},
            {"name": "Generic STM32F103C series"},
            {"name": "Generic STM32F103R series"},
            {"name": "Generic STM32F103V series"},
            {"name": "Generic STM32F103Z series"},
            {"name": "STM Nucleo F103RB"},
            {"name": "Microduino Core STM32"}
          ],
        "toolsDependencies": [
            {
              "packager": "arduino",
              "name": "arm-none-eabi-gcc",
              "version": "4.8.3-2014q1"
            }
          ]
    }


    ]


 

[Ivan Grokhotkov]

Hi Roger,

I second this, have the same issue with the ESP8266 core which gets updated quite often.

While adding new versions into json file is possible, the list will grow quickly.

It would be nice if the IDE would at least remove the version currently installed when pulling a new one, even if the current version is no longer present in json file.

Regards,

Ivan

[Roger Clark]

Ivan

Perhaps we should submit this as an issue.

Ideally for me, if I remove a version from the package, I don't want the IDE to remove the installed files if the user goes to the board manager (possibly to install another board, not to change something on the STM32).

However I think the problem is that the IDE is not retaining any form of database of the boards that it has installed.

It is totally reliant on the package file and also the folder system (mainly the package files)

I think what the IDE really need to do is to keep its own master file (JSON or XML), to act as a database of what it has installed, possibly to copy the sections of the JSON file that it installs, into the folder for that core.

I know this is substantially more complicated for the IDE team to develop, but at the moment the whole system is prone to failure if any one of the package files has an error in it.

I can see the boards manager would work fine if it only handled cores that the IDE team managed, but now that its open for people to add whatever boards (package files) they like. I can see that there will be issues in the future for people because of download errors in package files or just authoring mistakes in package files.

At the moment I'm hand authoring my package file (I see you have a script to generate yours), and I often make mistakes which seem to crash the IDE (well prevent it starting)

I suppose another option, would be for the IDE to not, download all the package files every time the BM is opened, but instead have a refresh button next to each board. Then if after the new package file has been downloaded, the current / installed version was not present in the package file, the IDE could give the use the option to update their version to the new / only version, or leave it as it is.

BTW.

I'm not a Java programmer, so I don't know if Java supports some sort of lightweight database. For these sort of things I often just use XML as a form of database  (but this is mainly because I'm normally developing for Adobe AIR, which has quite good XML support)

[Federico Fissore]

Roger Clark ha scritto il 26/06/2015 alle 01:44:
> However I think the problem is that the IDE is not retaining any form of
> database of the boards that it has installed.

This is exactly the reason why the IDE behaves as you described.
It needs to recognize the installed stuff and its dependencies (tools)
but it doesn't have a local db, relying only on remote json files
content. Nothing is deleted however: it just doesn't get listed.

Having such a db is a nice improvement: we'll try to schedule it for a
next version.

Back to the initial issue (frequent releases) we are going to support
kind of nightly builds for cores

This is what we'll do for the Zero core:
1) main package_index.json file will keep on listing RELEASES
2) an additional package_zero_nightly_index.json (or something like
that) will list ONE version of the core only
3) this version (something high as 9.9.9) will MISS size and checksum fields
4) IDE 1.6.6, seeing those fields missing, will delete its previously
downloaded files every time users will click "install" in BM

This way, a user wishing to work with a development version of Zero
core, will update the core by clicking "remove" and then "install"

From a users perspective, this makes them sure releases are stable
stuff. Manually adding an additional url for having the nightlies makes
it clear that this is optional, advanced stuff

From a developer perspective, you'll have to create this one additional
package_index.json file ONCE and the only file that will need
maintenance (and that will keep on growing) will be the one where you'll
list releases

If your core is on github, the nightly will point to master, the
releases to tags.

What do you think?

Federico

[Roger Clark]

Federico

Thanks for the details of the Zero nightly builds package, I will analyze the zero files and do the same for the STM32 (and I suspect Ivan may want to do the same thing with the ESP8266)

PS. Can you let me know the package location for the zero nightly build.

[Federico Fissore]

Roger Clark ha scritto il 02/07/2015 alle 12:42:
> Federico
>
> Thanks for the details of the Zero nightly builds package, I will
> analyze the zero files and do the same for the STM32 (and I suspect Ivan
> may want to do the same thing with the ESP8266)
>
> PS. Can you let me know the package location for the zero nightly build.

We haven't yet because we are still at the need/idea stage. I think
we'll have something tomorrow or beginning next week as I plan to work
on it this afternoon

Please be aware that this stuff will work ONLY with 1.6.6. I think 1.6.5
will just break when trying to install something without size/checksum,
or will report CRC errors (haven't tried yet)

Federico

[Victor Aprea]

Federico, that seems like a pretty good strategy to me. 

Thanks,

Vic

Victor Aprea // Wicked Device

victor...@wickeddevice.com

T: @vicatcu

Federico

[Federico Fissore]

Pull request is ready

https://github.com/arduino/Arduino/pull/3449

It took less than estimated

Federico