Working on: additional boards listed in Boards Manager

[Federico Fissore]

Hi everyone

as you may read on the blog, we finally made up our minds about listing
additional boards in Boards Manager

Here is a quick spec list of how it will work

1) following Brian Cook's suggestion, files named
package_SOMETHING_index.json will be paired with main package_index.json
file. Their content will be merged and listed in Boards Manager.
BTW this will also ease testing a core in development: 1) place your
(for example) package_STM32_index.json file, 2) open Boards Manager, 3)
test it, 4) fix it, 5) release it on the web

2) Additional URLs will be specified adding a key in user's
preferences.txt file. When Boards Manager is opened, each url is downloaded.
We are postponing making a GUI for now, in order to save time and ship
the feature sooner. A GUI will come in the next next release.

2) While I think we all agree Todd Trecce's hack is simple and clever,
with this feature in place we can do what we already do with the Yun:
GPG sign package_index.json and verify the signature as soon as the main
package_index.json file is downloaded.
This will break Todd's hack but again, you'll just have to add a URL to
a text file and you'll get the very same result.

As soon as I have code to share, I'll make a pull request for everyone
to see and test

Comments are appreciated

Regards

Federico

[Todd Treece]

That sounds like a great solution! Anything that will allow users to easily install custom cores using the new board manager will be great. I'd much rather see something officially supported since the proxy hack has to relay all traffic from the IDE (including downloads), and the proxy only allows for one additional source of cores at a time.

Will the preference addition be part of v1.6.4, or a later release? Just curious so we can make a note on our proxy tutorial about what users should do for adding custom Adafruit cores.

Thanks,

-Todd

[Roger Clark]

Thanks Federico

Can you let us know when the new feature gets into the nightly build, as people often expect instant support on new features, so it would be good to be ahead of the game ;-)

Thanks

Roger Clark

www.stm32duino.com

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Brian Cook]

Hi Federico,

> as you may read on the blog, we finally made up our minds about
> listing additional boards in Boards Manager

Excellent news. Thank you for the work and the post.

> 1) following Brian Cook's suggestion, files named
> package_SOMETHING_index.json will be paired with main
> package_index.json file. Their content will be merged and listed in
> Boards Manager.
> BTW this will also ease testing a core in development: 1) place your
> (for example) package_STM32_index.json file, 2) open Boards Manager,
> 3) test it, 4) fix it, 5) release it on the web

https://groups.google.com/a/arduino.cc/d/msg/developers/E9ifjXwV2w0/-pw3F5Pjm3EJ

Just to reiterate, using a naming pattern that includes an organization...

package_OrganizationOrDeveloperName_PackageName.json

...will help avoid conflicts.

- Brian

[Nevada Smith]

That sounds cool. Can you also provide some logic on how the boards are ordered?
You may want to fix the order bug:
https://github.com/arduino/Arduino/issues/2650

And It'd also be nice if you could group different 3rd party boards. Because I've installed a few from other people and create my own boards and now the list is endless long and always sorts different from version to version.

[Mike]

I would agree with Nevada - it would be great to have a section for Intel boards, Adafruit, Sparkfun, Seeed, DigiStump, etc. with subboard types underneath to tame the clutter.

[Roger Clark]

I agree,

There needs to be some separation.

I have a massive proliferation of STM32 boards already, as there are loads of different generic boards. I'm trying to rationalise things by using sub-menu's but there are currently 8 top level boards, for STM32F103 alone (and there are a number of top level boards I've not even bother to add yet), plus when the STM32F4 code gets some more TLC there will be even more.

At the moment the way the sub menu's are handled, doesn't allow enough flexibility for me to combine a lot of similar boards, because the menus would need to cascade or effect each other in order to be able to do that.

[Federico Fissore]

Here is the PR

https://github.com/arduino/Arduino/pull/3055

It's feature complete: please give it spin :)

In order to add custom boards
1) prepare a json file mimicking arduino's
http://downloads.arduino.cc/packages/package_index.json
2) put it online somewhere
3) with the IDE close, edit preferences.txt file adding an entry like
boardsmanager.additional.urls=http://someurl...
4) reopen the IDE and open boards manager

Regards

Federico

Federico Fissore ha scrito il 29/04/2015 alle 09:54:

[Federico Fissore]

Todd Treece ha scrito il 29/04/2015 alle 18:45:

> That sounds like a great solution! Anything that will allow users to
> easily install custom cores using the new board manager will be great.
> I'd much rather see something officially supported since the proxy hack
> has to relay all traffic from the IDE (including downloads), and the
> proxy only allows for one additional source of cores at a time.
>
> Will the preference addition be part of v1.6.4, or a later release? Just
> curious so we can make a note on our proxy tutorial about what users
> should do for adding custom Adafruit cores.
>

Yes, we want this shipped for 1.6.4, which is due next week (bugs
permitting)

Regards

Federico

[Federico Fissore]

Brian Cook ha scrito il 30/04/2015 alle 03:25:

>
> Just to reiterate, using a naming pattern that includes an organization...
>
> package_OrganizationOrDeveloperName_PackageName.json
>
> ...will help avoid conflicts.
>

ATM every file which name starts with "package_" and ends with "_index"
is loaded

We may suggest a naming convention for sake of documentation or we may
force a specific name format and refuse to download from url which does
not comply

Are you for the second option?

Regards

Federico

[Federico Fissore]

Can you provide two lists with what you currently get and what you would
like to get? That will help figure out a sorting algorithm

Regards

Federico

[Brian Cook]

Federico,

>> Just to reiterate, using a naming pattern that includes an
>> organization...
>>
>> package_OrganizationOrDeveloperName_PackageName.json
>>
>> ...will help avoid conflicts.
>>
>
> ATM every file which name starts with "package_" and ends with
> "_index" is loaded
>
> We may suggest a naming convention for sake of documentation or we may
> force a specific name format and refuse to download from url which
> does not comply
>
> Are you for the second option?

No. The IDE should do what it does now (load all package_*_index
files). The post was meant as a suggestion for everyone who will be
creating packages (e.g. Roger Clark).


Thanks,
Brian

[Roger Clark]

One thing I'm starting to notice is that the height of the boards menu can easily get too tall to fit on some screens e.g even new laptops often are only 768 pixels tall, and on my system with the Due and STM32 and ESP8266 installed, they no longer fit

Albeit, I'm probably not your average user, but I can still see that people could have problems with this even if they don't have the STM32 boards

I can split the STM32 boards into smaller groups. In fact they should definately split, because the STM32F3 and STM32F4 is, at the moment, a completely separate code base (because no one has time to unify them.... (its a long story...)

Also.

Can we make the font on the board group title be bold rather than greyed out, or if bold is not an easy option, perhaps underline ??

If I knew my way around the code I'd do a PR but you guys can find in an instant what would take me ages to find.

Or is this just a generic thing ? ie menu sub titles are always grey ?

Sorry I can't help more:-(

[Brian Cook]

Federico,

> Here is the PR
> https://github.com/arduino/Arduino/pull/3055
> It's feature complete: please give it spin :)

I have some questions about the package_index.json file that comes with
the Nightly Build...

• The download files are tar.bz2, tar.gz, or zip. What other archive
formats are supported? I am especially interested in 7z (7-Zip). I
have found the compression to be slightly better than most and the
decompression to be significantly faster than most.

• What tool is used to generate the SHA hash (the "checksum")?

• Is the SHA hash (checksum) over the archive or the expanded files?

• Is "size" the size of the archive or the size of the expanded files?

• Is https supported for "url"?

Thanks,
Brian

[Todd Treece]

Brian,

I might be able to help with some of these questions. Here's the shell script I use to package up a folder: https://gist.github.com/toddtreece/3e70611bb403883de0d0

I'm using it with OS X, but it should work with Linux as well. The checksum and size are both referencing the archive, not the expanded files.

I think there's a bug currently with HTTPS support: https://github.com/arduino/Arduino/pull/3055#issuecomment-97893180

-Todd

[Federico Fissore]

Brian Cook ha scrito il 01/05/2015 alle 23:07:

> Federico,
>
>> Here is the PR
>> https://github.com/arduino/Arduino/pull/3055
>> It's feature complete: please give it spin :)
>
> I have some questions about the package_index.json file that comes with
> the Nightly Build...
>
> • The download files are tar.bz2, tar.gz, or zip. What other archive
> formats are supported? I am especially interested in 7z (7-Zip). I
> have found the compression to be slightly better than most and the
> decompression to be significantly faster than most.

Only those three. 7z (and xz and lzma) are currently not supported.

>
> • What tool is used to generate the SHA hash (the "checksum")?

Todd suggested "openssl dgst -sha256" or (linux) "sha256sum"

>
> • Is the SHA hash (checksum) over the archive or the expanded files?
>
> • Is "size" the size of the archive or the size of the expanded files?
>

As Todd has said, both are over the archive

> • Is https supported for "url"?

Yes, although some strongly configured webserver such as Adafruit's may
not work. See [1] for the reason why.

I've tested connecting to a github "raw" file and it works
https://raw.githubusercontent.com/ffissore/testjson/master/package_adafruit_index.json

[1] https://github.com/arduino/Arduino/pull/3055#issuecomment-98645014

[Brian Cook]

Federico,

Thank you for the reply.

>>> Here is the PR
>>> https://github.com/arduino/Arduino/pull/3055
>>> It's feature complete: please give it spin :)
>>
>> I have some questions about the package_index.json file that comes with
>> the Nightly Build...
>>
>> • The download files are tar.bz2, tar.gz, or zip. What other archive
>> formats are supported? I am especially interested in 7z (7-Zip). I
>> have found the compression to be slightly better than most and the
>> decompression to be significantly faster than most.
>
> Only those three. 7z (and xz and lzma) are currently not supported.

Bummer.

>> • Is https supported for "url"?
>
> Yes, although some strongly configured webserver such as Adafruit's
> may not work. See [1] for the reason why.

Got it.

> I've tested connecting to a github "raw" file and it works
> https://raw.githubusercontent.com/ffissore/testjson/master/package_adafruit_index.json
>

Nice!


- Brian

[William Westfield]

> I've tested connecting to a github "raw" file and it works
> https://raw.githubusercontent.com/ffissore/testjson/master/package_adafruit_index.json

So … how do I get the equivalent raw link for something like:
https://github.com/Optiboot/optiboot/releases/download/untagged-fe7607ef7b0eeb449cc8/package_optiboot_optiboot-additional_index.json

I’m having trouble finding documentation in amongst all the repositories that simply use raw…

Thanks
Bill W

[Peter Feerick]

Hi Bill,

I get a 404 for the link you gave... have you moved the file?

Anyway, to get an equivalent raw link I clicked on

• your v6.2 release of optiboot (https://github.com/Optiboot/optiboot/releases/tag/v6.2)
• then clicked on the v6.2 tag (https://github.com/Optiboot/optiboot/tree/v6.2),
• clicked on optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE (https://github.com/Optiboot/optiboot/blob/v6.2/optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE)
• and finally clicked on RAW at the top of the code view on the right (Raw - Blame - History) (https://raw.githubusercontent.com/Optiboot/optiboot/v6.2/optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE)

There is probably a better way, but it worked for me...

Pete

[William Westfield]

On May 29, 2015, at 2:45 AM, Peter Feerick <peter....@gmail.com> wrote:

> • clicked on optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE (https://github.com/Optiboot/optiboot/blob/v6.2/optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE)

> • and finally clicked on RAW at the top of the code view on the right (Raw - Blame - History) (https://raw.githubusercontent.com/Optiboot/optiboot/v6.2/optiboot/package_optiboot_optiboot-additional_index.json.TEMPLATE)
>

Yeah, but ideally I want a raw link to a file attached to the “release”, rather than one that’s inside the repository.

(I thought the problem might be that the release was in “draft” status, so I went ahead and published it. The link is now https://github.com/Optiboot/optiboot/releases/download/v6.2/package_optiboot_optiboot-additional_index.json ; clicking on that link gets )

Hmm. It looks like the link above is “raw enough”, in spite of apparently not working with the IDE when I first tried it. It’s getting further now…

==============================

So…
boards installed by the board manager require a different directory structure that older style installs, because they go into …/packages/<YourName>/hardware/avr/<VERSION>/<stuff> rather than …/sketchbook/hardware/<YourName>/avr/<stuff> ? That’s… annoying.
Given a zip file that works fine unzipped locally into sketchbook/hardware, the board manager gives me:

> processing.app.debug.TargetPlatformException: Could not find boards.txt in /Applications/arduino/Arduino-1.6.4.app/Contents/Java/portable/packages/Optiboot-additional/hardware/avr/6.2.0. Is it pre-1.5?

BillW

[Peter Feerick]

Bill,

As long as it's working for you now! :) It looks like the download links
on releases are to the actual files, rather than other placeholder
pages, etc.

Can you get rid of the square brackets around the name in the menu?
(plaform.txt name) Just looks a bit odd in comparision to all the other
board group names.

I tried unpacking the OptiBoot zip into the sketchbook of the 'portable'
install of Arduino 1.6.4 I have running (so under sketchbook I have
'hardware\Optiboot\avr\boards.txt')
... and it worked perfectly for me on Windows 7... I have six boards
listed under an 'Optiboot 6.2' heading... *but* if I add a subfolder
6.2.0 and move the contents of 'avr' into that (so
'hardware\Optiboot\avr\6.2.0\boards.txt') ... I get the exact same error
messages you did.

And since nothing happens for me if I drop the optiboot folder in the
packages folder... no errors .... no optiboot .... I presume 3rd party
hardware is only loaded from the packages subfolders if a matching json
package definition exists for them?

Pete

[William Westfield]

> a different directory structure

Apparently I have been confused which parts of the directory structure I’m supposed to supply, and which are created by the boards manager based on the metadata. So I would up with
/packages/Optiboot/hardware/avr/0.6.2/hardware/avr/0.6.2/…
which of course isn’t what it wanted…

I think it’s working now.
https://github.com/Optiboot/optiboot/releases/download/v6.2/package_optiboot_optiboot-additional_index.json

I have a nice little “install.sh” script doing most of the work for me, too.
https://github.com/Optiboot/optiboot/blob/master/optiboot/release.sh
It’s not very general at the moment, but might be useful as a model.

—————————

Now. Can I have “examples” associated with a Board? Or only with libraries?

BillW

[Peter Feerick]

It works perfectly on this end - simply adding the
'https://github.com/Optiboot/optiboot/releases/download/v6.2/package_optiboot_optiboot-additional_index.json'
link to the additional board manager URLs list resulted in immediate
detection, and I had no problems with installing it. Thanks for getting
rid of the '[' ']' s!

Now, can I ask a stupid question (or two) :) I've been out of touch with
Optiboot for a while, so have forgotten where it was at... what version
comes with the Arduino IDE / is pre-installed on Arduino Unos generally,
and is the best way to find out if it's worth updating the installed
version of optiboot trawling through the commit log on github?

Pete

[William Westfield]

On May 29, 2015, at 10:57 PM, Peter Feerick <peter....@gmail.com> wrote:

> I've been out of touch with Optiboot for a while, so have forgotten where it was at... what version comes with the Arduino IDE / is pre-installed on Arduino Unos generally, and is the best way to find out if it's worth updating

Arduinos are shipping with version 4.4, and the current version is 6.2 or so.
MOST of the changes have involved adding new chips (and additional options for them), improving the ease with which one can build custom versions, adjusting for new compiler versions, writing documentation, and stuff like that. Things that don’t affect the typical Uno user at all.

But you can now say
make atmega1280 LED=J0 UART=3 BAUD_RATE=1000000
Which is pretty cool. (not particularly elegant behind the scenes, but that’s not the point.)

Probably the best summary is the “Edit History” comment section in optiboot.c
https://github.com/Optiboot/optiboot/blob/master/optiboot/bootloaders/optiboot/optiboot.c#L153

For official/current arduino boards, the most important changes are probably:
1) passes Reset cause to application in R2 (mostly)
2) No longer overwrites flash if you try to write EEPROM.

(so, the big Makefile restructuring was responsible for going to v5.x, and the changes for EEPROM for bumping to 6.x (1280 and 1284 actually support writing EEPROM now.))

If there were a lot of important changes, I’d be trying harder to get them pulled into the arduino branch.

BillW