Boards Manager - external sources

[Roger Clark]

Hi Guys,

Do you think the development team would accept a pull request to modify the Boards Manager to be able to specify package_index.jason files (or file locations) other than from Arduino.cc - in order to support third party boards

The reason I'm asking, is that my third party hardware files rely on the Arm compilor, and this is nolonger installed by default, and even if you manually install the Due, the location of the compilor is not where it used to be

Although I could package the compilor as part of my overall hardware package, it seems very wasteful for users to need to download the Arm compilor, and I'm either going to need to add GCC ARM for Win32, Linux and OSX in the same repo, or create download packages that reference external ZIP packages (probably on Arduino.cc or another guthub repo)

So ideally, it would be good if the user could specify external / untrusted sources for the Boards Manager

I've now forked the Arduino / Arduino repo and will work on this myself

But I'm curious if the core team would accept such an update as a pull request,

But I"m worried that because of the issues with Arduino vs Arduino, that the reason that only Ardunio.cc packages are shown is to prevent Arduino.slc using the Arduino.llc IDE

Which would not be good news for the OpenSource community at large

[Alex Albino]

I would like to see something like this, too. :-)

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Federico Fissore]

Hi Roger

First of all (slightly OT), sorry for the lack of updates on the list: I
planned to sum up features and plans for the installers for yesterday
morning, but the issues arisen are keeping me from doing that


Roger Clark ha scrito il 31/03/2015 alle 02:27:

> Hi Guys,
>
> Do you think the development team would accept a pull request to modify
> the Boards Manager to be able to specify package_index.jason files (or
> file locations) other than from Arduino.cc - in order to support third
> party boards
>
> The reason I'm asking, is that my third party hardware files rely on the
> Arm compilor, and this is nolonger installed by default, and even if you
> manually install the Due, the location of the compilor is not where it
> used to be

As you've read on github, a release to the compiler path issue has been
released with the nightly

>
> Although I could package the compilor as part of my overall hardware
> package, it seems very wasteful for users to need to download the Arm
> compilor, and I'm either going to need to add GCC ARM for Win32, Linux
> and OSX in the same repo, or create download packages that reference
> external ZIP packages (probably on Arduino.cc or another guthub repo)
>

Indeed the ideal is to only package your core and make it depends on the
arm tools, as our SAM core does

>
> So ideally, it would be good if the user could specify external /
> untrusted sources for the Boards Manager
>
>
> I've now forked the Arduino / Arduino repo and will work on this myself
>
> But I'm curious if the core team would accept such an update as a pull
> request,

We would also like to include additional contributed cores to the list.
This is both a strategic move and a support issue (will people mail us
for support on core XYZ?)
So, we would love to populate that list but we still have to make up our
minds about how to properly do that.
For now, your users will have to install the SAM core from the boards
manager

> But I"m worried that because of the issues with Arduino vs Arduino, that
> the reason that only Ardunio.cc packages are shown is to prevent
> Arduino.slc using the Arduino.llc IDE
>
>
> Which would not be good news for the OpenSource community at large
>

That thing has nothing to do with that. It's only about making life
easier to users by providing a smaller download and make it easier to
install cores and libs: we would like to make the average user forget
that "download the zip, unpack it there, rename the folder, does it
work? no? then double check the name, bla bla"

Federico

[Jeroen Doggen]

Just for reference.
This question was also raised on the Arduino forum: http://forum.arduino.cc/index.php?topic=311851.0

[Federico Fissore]

Jeroen Doggen ha scrito il 31/03/2015 alle 09:07:

> Just for reference.
> This question was also raised on the Arduino forum:
> http://forum.arduino.cc/index.php?topic=311851.0
>

Thanks, answer posted there too

[Brian Cook]

Federico,

> We would also like to include additional contributed cores to the
> list. This is both a strategic move and a support issue (will people
> mail us for support on core XYZ?)
> So, we would love to populate that list but we still have to make up
> our minds about how to properly do that.

Are you interested in suggestions?

- Brian

[Andrew Kroll]

Have a com box accept urls would be a good start.

Another great idea wold be to have a command switch or IPC pipe tell the IDE from a web browser javascript/java to download and/or add a url.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[Federico Fissore]

Brian Cook ha scrito il 31/03/2015 alle 09:40:

> Are you interested in suggestions?
>

Of course

[Brian Cook]

Hi Federico,

> We would also like to include additional contributed cores to the
> list. This is both a strategic move and a support issue (will people
> mail us for support on core XYZ?)
> So, we would love to populate that list but we still have to make up
> our minds about how to properly do that.
> For now, your users will have to install the SAM core from the boards
> manager

In the short term I suggest one change. The package list is in a file
named package_index.json. I suggest loading / merging all files named
package*.json from the directory containing package_index.json.

This provides a method for adding cores that is about the same effort as
exists now. Currently, adding a core means: closing the IDE,
downloading an archive (e.g. 7-Zip file), expanding the archive to the
correct location, starting the IDE. The new steps would roughly be:
close the IDE, download a json file to the correct location, start the
IDE, use the Board Manager to install the new core.

The change allows all affected parties to start testing the new Board
Manager without having to wait for a more complete solution.

The change is low risk. Instead of loading a single file the code would
load an arbitrary set of files then merge. If something goes terribly
wrong, the code can easily be reverted to what it is now.

I also suggest publishing a simple guideline for naming the package
file: package_OrganizationOrDeveloperName_PackageName.json seems like a
good pattern.

- Brian

[Federico Fissore]

Brian, I really like it! Easy and low risk!

I'll put it in the todo list. I'm not sure how fast I'll be at coding it
(list is already pretty full) but even if I come
up too late with an implementation, it will be done this way

Could this be a good candidate for a bounty on bountysource?

Anyway, thank you!

Federico



Brian Cook ha scrito il 01/04/2015 alle 22:27:

[Brian Cook]

Federico,

Thank you for the feedback.

Do not make it a bounty on my behalf. I can be patient and I do not
have to means to do it myself.

Thanks,
Brian

[William Westfield]

> The new steps would roughly be:
> close the IDE, download a json file to the correct location, start the
> IDE, use the Board Manager to install the new core.

Can the board manager json files point to local files? I don’t particularly like requiring “indeterminate” amounts of internet access (and server provisioning) to install/provide a board or library. Ideally I want to provide (or download) a .zip file, tell the IDE to install it, and have the .json be found there (in a standard place? Or exhaustively searched?) and run through the manager install process. Users shouldn’t even need to know whether it’s a library or a board-defining .zip.

(I also want a version of the IDE download that installs everything locally. If for no other reason than that I potentially give seminars in places that don’t provide wifi, or heavily filter access.)

BillW
WestfW

[Roger Clark]

I think there will be problems with asking the users to copy the package JSON files to the "correct folder"

I've just checked, and on Windows the %appdata% folder is not normally visible to users, so expecting them to copy into it, may be a bit to much for a lot of people

Albeit we are assuming they are installing non standard third party hardware.

However I can testify though managing the Arduino_STM32 repo that there are people using the STM32 boards, who would require help in finding the correct folder if it was not in the normal Documents/Arduino folder, and I expect the same applies to anyone using the ESP8266 hardware files etc

I guess one way around this is to provide a file selector mechanism inside the Boards manager, which would copy the new package file e.g. package_arduino_stm32.json to the "correct" location

I also can appreciate anyone in an academic environment where each user has their own account on a quite locked down installation of Windows (or Linux etc), where having to install the binaries etc into every user folder is a massive overhead and waste of disk space

Is there any reason that the compiler binaries can't be stored in the installation / tools folder etc like before

Looking at the total contents of my %appdata% folder on my main development machine - which has litterally hundreds of programs an utilities installed.

Hardly any of them store exe's in the %appdata% folder

If you read the Microsoft guidelines on the use of the AppData folder, https://msdn.microsoft.com/en-us/library/windows/apps/hh465094.aspx  its obvious that what the IDE team is using the folder for is incorrect

"Use roaming data to store a user's settings, preferences, and session info to create a cohesive app experience across multiple devices"

I may post an issue referencing these guidelines as I think its a serious mistake to use the folder for storing the compiler exe's etc

[William Westfield]

> the %appdata% folder

Also note that the stuff installed in %appdata% does not get removed during an “uninstall.”

That’s fine for library source code and such, but not so good for copies of several different compilers.

BillW/WestfW

[Todd Treece]

I was looking for a way to test out board packages, and wrote a simple proxy server yesterday to add custom packages to the Board Manager. Not sure if this is a worthwhile stop gap until custom boards can be added through official methods, but I thought I'd post it here in case it could help: https://github.com/adafruit/adafruit-arduino-proxy

It hasn't been tested heavily, but it seems to work ok.

[Roger Clark]

If you are on windows, you can simply add a line to the arduino.l4j.ini file in the install root

-DPACKAGE_INDEX_URL=http://www.yoururl.com/package_index.json

Of course this could be done with a simple local web server rather than a real server out on the web.

Then it will load a file of your choice

(PS. There is probably an equivalent on Linux and OSX)

[Todd Treece]

Roger,

Thanks for the tip. I'll have to see if OS X stores that file somewhere else. I wrote that simple proxy server because I couldn't find any mention of that with the OS X install when searching with ack. This returns no results in 1.6.3:

todd:Arduino.app todd$ ack PACKAGE_INDEX_URL

[Federico Fissore]

Roger Clark ha scrito il 03/04/2015 alle 02:35:

>
>
> I also can appreciate anyone in an academic environment where each user
> has their own account on a quite locked down installation of Windows (or
> Linux etc), where having to install the binaries etc into every user
> folder is a massive overhead and waste of disk space
>

Locked down windows at universities can't even store preferences.txt.
They solve the issue thanks a Shigeru Katemoto patch: by making a
"portable" folder into IDE folder, everything is stored in there, thus
allowing for a "portable IDE"

> Is there any reason that the compiler binaries can't be stored in the
> installation / tools folder etc like before
>

Introducing upgrade of cores/libs (with a different life cycle than the
IDE itself, ie: you could keep on using IDE 1.6.3 while having avr core
1.8.6) raised a permission issues: can a normal user write files in
c:\program files or /usr/share?

Usually no. Eclipse (the IDE) solves this using a .eclipse folder, local
to the user, where it stores all the additional plugin you install
through its plugin manager

>
> If you read the Microsoft guidelines on the use of the AppData folder,
> https://msdn.microsoft.com/en-us/library/windows/apps/hh465094.aspx its
> obvious that what the IDE team is using the folder for is incorrect
>

I didn't know windows kept Roaming data in sync with other devices
That's an issue we will address, by changing folder location to
something outside Roaming, but still local to the users home folder,
migrating these folder as soon as this new IDE will start up

>
> I may post an issue referencing these guidelines as I think its a
> serious mistake to use the folder for storing the compiler exe's etc
>

I saw you wrote it: https://github.com/arduino/Arduino/issues/2902

Thanks

[Federico Fissore]

William Westfield ha scrito il 03/04/2015 alle 10:07:

We conducted some users interviews before taking that step: the majority
preferred to keep cores across different sketchbooks, while sketchbooks
were used as "projects". So cores were to be stored outside of the
sketchbook, and libraries were to be local to it

As we move toward a separation of IDE and cores/libs, having cores lying
around when updating the IDE allows you to update it without having to
reinstall, for example, SAM cores

Regards

Federico

[Brian Cook]

Federico,

> > If you read the Microsoft guidelines on the use of the AppData folder,
> > https://msdn.microsoft.com/en-us/library/windows/apps/hh465094.aspx
> its
> > obvious that what the IDE team is using the folder for is incorrect
> >
>
> I didn't know windows kept Roaming data in sync with other devices
> That's an issue we will address, by changing folder location to
> something outside Roaming, but still local to the users home folder,
> migrating these folder as soon as this new IDE will start up

A shared folder seems like a better choice: ProgramData (aka All Users)
or C:\Program Files\Common Files\.

For a single-user computer, the user really won't care.

For a multi-user computer, the users will be pleased that harddrive
space is not unnecessarily wasted.

But, as I mentioned, shared locations can be difficult to access;
requiring an sympathetic IT administrator.

- Brian