Where would you save cores and compilers?

[Federico Fissore]

The target location for optional cores and compilers has become a
recurrent question but so far I haven't read a feasible proposal

So let's try gather them all: you have an IDE, you have installed it
using your OS utilities (apt-get, dmg, exe installer), you have
"downloadable plugins" and you want to install a new fancy one

Where will its files go?

Federico

[David Mellis]

One consideration that I think is important is retaining these add-ons / plugins across updates to the IDE itself. That is, I don't think people should have to re-install all the add-ons when they upgrade to a new version of the IDE.

Federico

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Brian Cook]

With recent versions of Windows the typical location is
(CSIDL_COMMON_DOCUMENTS)
C:\Users\Public\Documents\{Company}\{Product}\{Version}\ which might be
C:\Users\Public\Documents\Arduino\IDE\1.6\ for the Arduino IDE. I
believe that location was added in Windows 7.

With previous versions of Windows the typical location was
(CSIDL_COMMON_APPDATA) C:\ProgramData\{Company}\{Product}\{Version}\ or
its alias C:\Users\All Users\{Company}\{Product}\{Version}\. I believe
that location is available starting with XP (it was available after
updating Internet Explorer).

Those are both good locations for shared files. The caveat is that a
system administrator may have restricted access. But that is a
potential problem no matter where the files are located.

There is a shared location for programs (C:\Program Files\Common
Files\). But administrator access is required to add/modify.

- Brian

[William Westfield]

On Jun 8, 2015, at 8:38 AM, Federico Fissore <f.fi...@arduino.cc> wrote:

> you have "downloadable plugins" and you want to install a new fancy one
> Where will its files go?

What qualifies as a “plugin” and How big is it?

My objection to the current state of affairs is that I don’t think that 800+MiB of compilers is a good thing to make a “plugin.” The same people who might not have write access to the Programs area might equally be “not permitted” to use up that kind of space in their “personal file space” areas. They ARE applications, which might tweak anti-malware measures, and if they’re out on a network file system, then performance could be significantly impacted. And I really don’t want N copies of 800MiB cluttering things up.

Atmel Studio scatters it’s plugins all over the place, some being automatically installed, others requiring that you execute a separate installer. That sucks too.

You could work backward from expected usage:

1) As “systems administrator”, I want to put a copy of everything my users need in a nice protected place that they cannot modify. All they should have in user-specific storage is their sketches and preferences. MAYBE some 3rd party libraries, but maybe I really wanted them to write that code themselves, not just find something that already existed. Per-user storage is potentially slow and small. 3rd party hardware should go through me. The pre-6.0 IDE did this pretty well.


2) As a user of an otherwise locked down system, I want all arduino-related stuff on a flash drive that I can plug in anywhere that has everything arduino, and all my arduino stuff, on it. New stuff gets written there too. The current “portable” feature mostly works for this.

2a) like (2), but my “personal storage” is networked, and I want my sketches to go there instead. (“portable” doesn’t let you change the sketch folder, so it doesn’t work for this.)

2b) like (2), except I have a “Live CD/DCD" that I hand out to my class so they don’t have to install anything. But it’s read-only, so sketches/etc need to go elsewhere.


3) It’s my personal, dedicated computer. It came with a huge disk, and it’s all mine, so I neither understand nor care where anything lives as long as it shows up on my Desktop, Start Page, or equiv.

3a) like (3), only it’s a netbook with limited storage, and I’d rather have almost everything arduino go on the microSD card instead of the limited internal flash.

3b) like (3), but I used to administer xxxx, and I have very definite opinions about where things should go and how they should be grouped so that I can back things up conveniently. I probably hate the way that Microsoft thinks Applications should be scattered across directories, registries, and so on.


I don’t want my libraries to go away when I upgrade the IDE, but keeping the compilers around and forcing a manual upgrade is a recipe for the same sort of problems we used to have with linux Arduino installs having “random” versions of the compiler that don’t match … anything or anyone else. In fact, so far, the backward compatibility of … everything … has been relatively poor anyway. Also, while I’ve been complaining about a GB of user-specific files, the whole Arduino IDE (with all three compilers) is still pretty small compared to the typical game download or even many “windows update” attempts…

BillW

[Roger Clark]

Bill

I know that there is the idea that when people update their copy of Arduino, that they don't need to update the compilers, but looking at the Boards Manager, it seems that ever release so far has a corresponding Boards Manager entry

This seems to apply to both AVR and SAM (and probably the others)

Looking at my Boards Manager, my AVR board version seemed to be 1.6.2 and the IDE was at 1.6.4, but it didn't update the cores for AVR when it installed the latest IDE.

So as a test I changed the drop down menu to select 1.6.3 for AVR, and the Boards manager went off and downloaded a new set of cores for AVR, but also 40Mb worth of AVR compiler.

I suspect that if I then selected 1.6.4 it would do the same thing again.

I also noticed that there are versions of the AVR cores to 1.6.7 which is far ahead of the IDE.

If these are independent numbering schemes, I'd personally prefer if they used a different numbering system, as previously I've presumed that the numbering in those menu's related to the IDE version number, but it cant be that if I can pick 1.6.7

Surely when I updated the IDE from 1.6.3. to 1.6.4 it should have automatically updated any cores I had installed, if there was a new version - especially for AVR - as this seems like a change from previous functionality where IDE and cores version was linked. as the cores were part of the IDE install.

Which goes back to the initial install.

Most systems allow you to choose what components you want to install e.g. minimum, or full or custom

Hence when a sys admin installs Arduino, if they know that the users need SAM or Intel support, they could install it using their admin rights.

I agree that at a later date someone many change their mind and ask for a compiler they have not asked for at install time, but normally on Windows this is handled by the installer - not from inside the program. e.g. Ref MS Office etc

IMHO, Cores are different from compilers, they are not binary'ies and hence can be "installed" without admin rights, as long as they are put in the correct location.

So the storage location for cores is a bit more flexible.

But I can understand that on multi user systems, that each user having their own 10mb copy of a core is not a good idea.

so I don't have a ideal solution for this

I'm happy with Cores in UserSketchesFolder/hardware

[Federico Fissore]

Thank you Brian, that's one good place for storing stuff on Windows.
Let's see what else comes up for mac and linux

Federico

[Matthijs Kooijman]

Hi,

> With recent versions of Windows the typical location is
> (CSIDL_COMMON_DOCUMENTS)
> C:\Users\Public\Documents\{Company}\{Product}\{Version}\ which might be
> C:\Users\Public\Documents\Arduino\IDE\1.6\ for the Arduino IDE. I
> believe that location was added in Windows 7.

Wouldn't that have a single location shared by all users? I'd think
that you would want to have a per-user location, to prevent security and
denial-of-service problems (with one user updating a core, breaking
another users' sketches, or replacing gcc with a keylogger, to name a
few examples).

I can see how a shared location could be useful (no need to have dozen
identical copies of the same toolchain installed), but then the IDE
should perhaps do some checksumming of the files (though that still
leaves a race condition between checksumming and running gcc). Having a
single location that is writable only by the administrator could work,
though, where the IDE uses the toolchain from there if it's available
and installs its own in a per-user location if it's not?

Gr.

Matthijs

[Federico Fissore]

Thank you William. I think you're digressing. I've asked for disk
locations, and you suggestions seem vague to me. Can you list the actual
folders paths you have in mind?


William Westfield ha scrito il 09/06/2015 alle 05:25:

>
> What qualifies as a “plugin” and How big is it?
>

No one knows in advance because we don't make all of them [1]: it can be
as small as a custom AVR core using an already installed toolchain, or a
whole new core shipping toolchain, debuggers, simulators and gigabytes
of docs in PDF format

[1]
https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls

[Federico Fissore]

Thank you Roger. This is still digressing from the original question,
but my comments follow

Roger Clark ha scrito il 09/06/2015 alle 07:17:

>
> If these are independent numbering schemes, I'd personally prefer if
> they used a different numbering system, as previously I've presumed that
> the numbering in those menu's related to the IDE version number, but it
> cant be that if I can pick 1.6.7
>

Yes, versions became independent the moment we broke the double bond
between the IDE and its Arduino cores. It made no sense to release, for
example, AVR core 2.0.0 so we just kept on incrementing their versions.

> Surely when I updated the IDE from 1.6.3. to 1.6.4 it should have
> automatically updated any cores I had installed, if there was a new
> version - especially for AVR - as this seems like a change from previous
> functionality where IDE and cores version was linked. as the cores were
> part of the IDE install.
>

You want such behaviour, others want the opposite and they will complain
that, the moment they upgraded the IDE, their sketches stopped working
because of a broken core (for example: I released a version of avrdude
which was unusable with an external programmer)
We took the conservative path but we are working on a notification
system that will gently inform about available updates

>
> I agree that at a later date someone many change their mind and ask for
> a compiler they have not asked for at install time, but normally on
> Windows this is handled by the installer - not from inside the program.
> e.g. Ref MS Office etc
>

That's a nice suggestion but it will only work on Windows. What about
mac and linuxes?

Federico

[Peter Feerick]

How about  downloaded cores go into the $SKETCHBOOKFOLDER/hardware by default unless

(1) there exists a 'hardware' folder in the $IDEPROGRAMFOLDER (which means you would ship with no cores by default, and manage all installs via the board manager - or install the AVR core into the default $SKETCHBOOKFOLDER/hardware folder - or some other option I haven't considered) -

or

(2)  there exists a 'portable' folder, in which case they are installed in a 'hardware' sub-folder of 'portable'

or

(3) there exists an alternate specified path in preferences.txt for a 'cores' install folder

This allows for portable self-contained installations, centrally managed installations and defaults to sticking stuff in the user profiles folder otherwise. And if you need a network location or other central location, that is available too via a preference.

Pete

[Federico Fissore]

I agree, location should be private to each user

[Brian Cook]

Hi,

>> With recent versions of Windows the typical location is
>> (CSIDL_COMMON_DOCUMENTS)
>> C:\Users\Public\Documents\{Company}\{Product}\{Version}\ which might be
>> C:\Users\Public\Documents\Arduino\IDE\1.6\ for the Arduino IDE.  I
>> believe that location was added in Windows 7.
> Wouldn't that have a single location shared by all users?

I believe that's the goal; to identify a reasonable shared location for core related files.

> I'd think that you would want to have a per-user location...

At least on Windows, that's what the IDE currently uses; Roaming for the current user.

> denial-of-service problems (with one user updating a core, breaking
> another users' sketches,

I assume, with correct versioning, that would not be a problem.

> or replacing gcc with a keylogger

That could be a problem.

> I can see how a shared location could be useful (no need to have dozen
> identical copies of the same toolchain installed),

Exactly.

> but then the IDE should perhaps do some checksumming of the files

Seems like a reasonable solution.

> Having a single location that is writable only by the administrator could work,
> though, where the IDE uses the toolchain from there if it's available
> and installs its own in a per-user location if it's not?

I believe I mentioned it in my previous post: For Windows shared programs are typically placed in C:\Program Files\Common\ (there is a CSIDL for the location).

However, in my day job, I have dealt with alternative path solutions.  They can easily become a support nightmare.  It is far better to choose a single reasonable location for a given file-set type and stick with that choice.

- Brian

[Federico Fissore]

Thank you Peter, my comments follow

Peter Feerick ha scritto il 09/06/2015 alle 09:36:
> How about downloaded cores go into the $SKETCHBOOKFOLDER/hardware by
> default unless
>
> (1) there exists a 'hardware' folder in the $IDEPROGRAMFOLDER (which
> means you would ship with no cores by default, and manage all installs
> via the board manager - or install the AVR core into the default
> $SKETCHBOOKFOLDER/hardware folder - or some other option I haven't
> considered) -

When we were developing Boards Manager, we internally shipped what I
called "naked" IDEs (naked as motorcycles are [1]), with no cores at
all. We dropped that because that would have obliged first timers to an
additional install step, thus breaking UX

> (3) there exists an alternate specified path in preferences.txt for a
> 'cores' install folder
>

I like the idea of an customizable path: gives back control. Thank you

Federico

[1] http://www.ducati.com/bikes/monster/index.do

[Matthijs Kooijman]

Hey Federico,

> Thank you William. I think you're digressing. I've asked for disk
> locations, and you suggestions seem vague to me. Can you list the
> actual folders paths you have in mind?

I actually think an important point that William is making is: "It
depends". There are various usecases, each with different requirements
about the storage path...

Gr.

Matthijs

[Andrew Kroll]

Well, Linux -- I throw anything that is third party into /opt

Now, if I I would need additional hardware stuff shared between accounts (I don't but...) there's always the ability to make a local arduino user group and use symlinks.
You create a special local users area that is writable contains the shared bits that all users can add/modify/etc.
You symlink to what you want as needed, on-the-fly.
The execute security risk is reduced because the user must be part of the user group in order to be able to execute anything in this shared area. On top of this, you could always sandbox the IDE and limit what it can see and operate with.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[Federico Fissore]

Thank you Andrew

I think we should encourage a common and fast installation flow (like
the current "unzip and double click") while ensuring the IDE is open
enough (in its parameters) to be hacked the way you describe

Federico

[Andrew Kroll]

Well, symlinks on Linux "just work" with the IDE as-is. There is nothing to configure within the IDE.

I do this often in order to quickly change libraries in my own work dir. I just close the IDE, change the symlink, open the IDE and it automagically will see and use the different library.

This of course also works also with my own makefile system and Netbeans too.

I only use the IDE when I need to ensure that a library I work on will work for the noobs...

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Federico Fissore]

But each use case is driven by one possibly unknown json file provided
by someone I probably don't know: how can I make the IDE behave in
different ways while still maintaining it easy to use and hack?
For the record: I'm not saying "I can't" nor "it's too hard", I just
want you to propose a solution, because, at the end, this is gonna be
coded, and code demands details

federico

[William Westfield]

> Can you list the actual folders paths you have in mind?

Hmm. I have a thought experiment:

Suppose that I have downloaded the IDE and added assorted plugins and custom boards and libraries.
Everything works exactly how I want it to. Yeah!

Now, I want to copy that state to other peoples’ computers. My kids, my class, a seminar I’m teaching, an online tutorial, whatever. I would like to be able to say “here, take this and install it on your computer by doing xxx, and you should have all the same stuff I used to create things that I’m trying to give you.”

If I could do that, I don’t care where all of the bits were originally. But I need some kind of “repackaging” option. Instructions that involve copying three different sub-directory trees to three different places (system, libraries, documents), some of which are only root-writable, and some of which are normally “hidden”, is not good. The 1.0.x Arduino.app on MacOS was pretty close to achieving this. The 1.6.4 windows structure is further away :-(

I think the “repackage” operation/result would meet all the other use cases as well.

Does that make more sense? Is it possible?

(I used to laugh at the TI Launchpad tutorial, because “take the board out of its box” was somewhere around step 26, after you downloaded and (separately) installed the IDE, the Drivers, Stellarisware, Programming software, and etc. Arduino was SO much quicker - download and install ONE thing, plug in the board and go. But it’s getting worse; an ARM or Intel based Arduino now has two separate download steps…)

BillW

[Federico Fissore]

I think this is a super cool idea: I'll bring it to the UX designers
table. Thank you Bill

Federico

[Tom Igoe]

Though that’s a nice ideal, I don’t think it’s the reality of most consumer applications these days. For example, try copying Photoshop from one computer to another that way. Used to work 10 years ago, not so much now.

t.

[Peter Olson]

> On June 9, 2015 at 4:44 AM Federico Fissore <f.fi...@arduino.cc> wrote

>
> Matthijs Kooijman ha scritto il 09/06/2015 alle 10:08:
> > Hey Federico,
> >
> >> Thank you William. I think you're digressing. I've asked for disk
> >> locations, and you suggestions seem vague to me. Can you list the
> >> actual folders paths you have in mind?

I'm not sure where to insert my comment into this thread, but let me digress :-)

I program Arduino both for personal reasons and as my job.

At my job, I have no experience with 1.6.x: I am pinned at 1.5.8.1 from the
other guys.

In the past I have been happy (on GNU/Linux) to pick a directory anywhere I have
access to (such as /srv/arduino-1.0.5 ) and untar the distribution there and
symlink to it as needed.

At my job I have accommodated multiple versions with some effort by copying the
working version out of the way before the new install insists on deleting it.
This is harder on Windows 7 than on GNU/Linux. I am nervous about driver
support.

At my job, I must never be in a position where I cannot recreate a specific
version of source code and development environment as a binary package.

I understand I am not your target user.

As I automate, the procedures I use will probably run as a different user from
the one I use for development and testing.

At my job, I cannot pick a directory (as in GNU/Linux, at my job I am at Windows
7) to compartmentalize my development environment, so all this discussion about
the %APPDATA% directories makes me really nervous. I have to support machines
in the field, and I am unsure how to keep them current. (I am developing on
Due, so this is a specific issue now.)

Peter Olson

[Simon John]

I think we'd need a list of the possible files you mean.

Personally I'd say user cores/libraries can stay in ~/sketchbook/{hardware,libraries}/ and board manager/library manager should install to there, certainly not ~/.arduino15/ which should just contain preferences.txt (and whatever small config files are required, not binaries).

Toolchains (gcc, avrdude etc.) have to go in /usr/bin/ - even if that means symlinking them from /usr/share/arduino/tools/, that's how Debian/Ubuntu package it. If you really must have an all-in-one location then i'd say /opt/, possibly with symlinks into /usr/local/bin/ or /usr/bin/

"system" cores/libraries, i.e. the ones that come with the IDE, need to go in /usr/share/arduino/{hardware,libraries}/

Really you just need to see how distro's are already packaging the IDE and mimic that.

One thing I really dislike about the current directory structure is the versioning, what use is having last.ide.1.6.4.hardwarepath, last.ide.1.6.3.daterun, and even 1.6.4-628-g545ffde in the cores (~/.arduino15/packages/*) is no use to anyone surely?

[Federico Fissore]

Thank you Simon, unfortunately and AFAIK what you propose is undoable

Simon John ha scritto il 09/06/2015 alle 14:52:
>
> Toolchains (gcc, avrdude etc.) have to go in /usr/bin/ - even if that
> means symlinking them from /usr/share/arduino/tools/, that's how
> Debian/Ubuntu package it. If you really must have an all-in-one location
> then i'd say /opt/, possibly with symlinks into /usr/local/bin/ or /usr/bin/
>
> "system" cores/libraries, i.e. the ones that come with the IDE, need to
> go in /usr/share/arduino/{hardware,libraries}/

Normal users do not have write access to those locations. And even
showing dialogs or whatever to get temporary Super User permissions is a
no go: every time you say "dialog" a UX designer dies

>
> Really you just need to see how distro's are already packaging the IDE
> and mimic that.

I will not mimic what distros do: I'd rather let them package the IDE
the proper way and disable Boards Manager.
One example of this good practice is firefox: ever noticed firefox on
windows autoupdates itself, while it doesn't on linux?

>
> One thing I really dislike about the current directory structure is the
> versioning, what use is having last.ide.1.6.4.hardwarepath,
> last.ide.1.6.3.daterun, and even 1.6.4-628-g545ffde in the cores
> (~/.arduino15/packages/*) is no use to anyone surely?
>

Yes, that's a bit ugly, I agree, but it has a cool and positive side
effect: we don't need yet-another-file to use as software catalog. The
catalog is the folders structure itself: having hardware/sam/1.6.4 means
I've installed sam core, version 1.6.4

Federico

[David Mellis]

One requirement that seems to be emerging from this discussion is a way for an administrator to easily install the Arduino software plus any required hardware support -- for later use by non-admin users. It seems like that might require the installation of the hardware support in non-user specific folders, since presumably normal users can't access the administrators personal folders. I don't have a good solution for this, unfortunately, but it does seem like a good use case to support.

[Federico Fissore]

One way to solve this requirement is by working with attempts
The IDE may attempt to write in its own folder and, if unable to,
fallback to ... something
It's the goal of this discussion to find out where, so let's keep it going

[Federico Fissore]

For the record, such result is already achieavable using portable folder

[William Westfield]

> I don’t think it’s the reality of most consumer applications these days.

True. Which SUCKS. Most backup utilities have given up on even trying to back up installed applications. After all, it would provide a mechanism that might be used for piracy. ("If you need to reinstall, make sure you have your license key handy. I’t on the pamphlet that came with your SW that you haven’t looked at since you installed it the first time.” Sigh.)

Open Source stuff doesn’t have the security requirement, and should be easier to work with.

BillW

[Johney Five]

First I'm a total newbie here.I continually get these emails gladly reading to try to gain more comprehension of the new toy I got to play around with my first thing I think will try to make will be a simple doorbell that will play a  selected song as the chime.my gf expressed interest I. This after the 2 piniacolada song from American dad.

My contribution to this post string is this: why can't it just be made a more simple feat that users can keep all content on their system at once. If you want to strive for simplicity and compatibility simply make references to all cores and dependencies and compatibility requirements a system defined operation and intelligent uploads for sketches. So you can have library version x of x drivers for x cores as the need arises. Suppose you want to have a device with sd capabilities for one card type but no need for others you should be able to isolate that bit of the resource automatically when uploading the sketch by defining *SanDisk 128mb parameters or *complete upload of library. This could reduce the workload of people like me who are wanting to try to use this stuff but have no prior experience.... I have seen a lot of issues of compatibility issues complained about here and I think this could be solved if we could do this in the system. Better yet it would be nice to be able to emulate our board in the system and be able to see if stuff works before trying it... I know myself if I have a failed attempt when I think I've finished something and got it app put together I'm apt to toss it aside for weeks if it don't work as expected. Make it so we can do our wiring on a system and check all variables.... I know the biggest thing keeping me from actually building anything yet is I'm scared to burn up my chip with over voltage. Supposedly I can give it anywhere from 5-12v no clue on amperage no clue on estimating run times with select power supplies.... ie I have 20 18650 cells I have a 12v car jumpstart pack I have a 8cell  AA clip any of which would likely fry the board if hooked up. Even the 18650 cells if run all in parallel with only 3.7v pushing a combined 300 or so amps would no doubt fry it...

I have no clue how these emails get circulated either how one starts them or anything so please forgive me for my including my own personal issues in these matters....
I would greatly appreciate if any of you had the time to directly help me get more familiar with the territory and would take the time to contact me directly at fir...@gmail.com thanks for your concideration

On Jun 8, 2015 10:38 AM, "Federico Fissore" <f.fi...@arduino.cc> wrote:

The target location for optional cores and compilers has become a recurrent question but so far I haven't read a feasible proposal

So let's try gather them all: you have an IDE, you have installed it using your OS utilities (apt-get, dmg, exe installer), you have "downloadable plugins" and you want to install a new fancy one

Where will its files go?

[Carlos Pereira]

It looks like at least some of the things discussed here are making their way into the repository (https://github.com/arduino/Arduino/pull/3404).

I was wondering, where are these things documented (if they are already), or where should we start documenting them. Including things like how the IDE behaviour changes if there is a "portable" folder created.

The getting started page (https://www.arduino.cc/en/Guide/HomePage) is the only place I've seen with articles that could cover these kind of features, but a lot of these pages haven't been kept updated. Ideally we should have a repository where people could add pull requests, in the same way that we have the reference repositories in different languages (https://github.com/arduino/reference-en).

Perhaps this kind of discussion should go into its own post, however the main reason I ask here is because I am not sure if there is something already in place that I haven't been able to find.