Why is IDE using puting executables under users .arduino15 rather than in sketchbook or better in the IDE tree?
437 views
Skip to first unread message
Bill Perry
May 12, 2015, 4:35:27 PM5/12/15
to devel...@arduino.cc
I just starting using the 1.6.4 IDE and it doesn't come with "out of the box" DUE support.
When I installed the DUE support, it installed it all down under the users home directory under the users .arduino15 directory.
I see many potential issues with not keeping add on executable tools in the same tree as the IDE install tree
(i.e. installing them in the same tree with the base IDE tools vs in a separate area)
But if packaging addons are going be disjoint and separate area from the IDE install directory itself, why is it not using the users sketchbook area as was done when adding 3rd party hardware cores, libraries, etc...?
In my view any/all packaging "stuff" should be down under the users sketchbook area so that it is all together, particularly any packaging stuff that is not executables.
The reason this can be important is that a user may choose to relocate his sketchbook area away from his home directory.
This might be done for a variety of reasons including space reasons or for sharing reasons.
In my case I test a library against up to 15 different IDE versions and I relocated the sketchbook area to a network share so that it can be shared across many different environments including different operating systems.
Currently the users home directory doesn't really contain anything because it doesn't have to because anything that is needed is under the sketchbook directory or the IDE install directory and this new model of putting packages under the .arduino15 directory breaks all of that.
Other environments may also break or have issues because of this.
======================================================================
Now consider some of the implications of separating the add on IDE and compiler tools away from where the IDE itself is installed.
And more importantly the implications of putting executables down under the users home directory.
My view is that is bad practice to put executable and entire compiler tools down under the user's home directory.
Currently, (Prior to about 1.6.4) all the actual executable tools are all together under the tree where the IDE is installed.
This creates a nice atomic blob for sharing and testing. It puts everything all together (at least from a tool perspective) for a given IDE install - which is specific to an OS and processor.
There are no tools or executables under the user's home directory.
Having everything together under the IDE install directory allows easily sharing everything for a given release and keeps everything together and does not require that every single user have their own copy of the add on tools.
Remote mounted home directories can also have problems with placing files under .arduino15 since this directory will now have actual executables.
If the home directory is being used across multiple OS's or different architectures then this strategy no longer works.
My preference would be move all the "packages" stuff out of the users .arduino15 directory and into the users sketchbook directory but also to have an option to place them (tools and libraries) into the IDE install directory of the currently running IDE to allow the IDE install directory be an atomic unit of all the tools.
It also allows creating atomic blobs of IDEs which contain tools and libraries which is great for regression testing.
I understand why there can be limitations on writing to the IDE install directory (for good reasons), which is what originally lead to the creation of the sketchbook area, but now that we are talking about executables, I think things are different and that there needs to be a way to get the updated addon tools back into the IDE install directory - even if it is only for those with appropriate write permissions.
It seems really messy to start assuming that a machine is only being used by a single user so it is ok to place add on extensions to the IDE, including tool executables down under the user's home directory.
And then what about classroom environments? It seems messy to have to run around and update everyone's home directory for tools and libraries rather than being able to have a central location for the tools and libraries - which could be done if the IDE could place add the on packages inside the IDE install directory.
I'm also trying to figure out how this methodology is going to fit in with things like the debian packaging system.
===================
And oh by the way, a bit off topic but the IDE leaves around a few turds when the DUE 1.6.4 package is uninstalled. There are some sub directories left around under .arduino15 and some symbolic links on a couple of .o files that no longer point to anything.
When I installed the DUE support, it installed it all down under the users home directory under the users .arduino15 directory.
I see many potential issues with not keeping add on executable tools in the same tree as the IDE install tree
(i.e. installing them in the same tree with the base IDE tools vs in a separate area)
But if packaging addons are going be disjoint and separate area from the IDE install directory itself, why is it not using the users sketchbook area as was done when adding 3rd party hardware cores, libraries, etc...?
In my view any/all packaging "stuff" should be down under the users sketchbook area so that it is all together, particularly any packaging stuff that is not executables.
The reason this can be important is that a user may choose to relocate his sketchbook area away from his home directory.
This might be done for a variety of reasons including space reasons or for sharing reasons.
In my case I test a library against up to 15 different IDE versions and I relocated the sketchbook area to a network share so that it can be shared across many different environments including different operating systems.
Currently the users home directory doesn't really contain anything because it doesn't have to because anything that is needed is under the sketchbook directory or the IDE install directory and this new model of putting packages under the .arduino15 directory breaks all of that.
Other environments may also break or have issues because of this.
======================================================================
Now consider some of the implications of separating the add on IDE and compiler tools away from where the IDE itself is installed.
And more importantly the implications of putting executables down under the users home directory.
My view is that is bad practice to put executable and entire compiler tools down under the user's home directory.
Currently, (Prior to about 1.6.4) all the actual executable tools are all together under the tree where the IDE is installed.
This creates a nice atomic blob for sharing and testing. It puts everything all together (at least from a tool perspective) for a given IDE install - which is specific to an OS and processor.
There are no tools or executables under the user's home directory.
Having everything together under the IDE install directory allows easily sharing everything for a given release and keeps everything together and does not require that every single user have their own copy of the add on tools.
Remote mounted home directories can also have problems with placing files under .arduino15 since this directory will now have actual executables.
If the home directory is being used across multiple OS's or different architectures then this strategy no longer works.
My preference would be move all the "packages" stuff out of the users .arduino15 directory and into the users sketchbook directory but also to have an option to place them (tools and libraries) into the IDE install directory of the currently running IDE to allow the IDE install directory be an atomic unit of all the tools.
It also allows creating atomic blobs of IDEs which contain tools and libraries which is great for regression testing.
I understand why there can be limitations on writing to the IDE install directory (for good reasons), which is what originally lead to the creation of the sketchbook area, but now that we are talking about executables, I think things are different and that there needs to be a way to get the updated addon tools back into the IDE install directory - even if it is only for those with appropriate write permissions.
It seems really messy to start assuming that a machine is only being used by a single user so it is ok to place add on extensions to the IDE, including tool executables down under the user's home directory.
And then what about classroom environments? It seems messy to have to run around and update everyone's home directory for tools and libraries rather than being able to have a central location for the tools and libraries - which could be done if the IDE could place add the on packages inside the IDE install directory.
I'm also trying to figure out how this methodology is going to fit in with things like the debian packaging system.
===================
And oh by the way, a bit off topic but the IDE leaves around a few turds when the DUE 1.6.4 package is uninstalled. There are some sub directories left around under .arduino15 and some symbolic links on a couple of .o files that no longer point to anything.
Mattias Schlenker
May 12, 2015, 4:52:02 PM5/12/15
to devel...@arduino.cc
This is also being discussed in the github issue tracker. I am not happy with the situation as well since my $HOME tends to be mounted noexec and I still have one machine left that just runs 32 bit Ubuntu smoother. And I am syncing home directories across machines.
To put it to extremes I got some customers that just want to roam with a home directory between 64 bit Linux and 32 bit Windows. For firefox or thunderbird this is possible without weird hacks.
I think a viable solution would be to ask for a root password and then install to the IDEs directory. If no root password is provided, the installer asks whether the installation should be done to his/her home or profile.
I think this fits well for both cases: Yours and mine were binaries in the home suck and universities where students should be able to install board support without having to ask the admin.
Yours, Mattias
--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.
Roger Clark
May 12, 2015, 6:38:50 PM5/12/15
to devel...@arduino.cc
I've also posted to the GitHub issue, and agree its not a good decision to put compilers etc into this location.
Its also against the Microsoft guidelines (on WIndows) and is likely to be Synced to the cloud.The response I got from Federico was that he had consulted people and they had agreed the approach they took was the correct one.
However its unclear who he consulted or what the reasoning was for the decision.
So far, I have not seen anyone else saying that it was a good decision. AFIK All comments have been negative.
Cheers
Roger
Bill Perry
May 12, 2015, 6:39:32 PM5/12/15
to devel...@arduino.cc
On 2015-05-12 15:51, Mattias Schlenker wrote:
>
> This is also being discussed in the github issue tracker.
>
What is the Issue number? or link?
>
> This is also being discussed in the github issue tracker.
>
I looked but couldn't seem to locate it.
> I am not happy with the situation as well since my $HOME tends to be mounted noexec and I still have one machine left that just runs 32 bit Ubuntu smoother. And I am syncing home directories across machines.
>
> To put it to extremes I got some customers that just want to roam with a home directory between 64 bit Linux and 32 bit Windows. For firefox or thunderbird this is possible without weird hacks.
>
Currently the IDE is using a brain dead "windows" single-user single machine type mindset.
But even on Windows, applications don't install the application or their add-ons in the user's home directory; they typically install in their own area.
> I think a viable solution would be to ask for a root password and then install to the IDEs directory. If no root password is provided, the installer asks whether the installation should be done to his/her home or profile.
>
I don't want a program to just jump to root privileges.
For example, maybe there is a "arduino" group that is created and used for managing all the arduino IDE stuff.
If i'm in that group, I should not need to run the IDE with escalated privileges to do the maintenance.
Also, I'm not sure how that would work since the IDE is already running prior to needing the escalation and I for sure don't want the IDE to ever have root permissions much less when starting.
> I think this fits well for both cases: Yours and mine were binaries in the home suck and universities where students should be able to install board support without having to ask the admin.
>
> Yours, Mattias
>
>
You could plug in your "Arduino" stick and start working as all your tools, libraries, add on packages, sketches, etc could all be on the USB stick.
All that is missing is a tweaked start up script and possible a tweak in the IDE itsel to allow pointing IDE to a preferences.txt file on the flash drive to be able bring up the IDE without needing a users home directory for anything.
Peter Feerick
May 13, 2015, 4:56:13 AM5/13/15
to devel...@arduino.cc
Just regarding what Federico did (or didn't say), he said back on the 10th of April that "We conducted some users interviews before taking that step: the majority preferred to keep cores across different sketchbooks, while sketchbooks were used as "projects". So cores were to be stored outside of the sketchbook, and libraries were to be local to it"
.
So it wasn't that the implementation was 'correct to Microsoft guidelines' but was 'correct based on the response on the users questioned'.
The github issue may be this one: https://github.com/arduino/Arduino/issues/2902
Regarding the "portable IDE", Federico also said this around the same time : "by making a "portable" folder in the IDE folder, everything is stored in there, thus allowing for a 'portable IDE' ". I have just tried this, and this is 'nearly' satisfactory. While it is great for a flash drive / portable setup - as it stores all the cores and the sketchbook in the 'portable' subfolder, it isn't good enough for what is being discussed here.
I would suggest that a preferences.txt option be created, which would trigger the IDE to keep additional boards and the IDEs preferences in the IDE folder. An alternative could be creating a folder with a name like 'profile' - thus very similar to the portable mode currently used.
Pete
Peter Feerick
BIT, BLDes CQU
BIT, BLDes CQU
Bill Perry
May 13, 2015, 1:32:00 PM5/13/15
to devel...@arduino.cc
On 2015-05-13 03:55, Peter Feerick
wrote:
Just regarding what Federico did (or didn't say), he said back on the 10th of April that "We conducted some users interviews before taking that step: the majority preferred to keep cores across different sketchbooks, while sketchbooks were used as "projects". So cores were to be stored outside of the sketchbook, and libraries were to be local to it" .
So it wasn't that the implementation was 'correct to Microsoft guidelines' but was 'correct based on the response on the users questioned'.
Actually that is an incorrect conclusion base on the quote above.
The quote above above doesn't fully answer the question of where to place the add on core components.
It merely stated that the users said that they wanted to use it across different sketchbooks.
It is also a bit odd as how many users actually use multiple sketchbooks?
I see many forum posts where users are struggling to just to use one sketchbook location.
And for all the crazy stuff I do with 15+ IDE versions, on multiple OS's and platforms, I still only have one sketchbook directory.
Maybe others use multiple sketchbooks, but I don't since I want libraries and sketches to easily be re-used across different IDEs (arduino.cc, chipkit, arduino.org, etc...) including different versions of the IDE.
The quote above merely said that those users that were polled preferred that the "cores" be outside the sketchbooks. It didn't say they asked them anything about where they wanted them to live.
And then did the users polled really understand what adding a core really means in terms of files, executables etc...?
It also does not mention the desired behavior of those cores across different IDE release versions - which I view as the more important issue.
So beyond the desire to use/keep cores used across "sketchbooks" that quote above is silent.
To me what is more important is what happens across IDE versions. Do the files stay with a given IDE or move across to different IDE versions?
There are arguments for doing it either way.
Having an option to bundle everything together (storing the add on cores and executables in the IDE install tree) is best for regression testing and testing against different platforms
as well as allowing multiple users to all share the updated IDE executables.
I still believe that it is totally ludicrous to put binary executables down under each user's home directory.
Not only is it wasting space since every user has to install them under their home directory but there are many real world scenarios where this creates problems and doesn't work.
(home directory permissions, remote mounts, home directory space concerns, etc...)
I also believe it is a very bad practice to extend executables and tools (in this case the arduino IDE) that are not down under that user's home directory with tools and executables that are now under a user's home directory.
You have just created a tight coupling dependency that cannot be guaranteed since the user can't necessarily control the IDE binaries but is responsible for their extensions.
Also, how do you handle ensuring that older versions of the IDE can run and coexist with newer versions when cores have been added?
Since the IDE want's to store its add ons down under the users home directory, there can be issues when using multiple/different versions of the IDE.
Looking forward how could 1.6.4 have its own set of cores kept separate from say a 1.7.0 and its add on cores?
This becomes a real issue if there start to be backward compatibility issues like say a new arm toolset is released with 1.7.0
it works and is required with 1.7.0 but does not work with 1.6.4
Having the ability to store the add on core down under the IDE install tree solves many of these issues, if you want to keep the core add on packages and the IDE as an atomic unit which is how it worked prior to the ability to add on a core through the IDE.
This new addon methodology was simply not fully thought out.
The quote above above doesn't fully answer the question of where to place the add on core components.
It merely stated that the users said that they wanted to use it across different sketchbooks.
It is also a bit odd as how many users actually use multiple sketchbooks?
I see many forum posts where users are struggling to just to use one sketchbook location.
And for all the crazy stuff I do with 15+ IDE versions, on multiple OS's and platforms, I still only have one sketchbook directory.
Maybe others use multiple sketchbooks, but I don't since I want libraries and sketches to easily be re-used across different IDEs (arduino.cc, chipkit, arduino.org, etc...) including different versions of the IDE.
The quote above merely said that those users that were polled preferred that the "cores" be outside the sketchbooks. It didn't say they asked them anything about where they wanted them to live.
And then did the users polled really understand what adding a core really means in terms of files, executables etc...?
It also does not mention the desired behavior of those cores across different IDE release versions - which I view as the more important issue.
So beyond the desire to use/keep cores used across "sketchbooks" that quote above is silent.
To me what is more important is what happens across IDE versions. Do the files stay with a given IDE or move across to different IDE versions?
There are arguments for doing it either way.
Having an option to bundle everything together (storing the add on cores and executables in the IDE install tree) is best for regression testing and testing against different platforms
as well as allowing multiple users to all share the updated IDE executables.
I still believe that it is totally ludicrous to put binary executables down under each user's home directory.
Not only is it wasting space since every user has to install them under their home directory but there are many real world scenarios where this creates problems and doesn't work.
(home directory permissions, remote mounts, home directory space concerns, etc...)
I also believe it is a very bad practice to extend executables and tools (in this case the arduino IDE) that are not down under that user's home directory with tools and executables that are now under a user's home directory.
You have just created a tight coupling dependency that cannot be guaranteed since the user can't necessarily control the IDE binaries but is responsible for their extensions.
Also, how do you handle ensuring that older versions of the IDE can run and coexist with newer versions when cores have been added?
Since the IDE want's to store its add ons down under the users home directory, there can be issues when using multiple/different versions of the IDE.
Looking forward how could 1.6.4 have its own set of cores kept separate from say a 1.7.0 and its add on cores?
This becomes a real issue if there start to be backward compatibility issues like say a new arm toolset is released with 1.7.0
it works and is required with 1.7.0 but does not work with 1.6.4
Having the ability to store the add on core down under the IDE install tree solves many of these issues, if you want to keep the core add on packages and the IDE as an atomic unit which is how it worked prior to the ability to add on a core through the IDE.
This new addon methodology was simply not fully thought out.
Peter Feerick
May 13, 2015, 7:44:41 PM5/13/15
to devel...@arduino.cc
Bill,
The only comment I will make in regard to the aforementioned quote is that from what Federico later said about atomicity, he seems to only be thinking along the lines of one 'installed' copy of the Arduino IDE, and conceptually separating the IDE from the cores so that one can update without affecting the other. However, I'm not Federico, so I'm not going to put words in his mouth. I'm sure he'll comment and explain further at some point. And I wonder about the multiple sketchbook folders also - I have multiple sketches and subfolders, but still only one sketchbook folder.
I also don't like the new structure, for the security and space reasons you mentioned before, but also for the fact that I wouldn't be able to have five different versions installed side by side which don't affect each other. I currently have several versions, ranging from 0022 to 1.0 to 1.0.6 and 1.6.4, plus teensy mods and digistump mods, etc, and they all peacefully co-exist, and the install folders contain all the IDE files and cores needed. I can then copy the core folder to another computer, and access the sketchbook folder over the network, and I'm on my away again.
That's why at the least, a preference option is needed to revert the old behaviour, if the new behaviour is really what is wanted by some select group. Portable mode is there, so why not 'IDE+cores' mode.
Pete
Tim Leek
May 13, 2015, 8:49:00 PM5/13/15
to devel...@arduino.cc
Bill,
Would other apps store 'user data' under the windows "program files" folder (where the ide is usually installed)?
The \arduino15x sub folder is currently being used. That could easily become \arduino17x in the future.
Tim
Bill Perry
May 13, 2015, 9:16:33 PM5/13/15
to devel...@arduino.cc
On 2015-05-13 19:49, Tim Leek wrote:
> Bill,
>
> Would other apps store 'user data' under the windows "program files" folder (where the ide is usually installed)?
In this case were are not really talking about user data.
> Bill,
>
> Would other apps store 'user data' under the windows "program files" folder (where the ide is usually installed)?
We are talking about executable extensions to an installed a program. And the executable extensions are dependent the installed program but are not co-located with the installed program but rather are being strored under the users private area.
I place a big distinction between libraries and a new core which also contains executables.
libraries can run across many different IDEs and are not tied a give OS.
The core extension/add-ons are potentially tied to a given IDE and are specific OS, which makes it impossible to share home directories across different OS enviroments.
Another problem you then start to have is that if you have multiple IDEs installed, when you start to install the add-ons they can start to stomp on each other.
With this new methodology, you have no way to create atomic blobs of different versions of the IDE on the same machine for testing and support.
But in the bigger picture, I still think it is very bad practice to extend an installed program with new executables that are stored down under the users home directory rather than installed along side the original installation.
Can anyone identify another commercial product that does this?
>
> The \arduino15x sub folder is currently being used. That could easily become \arduino17x in the future.
>
> Tim
>
>
Peter Feerick
May 13, 2015, 9:36:07 PM5/13/15
to devel...@arduino.cc
I'm will Bill on this one. It is just messy, insecure, and
cumbersome. And I would like to see how a compiler is considered
'user data'... it is really optional program files - I just install
that support and then don't think about it again.
If you extent that idea further, the next version of Visual Studio should also just install the IDE in the Program Files folder, and the Visual C#, C, VB, etc compilers in your app data folder also. Where does it stop? Is it still a good idea? I'll leave that for you to decide :) And that's before considering that roaming profiles are synced - either over a network or via the intenet...
Pete
If you extent that idea further, the next version of Visual Studio should also just install the IDE in the Program Files folder, and the Visual C#, C, VB, etc compilers in your app data folder also. Where does it stop? Is it still a good idea? I'll leave that for you to decide :) And that's before considering that roaming profiles are synced - either over a network or via the intenet...
Pete
Roger Clark
May 13, 2015, 9:38:34 PM5/13/15
to devel...@arduino.cc
Tim,
Take a look in your %appdata% folder on windows, and see how many other exe's or DLL's etc are store in there.My machine is extensively use for commercial dev work, so I have a number of commerical IDE's installed,
I just searched and the only exe's were the Arduino ones.>
>
Tim Leek
May 13, 2015, 10:11:31 PM5/13/15
to devel...@arduino.cc
Bill,
All good points, however I see 'Downloads' and my 'Onedrive' below the user folder which can contain .exe's.
I've thought some more. Some of the tools are 400MB+ and multiple users of the same machine, for example each student in a class room, would need to download packages. Alternatively, central admin installs would be difficult causing a huge file copy process when students login.
So I fully agree that there needs to be some flexibility on the location. Due to the size of some tool chains there does need to be a "shared" location and I can see that for people like yourself Ide specific tools would be workable.
Maybe a "file>preferences" path field? If a relative path is entered that gives more advanced users the ability to have local Ide tool chains. If a full path is entered then all users share the tools at the specified location, otherwise stick with it as it is retaining the future ability to drop older tools by changing arduino15 to something else.
Tim
Tim Leek
Visual Micro Limited
> Date: Wed, 13 May 2015 20:16:26 -0500
> From: bper...@gmail.com
> To: devel...@arduino.cc
> Subject: Re: [Developers] Why is IDE using puting executables under users .arduino15 rather than in sketchbook or better in the IDE tree?
> From: bper...@gmail.com
> To: devel...@arduino.cc
> Subject: Re: [Developers] Why is IDE using puting executables under users .arduino15 rather than in sketchbook or better in the IDE tree?
> --
> You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/OEFyNG_r10I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.
> You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/OEFyNG_r10I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.
Tim Leek
May 13, 2015, 10:16:56 PM5/13/15
to devel...@arduino.cc
I don't disagree entirely but I certainly agree with moving out of the Roaming folder, Local might be better as a default shared location (see my prev msg).
Tim
Date: Thu, 14 May 2015 11:38:31 +1000
Date: Thu, 14 May 2015 11:38:31 +1000
Subject: Re: [Developers] Why is IDE using puting executables under users .arduino15 rather than in sketchbook or better in the IDE tree?
You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/OEFyNG_r10I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.
To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/OEFyNG_r10I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.
Tim Leek
May 13, 2015, 10:27:18 PM5/13/15
to devel...@arduino.cc
On second thoughts it would be more efficient for administrators if the default was Ide specific tool chains. It avoids having to clone a preferences.txt into each users arduino15 folder just to set the shared location. So you are right, it might as well be Ide specific and would make Ide testing a great deal easier.
Tim
Andrew Kroll
May 17, 2015, 12:33:13 PM5/17/15
to devel...@arduino.cc
NetBeans stores extra stuff (executables, jars, etc) in a special place in the user's home directory too.
This is done so that a user does not need root permissions (I'm guessing there is some equivalent on mac and windows) to install anything extra.
That said, I don't have an issue with it.This is done so that a user does not need root permissions (I'm guessing there is some equivalent on mac and windows) to install anything extra.
--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.
--
Reply all
Reply to author
Forward
0 new messages