Building Arduino without the toolchain

[Matthijs Kooijman]

Hi folks,

I'm working with th Arduino software from git now, to do IDE
development. However, I've found that when I build the IDE (using ant
in the build directory), it does not only build the ide, but also
unpacks the embedded avr toolchain and downloads, unpacks an arm
toolchain and copies some other tools (avrdude, bossacs, adk2).

However, I want to my system's toolchain instead of the
Arduino bundled one (both since it's newer and because I have local
patches). Right now, I just deleted all the relevant lines from
build.xml, preventing the toolchain to be unpacked. This causes the IDE
to default to the system toolchain, which is what I want. However,
modifying build.xml isn't very pretty...

It seems I can partly get away with this by calling the "linux-build"
target instead of the default "build" target (which calls
"linux64-build"). AFAICS this would prevent unpacking the actual
toolchains, but still copies avrdude and friends.

Is there some official way to disable building the toolchain? If not,
would there be interest in having one? Perhaps separating the toolchain
stuff into separate targets? Or perhaps having some build system option
(not sure how that works in ant yet, though).


Gr

Matthijs

[Matthijs Kooijman]

Hey folks,

one related observation: If you don't unpack the avr toolchain, Arduino
will fallback to the system-wide compiler already, but not for avrdude.

For the compiler, this is due to the following code:

static public String getAvrBasePath() {
String path = getHardwarePath() + File.separator + "tools" +
File.separator + "avr" + File.separator + "bin" + File.separator;
if (Base.isLinux() && !(new File(path)).exists()) {
return ""; // use distribution provided avr tools if bundled tools missing
}
return path;
}

Which is used to set the value of "compiler.path" which is used in
platform.txt:

recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.c.flags} [more arguments]

However, no such feature exists for avrdude, since platform.txt
contains:

tools.avrdude.cmd.path={runtime.ide.path}/hardware/tools/avr/bin/avrdude

So that always looks inside the Arduino bundled stuff.

However, looking more closely, I see the command is really specified
twice:

tools.avrdude.cmd.path={runtime.ide.path}/hardware/tools/avr/bin/avrdude
tools.avrdude.config.path={runtime.ide.path}/hardware/tools/avr/etc/avrdude.conf
tools.avrdude.cmd.path.linux={runtime.ide.path}/hardware/tools/avrdude
tools.avrdude.config.path.linux={runtime.ide.path}/hardware/tools/avrdude.conf

Does this mean only the latter is used and the former two lines are
ignored, or is this a feature where Arduino tries both commands and uses
the one that matches?


Note that the 1.x series had some code in AvrdudeUploader.java, that
took care of falling back to the system-wide avrdude:

if(Base.isLinux()) {
if ((new File(Base.getHardwarePath() + "/tools/" + "avrdude")).exists()) {
commandDownloader.add(Base.getHardwarePath() + "/tools/" + "avrdude");
commandDownloader.add("-C" + Base.getHardwarePath() + "/tools/avrdude.conf");
} else {
commandDownloader.add("avrdude");
}
}

Gr.

Matthijs

[Matthijs Kooijman]

Hey folks,

any comments on my below mail?

I'm happy to work on this, if we can decide on a way to properly fix
this.

For the second part, to use the system-supplied toolchain if not local
one is present, I'd propose to:
- Unpack the toolchains in a more consistent location. Now avr is in
build/linux/work/hardware/tools/avr and sam is in
build/linux/work/hardware/tools/g++_arm_none_eabi, let's make the
latter build/linux/work/hardware/tools/${arch} instead.
- Put the relevant tools bin directory
(build/linux/work/hardware/tools/${arch}/bin) in the system PATH
during compilation.
- Modify the recipes in platform.txt to just use the plain commandnames
instead of full paths to the commands.

This approach changes the compiler, avrdude and other tool handlings,
but makes them more uniform and also simpler: It lets the system shell
handle the falling back to system binaries instead of hardcoding this in
the IDE (which would need it coded for every command called).

Note that I'm using linux examples here, but I plan to fix things for
macosx and windows as well (though I'd need a bit of help from others
with testing, probably).


As for building without the toolchain, I'm not sure what the best
approach is here. It seems useful to provide an alternative build target
(like "ide-build") that just builds the platform-indpependent stuff and
have the linux{,32,64}-build targets depend on that and add the
toolchain stuff, but then you might need another -run target to be able
to run without installing a toolchain as well, which might get a bit
verbose.

Also, I've found that there are multiple toolchain-like things:
- The actual compilers (avr-gcc and arm-gcc)
- Uploader tools (avrdude, bossac and adk2tool)
- Native libraries (rxtx, libastylej)

If you would want to control installing these different things
separately (or perhaps throw in the first two together, but the third
one certainly looks distinct to me), the number of targets you'd need
for controlling this would quickly increase.

An alternative would be to use ant properties for this, e.g., you
specify:

ant build -Dinstall-toolchain=false -Dinstall-native-libraries=false

Perhaps also a -Dinstall-java-libraries=false to prevent installing the
jars?

Not sure how to implement this in the actual build.xml file yet, I guess
you can just make parts of it conditional and/or use conditional
dependencies (haven't looked closely yet).

I found some further inconsistencies that may be worth fixing at the
same time:
- The avr toolchain tarball is included in git, while the arm toolchain
is downloaded at build time. I guess this is because the arm
toolchain is just a regular gcc tarball, while the avr one has been
post-processed to contain the right pathnames etc.? Or perhaps the
difference is just historical?
- Some files are downloaded from downloads.arduino.cc and some from
arduino.googlecode.com.
- The rxtx (native) library is contained in git, while the libastylej
(native) library is downloaded. Also, various jars are contained in
git as well.
- git contains a jre.tgz, even though it is (fortunately) not used.
- Most toolchain bits are in build/linux/dist, except the avr toolchain
and libastylej.
- linux32 and linux64 are different platforms, but share the
build/linux directory. Why not use build/linux32 and build/linux64
instead?
- The windows-dist target builds an "expert" version, which excludes
the "java" subdirectory, but I couldn't find what files are supposed
to be in there?

Gr.

Matthijs

[William "Chops" Westfield]

On Nov 24, 2013, at 2:34 AM, Matthijs Kooijman wrote:

>> However, I want to my system's toolchain instead of the
>> Arduino bundled one (both since it's newer and because I have local
>> patches).

:

>> Is there some official way to disable building the toolchain?

Why should the official build support the creation of an unsupported configuration?
The unix versions of Arduino used to be dependent on a system avr toolchain, and it caused "many" problems due to broken packaging, broken compiler versions, and compiler/avr-libc version incompatibilities…

WestfW

[Matthijs Kooijman]

Hey William,

> >> Is there some official way to disable building the toolchain?
>
> Why should the official build support the creation of an unsupported configuration?

Because:
- Users like myself might want the system toolchain.
- Distributions like Debian have a policy to never distribute the same
software twice, so they must use the system toolchain (they already
do this, but they need to patch the Arduino source now AFAIK).
- Third-party hardware might need newer toolchain versions or a patched
toolchain.

> The unix versions of Arduino used to be dependent on a system avr
> toolchain, and it caused "many" problems due to broken packaging,
> broken compiler versions, and compiler/avr-libc version
> incompatibilities…

I'm certainly not suggesting to make this the default! I entirely agree
that for most users, having an embedded toolchain is the easiest to
install and support. However, I'm just arguing that it should be easier
for users (and distribution packagers) to use the system toolchain
instead of the embedded one (when building the IDE from source).

I might even go as far as suggesting to offer an alternative
downloadable version that does not include the toolchain, but I can
imagine that causing problems for people who download the wrong version.
Also, with my proposed changes, you could just delete the
hardware/tools/avr directory after installation and the IDE would
automatically fall back to the system toolchain, so a toolchain-less
download wouldn't be really needed.

Does this clarify my intent and address your concerns?

Gr.

Matthijs

[Paul Stoffregen]

> I might even go as far as suggesting to offer an alternative
> downloadable version that does not include the toolchain, but I can
> imagine that causing problems for people who download the wrong version.

Perhaps you weren't involved a few years ago, back when the Linux
version didn't include a known-good toolchain.

It was a TECH SUPPORT NIGHTMARE. Especially Gentoo's broken avr
toolchain packages caused a lot of grief. Fedora and Arch also shipped
broken toolchains at various times. Perfectly fine, but newer and
incompatible toolchains on Ubuntu also started to become a problem,
though nothing like the scale of Gentoo's trouble.

> Also, with my proposed changes, you could just delete the
> hardware/tools/avr directory after installation and the IDE would
> automatically fall back to the system toolchain,

That's the way I wrote the patch that added the Linux toolchain. Perhaps
this ability was lost in the 1.0 -> 1.5 transition?

I know on 1.0.X, all you need to do is "rm -r hardware/tools/avr", and
then the IDE will use the system toolchain. Debian and the other
distros seem to have no problems preparing their packages with this
approach.

> so a toolchain-less download wouldn't be really needed. Does this
> clarify my intent and address your concerns? Gr. Matthijs

I'm not concerned about loading up build.xml will all sorts of optional
stuff. Maybe Cristian might be, but it does not concern me.

Just please don't return us to the dark days when the Linux download
didn't closely match Mac and Windows. Those days, thankfully, are in
the distant past... so long ago that people may have forgotten or simply
not have used Arduino that long. But let me tell you from personal
experience, as someone who regularly helps users on forums, by email,
and sometimes via phone and in person, supporting Arduino on Linux was
indeed terrible back then.

[Matthijs Kooijman]

Hey Paul,

> >I might even go as far as suggesting to offer an alternative
> >downloadable version that does not include the toolchain, but I
> >can imagine that causing problems for people who download the
> >wrong version.
>
> Perhaps you weren't involved a few years ago, back when the Linux
> version didn't include a known-good toolchain.
>
> It was a TECH SUPPORT NIGHTMARE. Especially Gentoo's broken avr
> toolchain packages caused a lot of grief. Fedora and Arch also
> shipped broken toolchains at various times. Perfectly fine, but
> newer and incompatible toolchains on Ubuntu also started to become a
> problem, though nothing like the scale of Gentoo's trouble.

Let me emphasize again that I'm not proposing to stop offering the
embedded toolchain version, since I can imagine the grief caused by
different toolchain versions as well as poorly supported toolchains by
distributions.
j

> >Also, with my proposed changes, you could just delete the
> >hardware/tools/avr directory after installation and the IDE would
> >automatically fall back to the system toolchain,
>
> That's the way I wrote the patch that added the Linux toolchain.
> Perhaps this ability was lost in the 1.0 -> 1.5 transition?
>
> I know on 1.0.X, all you need to do is "rm -r hardware/tools/avr",
> and then the IDE will use the system toolchain. Debian and the
> other distros seem to have no problems preparing their packages with
> this approach.

As stated in my original mail, this still works for the compiler, but
the avrdude call does not fall back to the system path, because the
avrdude path is now semi-hardcoded in platform.txt:

https://github.com/arduino/Arduino/blob/ide-1.5.x/hardware/arduino/avr/platform.txt#L68

Gr.

Matthijs

[Matthew Ford]

Hi All,
I have a library of parsers for pfodApp (www.pfod.com.au)
I have updated one of the examples.
but when I test out the installation using 1.0.5 Import Library -> Add
Library

It fails with a library named pfod_ArduinoLibraries already exists.

How is the user supposed to update libraries using the IDE?

matthew

[Rob Tillaart]

I would try:

- Stop the IDE, 

- move the existing library to some backup place, 

- put the new version in place, 

- start the IDE 

please give it  a try 

Note this kind of questions you can perfectly ask on the forum! So if the above recipe doesn't work please ask on the forum,

(This mailing list is mostly for development of the core (and yes what you told sounds like a bug)

matthew

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+unsubscribe@arduino.cc.

[Matthew Ford]

Also just noticed that the original import did not work
The zip file has 3 libraries in it,
The IDE created a folder the same name as the zip file and extracted the
library files to it.
This does not work

So I would suggest that
i) the IDE be modified to allow libraries to be updated
ii) if the zip file just has a folder or folders at the top level then
these be extracted to the library dir

In the mean time I will have to advise my users not to use this Add
Library feature of the IDE
matthew

[Matthew Ford]

Thanks Rob,
of course I can do that,

But it is not something I would expect to have to explain to my users.

The IDE advertises an Add Library function to make it easy to add libraries.
Not much use if you have to then explain to the user why it does not
work (for updates)
much better to either fix the IDE or remove the function and educate the
user first off

Arduino is all about the user experience, not just being able to do it.

matthew

>> email to developers+...@arduino.cc.
>>

[Rob Tillaart]

> Arduino is all about the user experience, not just being able to do it.

Agree

email to developers+unsubscribe@arduino.cc.

[William "Chops" Westfield]

> I'm certainly not suggesting to make this the default!

But you think that SOMETIMES it would make sense for people who are doing work on the source of the IDE, to be using a different toolchain than the normal users? I can't see it…

In 1.5.x, I would think that it ought to be possible to have an additional board/platform that used system tools rather than bundled tools. Something that could happen at the user config-level rather than during the build, no build changes required. Would that address your needs?

WestfW

[Matthijs Kooijman]

Hey William,

> > I'm certainly not suggesting to make this the default!
>
> But you think that SOMETIMES it would make sense for people who are
> doing work on the source of the IDE, to be using a different toolchain
> than the normal users? I can't see it…

Or people who need newer compilers for specific hardware, or who need
C++11 features (but still want to use the Arduino framework), etc.

> In 1.5.x, I would think that it ought to be possible to have an
> additional board/platform that used system tools rather than bundled
> tools. Something that could happen at the user config-level rather
> than during the build, no build changes required. Would that address
> your needs?

You mean to change the compilation recipes in platform.txt to remove the
full compiler paths? For my third-party hardware platform needs, that
seems like a good suggestion indeed, thanks. As an added bonus we can do
that and have things work with currently released 1.5 versions, AFAICS.

However, the original point still stands: I'd like to make it easier for
people to work with an external toolchain. For example, I have a library
that needs C++11 to work, so even for some or my normal Arduino projects
I can't use the embedded toolchain.

I'm not sure where you're heading with your arguments. Do you think it
is a bad idea to make the build system more flexible? Is there some kind
of cost or downside to this (apart from the coding work, which I'm
offering to do) that needs a significant usecase to be worth the
trouble?


If I can't convince you that making the build system more flexible is a
good idea, I could also argue that the 1.5 code now contains a
regression and inconsistency: The 1.0 code falls back to the system
toolchain already, and the 1.5 code contains explicit code to do so (on
Linux only IIRC) for the compiler (but not for avrdude). So apparently
in the past, it has already been decided that falling back to the system
toolchain is a good idea?

Gr.

Matthijs

[Cristian Maglie]

Hi Matthijs,

In data domenica 24 novembre 2013 11:34:18, Matthijs Kooijman ha scritto:
> For the second part, to use the system-supplied toolchain if not local
> one is present, I'd propose to:
> - Unpack the toolchains in a more consistent location. Now avr is in
> build/linux/work/hardware/tools/avr and sam is in
> build/linux/work/hardware/tools/g++_arm_none_eabi, let's make the
> latter build/linux/work/hardware/tools/${arch} instead.
> - Put the relevant tools bin directory
> (build/linux/work/hardware/tools/${arch}/bin) in the system PATH
> during compilation.
> - Modify the recipes in platform.txt to just use the plain commandnames
> instead of full paths to the commands.

I'm not a big fan of mixing together various tools inside the same bin/
folder, linux distribution do it when you install packages but packages are
maintained for consistency. I'd prefer to rename "tools/avr" /
"tools/g++_arm_none_eabi" to a more specific name like "avr-gcc-4.3.4" / "arm-
gcc-none-eabi-4.4.1", put avrdude into a specific folder "avrdude-5.11" and so
on. But this is not on the top priorities now.

Moreover adding to the environment PATH, as you suggest, implies also work for
developers of 3rd party plugins based on platform.txt, that I will likely
avoid.

Instead, what about to reproduce Paul's check but on every tool not only
avrgcc and avrdude? Just to be clear I'm thinking about an exec() method
extension something like:

void execOrFallbackToSystemPath(cmd) {
if (cmd.exists()) {
// go with the full path
exec(cmd);
} else {
// exec without path, let the s.o. find the right one
exec(cmd.baseName());
}
}

This way we can remove getAvrBasePath() method and set compiler.path variable
in avr/platform.txt accordingly (removing another "hidden" logic inside the
IDE).

> As for building without the toolchain, I'm not sure what the best
> approach is here.

[...]

> Also, I've found that there are multiple toolchain-like things:
> - The actual compilers (avr-gcc and arm-gcc)
> - Uploader tools (avrdude, bossac and adk2tool)
> - Native libraries (rxtx, libastylej)

I want to back what the others said: Arduino will never distribute a core
without the offical supported toolchain, so its not in the interest of Arduino
making it conditional in the build.

Said that, if you really want to go this path, I don't see any concerns in
improving the build process. I don't have any strong suggestion, probably is
better if you start with a small piece, for example you can try to make the
install AVR toolchain conditional, and put a PR with only that to see how it
looks like.

For the compilers/tools my little experience suggests that the less painful
way is to use ant properties, but (unexpectedly) this requires a massive
refactoring of build.xml because of the way conditionals are implemented in
ant (or maybe I used ant internals in the wrong way...).

About the native libraries its probably better to use maven or ivy, but this
is another big topic.

> I found some further inconsistencies that may be worth fixing at the
> same time:
> - The avr toolchain tarball is included in git, while the arm toolchain
> is downloaded at build time. I guess this is because the arm
> toolchain is just a regular gcc tarball, while the avr one has been
> post-processed to contain the right pathnames etc.? Or perhaps the
> difference is just historical?

Most of the build.xml content is historical and was made before I even know
about Arduino at all. Probably when the repository was under SVN storing
binary files in it wasn't a big concern as it is with git.
AFAIK archives are preprocessed to have the right root folder.

> - Some files are downloaded from downloads.arduino.cc and some from
> arduino.googlecode.com.

that was before googlecode decided to close "downloads". Eventually all the
download will be moved to the Arduino repository.

> - The rxtx (native) library is contained in git, while the libastylej
> (native) library is downloaded. Also, various jars are contained in
> git as well.

also this one was a recent addition.

> - git contains a jre.tgz, even though it is (fortunately) not used.

> - The windows-dist target builds an "expert" version, which excludes
> the "java" subdirectory, but I couldn't find what files are supposed
> to be in there?

it is used on windows build to provide a JRE. In the latest ide-1.5.x the
archive is downloaded from Arduino repository (after an upgrade of the java
runtime it was moved there).

> - Most toolchain bits are in build/linux/dist, except the avr toolchain
> and libastylej.

another recent change.

> - linux32 and linux64 are different platforms, but share the
> build/linux directory. Why not use build/linux32 and build/linux64
> instead?

I don't know the reason, probably at the time, linux64 was very similiar to
linux32, so it seemed overkill to duplicate everything.

C

[Matthew Ford]

Opened issue for this https://github.com/arduino/Arduino/issues/1704

Once a library is installed the IDE V1.0.5 does not allow it to be
installed again.
Request removing this check

It has been suggested a dialog be opened to alert user the library
exists, but I am not sure what this achieves. With no concept of library
version nos what is the user expected to do other then always say overwrite.

Also If touching this function, request that the link between zip name
and library name be overridable.

For example if the zip has just a top level directory (or directories)
then these should be installed as the libraries. This allows the zip
name to include a version number so users can tell which version of the
library they are installing.

It also allows multiple libraries to be installed from one zip file.
.eg.
pfodParserLib_V2.zip
contains just
-> pfodParser (dir)
then install pfodParser (dir) under libraries dir

pfodParserLib_V2.zip
contains just
-> pfodParser (dir)
-> pfodCmdParser (dir)
then install pfodParser (dir) and pfodCmdParser (dir) under libraries dir.
i.e. installs two libraries.

For backward compatibility, if the top level of the zip file has any
non-directory entries then use existing install process which uses the
name of the zip as the library name.


On 28/11/2013 1:30 AM, Cristian Maglie wrote:
> IMHO This is a good suggestion. Probably is better to put a dialog asking
> confirmation before upgrading (overwriting) the library. May you open an issue
> on github to keep track of this?
>
> C

[Tom Igoe]

I’m having trouble making sense of this, exactly. Perhaps you could submit a patch that folks could look at to understand and evaluate the proposal?

t.

[Peter Feerick]

Tom,

I'm not up to writing any patches, but how I see it is this.

Current behaviour of the IDE when you use the 'add libary' command is to
extract the contents of the zip file to a folder with the name as the
zip file. This causes issues if the name of the zip file is not a valid
library name (eg. contains spaces or hyphens, -master, for example).

I would suggest that better behaviour would be
a) if there is a single top level folder in the zip, extract directly to
libraries folder -sanatize to remove '-master' if present - deals with
github zip downloads.
b) if there are multiple folders (perhaps with example subfolders? how
best to identify multi-library condition?), extract directly to
libraries folder
c) if there are files and top level 'examples' folder, extract to folder
same name as zip to libraries folder, whilst sanitizing the folder name
by replacing '-' with '_' and removing '-master'.

I would like to see a simple warning if there is a library with the same
name present, only if it is that - 'A library with the same name ('xyz')
is already installed . Do you wish to overwrite it? [Yes] [No] ' Is just
a simple sanity check so you don't install a conflicting library by
accident.

Cheers
Peter

Hope that helps

[Matthijs Kooijman]

Hi Christian,

thanks for your in-depth response.

> I'm not a big fan of mixing together various tools inside the same bin/
> folder, linux distribution do it when you install packages but packages are
> maintained for consistency. I'd prefer to rename "tools/avr" /
> "tools/g++_arm_none_eabi" to a more specific name like "avr-gcc-4.3.4" / "arm-
> gcc-none-eabi-4.4.1", put avrdude into a specific folder "avrdude-5.11" and so
> on. But this is not on the top priorities now.

Ok, that makes sense. However, wouldn't including the version number in
the folder make it harder for third-party platform.txt files to refer to
the right directory? Or were you thinking about using wildcards or
exposing some new magic variables to find these version numbers?

> Moreover adding to the environment PATH, as you suggest, implies also work for
> developers of 3rd party plugins based on platform.txt, that I will likely
> avoid.

Ah, good point.

> Instead, what about to reproduce Paul's check but on every tool not only
> avrgcc and avrdude? Just to be clear I'm thinking about an exec() method
> extension something like:
>
> void execOrFallbackToSystemPath(cmd) {
> if (cmd.exists()) {
> // go with the full path
> exec(cmd);
> } else {
> // exec without path, let the s.o. find the right one
> exec(cmd.baseName());
> }
> }
>
> This way we can remove getAvrBasePath() method and set compiler.path variable
> in avr/platform.txt accordingly (removing another "hidden" logic inside the
> IDE).

I considered this option, but didn't like reproducing what's essentially
the same thing the shell is already doing with $PATH. However, your
point about 3rd party tools makes sense, so this seems like the right
option to me now.

> > As for building without the toolchain, I'm not sure what the best
> > approach is here.
> [...]
> > Also, I've found that there are multiple toolchain-like things:
> > - The actual compilers (avr-gcc and arm-gcc)
> > - Uploader tools (avrdude, bossac and adk2tool)
> > - Native libraries (rxtx, libastylej)
>
> I want to back what the others said: Arduino will never distribute a core
> without the offical supported toolchain, so its not in the interest of Arduino
> making it conditional in the build.

Understood.

> Said that, if you really want to go this path, I don't see any concerns in
> improving the build process. I don't have any strong suggestion, probably is
> better if you start with a small piece, for example you can try to make the
> install AVR toolchain conditional, and put a PR with only that to see how it
> looks like.
>
> For the compilers/tools my little experience suggests that the less painful
> way is to use ant properties, but (unexpectedly) this requires a massive
> refactoring of build.xml because of the way conditionals are implemented in
> ant (or maybe I used ant internals in the wrong way...).

Thanks, I'll read up on ant and see if I can prepare something.

> About the native libraries its probably better to use maven or ivy,
> but this is another big topic.

That would require replacing the current ant setup altogether, right? I
guess that's not something for now, though perhaps making things a bit
more consistent could be done now.

> > I found some further inconsistencies that may be worth fixing at the
> > same time:
> > - The avr toolchain tarball is included in git, while the arm toolchain
> > is downloaded at build time. I guess this is because the arm
> > toolchain is just a regular gcc tarball, while the avr one has been
> > post-processed to contain the right pathnames etc.? Or perhaps the
> > difference is just historical?
>
> Most of the build.xml content is historical and was made before I even know
> about Arduino at all. Probably when the repository was under SVN storing
> binary files in it wasn't a big concern as it is with git.

So that suggests that keepin tarballs out of git is the preferred way of
doing things now?

> AFAIK archives are preprocessed to have the right root folder.

Right. To me, it seems more elegant to just use the original tarballs
published by upstream and do all the necessary pathname fidgeting during
the ant build process. That makes it more clear what changes are made to
the tarball and makes it easier to upgrade later on. Not sure if all the
tarballs are from some upstream, though, if they're custom builds for
Arduino then this point is a bit moot.

> > - Some files are downloaded from downloads.arduino.cc and some from
> > arduino.googlecode.com.
>
> that was before googlecode decided to close "downloads". Eventually all the
> download will be moved to the Arduino repository.

You mean stored inside git? Or downloaded from arduino.cc?

> > - The rxtx (native) library is contained in git, while the libastylej
> > (native) library is downloaded. Also, various jars are contained in
> > git as well.
>
> also this one was a recent addition.

Ok, so again, doing external downloads is the "new" and preferred way.

>
> > - git contains a jre.tgz, even though it is (fortunately) not used.
> > - The windows-dist target builds an "expert" version, which excludes
> > the "java" subdirectory, but I couldn't find what files are supposed
> > to be in there?
>
> it is used on windows build to provide a JRE. In the latest ide-1.5.x the
> archive is downloaded from Arduino repository (after an upgrade of the java
> runtime it was moved there).

Right.

>
> > - Most toolchain bits are in build/linux/dist, except the avr toolchain
> > and libastylej.
>
> another recent change.

In this case, I think that inside dist/ would be the preferred location,
right? So libastylej, even though it's a recent addition, is in the
wrong place? Or actually, libastylej is not even in the repo, it's just
downloaded and temporarily unpacked to build/linux, which probably isn't
very elegant. Perhaps something like build/linux/work/temp would be
better for that?

> > - linux32 and linux64 are different platforms, but share the
> > build/linux directory. Why not use build/linux32 and build/linux64
> > instead?
>
> I don't know the reason, probably at the time, linux64 was very similiar to
> linux32, so it seemed overkill to duplicate everything.

No need to duplicate everything, they can still have the same generial
dependencies. I might see what happens if you split these, it might make
the build.xml a bit more clear.

Gr.

Matthijs

[Matthijs Kooijman]

Hey Christian,

> > Instead, what about to reproduce Paul's check but on every tool not only
> > avrgcc and avrdude? Just to be clear I'm thinking about an exec() method
> > extension something like:
> >
> > void execOrFallbackToSystemPath(cmd) {
> > if (cmd.exists()) {
> > // go with the full path
> > exec(cmd);
> > } else {
> > // exec without path, let the s.o. find the right one
> > exec(cmd.baseName());
> > }
> > }
> >
> > This way we can remove getAvrBasePath() method and set compiler.path variable
> > in avr/platform.txt accordingly (removing another "hidden" logic inside the
> > IDE).
>
> I considered this option, but didn't like reproducing what's essentially
> the same thing the shell is already doing with $PATH. However, your
> point about 3rd party tools makes sense, so this seems like the right
> option to me now.

I implemented the above, see my pullrequest:

https://github.com/arduino/Arduino/pull/1722

I ended up doing only the minimal changes needed to make installing the
toolchain and native libraries optional. I wanted to split the tools
(avrdude etc.) from the toolchain (gcc, binutils), but on Windows and
MacOS X these are shipped in the same tarball, so I ended up just
controlling all with the "install_toolchain" property (the seperation
was mostly "because I can", I don't have a specific need for it now).

Furthermore, I also added a "link_hardware" property, which causes
$platform/work/hardware/arduino to become a symlink to the original
files, so you can change them without having to re-run ant after every
change to test them.

So far, I've only done some minimal testing on Linux, if we like the way
the code is heading, we should do some more testing on Windows and MacOS
X as well, of course.


Finally, I ran into a new problem: The fallback to the system path now
works for the avrude binary, but platform.txt also passes the -C option
and the path to the bundled avrdude.conf. If using the system-wide
avrdude, it should also use the system-wide config file (i.e., don't
pass a -C option).

The current recipe system can't seem to express this in any useful
way and I'm not sure how to allow this in a somewhat generic manner. I
could imagine adding a special replacement syntax for recipes, like:

tools.avrdude.configopt="{{ exists(config.path) ? "-C{config.path}" : }}
tools.avrdude.upload.pattern="{cmd.path}" {configopt} {upload.verbose} ...

E.g., allow a ternary if syntax and some kind of "exists" function
within {{ }}. This does sound terribly over-engineered and too complex
to be useful, though...

An alternative I could imagine, is to specify multiple different
possible commands and let the IDE use the first one whose executable
exists and the last one if none exist? E.g.,

tools.avrdude.upload.pattern="{cmd.path}" "-C{config.path}" {upload.verbose} ...
tools.avrdude.upload.pattern=avrdude {upload.verbose} ...

It's less flexible, but it would save this particular problem at
least...

Any other suggestions?

Gr.

Matthijs

[Matthijs Kooijman]

Hey folks,

I'm dragging up this thread again, since we still haven't found a
satisfactory solution yet. For reference, code for my earlier proposal
is here: https://github.com/arduino/Arduino/pull/1722

This issue consists of to parts: The actual toolchain and the hardware
folder (which, thinking on it more, are really two separate issues that
happened to be discussed in this thread).

I'll start with the easy one, the hardware folder.

- build.xml copies the hardware/ folder into build/linux/work/
and then the IDE has a semi-hardcoded path to it, which it uses to
find platform.txt.

https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/Base.java#L2062

Additionally, this path is used to find the core to compile, and add
it to the include path.
https://github.com/arduino/Arduino/blob/ide-1.5.x/hardware/arduino/avr/platform.txt#L38
https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/debug/Compiler.java#L542
https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/debug/Compiler.java#L93
https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/debug/Compiler.java#L210

- I want to use the arduino core in the git repo at hardware/ instead
of the copy in build/linux/work/ (so I don't have to rerun ant on
every change to the core).

- I changed build.xml to create a build/linux/work/hardare/arduino
symlink instead of copying the hardware folder. This took a bit of
fiddling to get ant to play nice with symlinks, but seems to work as
expected, except that it doesn't work for architecture-independent
libraries yet (but that should be a matter of making
build/linux/work/libraries a symlink too).


Now, the other part is the toolchain.

- build.xml unpacks a toolchain into build/linux/work and then
platform.txt points the IDE to use it (indirectly, through
compiler.path).

https://github.com/arduino/Arduino/blob/ide-1.5.x/hardware/arduino/avr/platform.txt#L38
https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/debug/Compiler.java#L201
https://github.com/arduino/Arduino/blob/ide-1.5.x/app/src/processing/app/Base.java#L2079

And for avrdude, through runtime.ide.path:

https://github.com/arduino/Arduino/blob/ide-1.5.x/hardware/arduino/avr/platform.txt#L68

- I want to use my system toolchain instad of the bundled one.

- I changed build.xml to optionally not install the toolchain, which
worked.

- I changed the java source to remove the directory name from compile
and upload commands if the full command path did not exist.

This solution worked for gcc, but for avrdude it would run the
system-wide avruded, but point it at the (non-existing) avrdude.conf in
build/linux/work, which broke uploading.

I've spent quite some time thinking about a proper solution for this,
but pretty much everything I can come up with (multiple commands, use
the first one that exists, add "if statements" to platform.txt) are too
complicated to be useful for this fairlly narrow usecase.

An obvious solution for this problem is of course to just modify
platform.txt to remove any paths so the system path is used and add the
right path to the system-wide avrdude.conf. However, this will cause my
git repository to be always modified, which is annoying with rebases,
git commit -a, etc.

To fix that, I can imagine that the IDE would read platform preferences
from a file called platform.local.txt (or something like that) as well
(merging it with the regular platform.txt). This would allow users to
create a platform.local.txt to override the necessary commands and
variables to use the system-wide tools. I haven't checked the source in
detail, but this shouldn't be a very invasive change and would
completely solve the problem, I think.


Summarizing, I propose to:

- Allow making build/linux/work/hardware/arduino and
build/linux/work/libraries symlinks instead of copies. The hardware
bit is implemented already, but it should be extended for libraries
as well.

https://github.com/matthijskooijman/Arduino/commit/cfd8472c1ca73d0d6853153ba30460e4585be9b0

Note that this change is separate from the next one - it is even
useful if you want keep using the bundled compiler but don't want to
rerun ant whenever you change a core file.

- Instead of reading just "platform.txt", also read
"platform.local.txt", having the latter override values in the
former. No platform.local.txt file would be supplied in git or in
official distributions.


Does this sound acceptable?

Gr.

Matthijs

[Cristian Maglie]

Hi Matthijs!

In data giovedì 27 marzo 2014 23:25:07, Matthijs Kooijman ha scritto:
> - I want to use the arduino core in the git repo at hardware/ instead
> of the copy in build/linux/work/ (so I don't have to rerun ant on
> every change to the core).

Thinking on this problem a bit more: have you considered using the skethbook's
hardware folder? you can use the $SKETCHBOOK/hardware folder to link to your
working directory, for example having:

/your/git/repository/MyCore/avr/...

a symbolic link like the following should work:

$SKETCHBOOK/hardware/MyCore -> /your/git/repository/MyCore/

This solution will avoid adding complexity to the build.xml.

> Now, the other part is the toolchain.

[...cut...]

> I've spent quite some time thinking about a proper solution for this,
> but pretty much everything I can come up with (multiple commands, use
> the first one that exists, add "if statements" to platform.txt) are too
> complicated to be useful for this fairlly narrow usecase.

I neither find a proper solution for this case, and I like the workaround you
propose here:

> - Instead of reading just "platform.txt", also read
> "platform.local.txt", having the latter override values in the
> former. No platform.local.txt file would be supplied in git or in
> official distributions.

It seems a good compromise.

Probably the best solution would be to upgrade avrdude to the latest version?

C

[Matthijs Kooijman]

Hi Christian,

> In data giovedì 27 marzo 2014 23:25:07, Matthijs Kooijman ha scritto:
> > - I want to use the arduino core in the git repo at hardware/ instead
> > of the copy in build/linux/work/ (so I don't have to rerun ant on
> > every change to the core).
>
> Thinking on this problem a bit more: have you considered using the skethbook's
> hardware folder? you can use the $SKETCHBOOK/hardware folder to link to your
> working directory, for example having:
>
> /your/git/repository/MyCore/avr/...
>
> a symbolic link like the following should work:
>
> $SKETCHBOOK/hardware/MyCore -> /your/git/repository/MyCore/
>
> This solution will avoid adding complexity to the build.xml.

Good point!

Just tried makeing a symlink:

$SKETCHBOOK/hardware/arduino -> /path/to/Arduino.git/hardware/arduino

and then works and properly overrides the builtin arduino core.

However, for the (non-target-specific libraries), this doesn't work
unless I create a symlink under $SKETCHBOOK/libraries for _every_
library individually. I would like to just do:

$SKETCHBOOK/libraries/arduino -> /path/to/Arduino.git/libraries

but the IDE doesn't currently like that. Since I've been annoyed by the
lack of subdirectory support in the IDE before, I spent today
implementing this:

https://github.com/arduino/Arduino/pull/1986

You can now use subdirectories to organize the libraries inside the
sketchbook (also for the bundled libraries, if we'd want that).

> > - Instead of reading just "platform.txt", also read
> > "platform.local.txt", having the latter override values in the
> > former. No platform.local.txt file would be supplied in git or in
> > official distributions.
>
> It seems a good compromise.

Ok, I implemented this here:

https://github.com/arduino/Arduino/pull/1985

In addition, there's a commit in there which makes the avr
platform.txt define compiler.path explicitely, instead of relying on the
(avr-specific and ugly) default value defined by the IDE. See the commit
for some more rationale.

> Probably the best solution would be to upgrade avrdude to the latest version?

Hmm, what would this solve? It might remove my need to use the system
avrdude for now, but at some point this might change again and I'd
rather just solve things properly now. Also, with platform.local.txt,
this problem is already solved. Or did I misunderstand your meaning
here?

Gr.

Matthijs