Library conflicts

[Paul Stoffregen]

I've submitted a pull request to address the conflict that can occur
when 2 libraries have the same header.

For example, consider an installation where library "Bar" contains Foo.h
and Bar.h, and library "Foo" contains Foo.h and Bar.h.

When a user compiles:

#include <Bar.h>
void setup() { Bar.begin(); }
void loop() { Bar.print(); }

Obviously the user intends to include the Bar library. Today, the
Arduino IDE will attempt to use the Foo library, because it contains the
file Bar.h. Foo is (usually) found after Bar, causing the IDE to
overwrite its finding that Bar.h is in the Bar library.

This simple patch improves Arduino library list building, so a library
with matching directory is given preference.

https://github.com/arduino/Arduino/pull/1853

I realize many of the existing conflicts have workarounds, like moving
Adafruit_GFX.h to a subdirectory in Robot_Control. However, the issue
remains that any library can accidentally override usage of another,
simply by having a header file with the same name.

This patch should ensure Arduino always compiles the correct library
when the #include header in the user's sketch matches the library's
directory name. I believe all existing libraries follow this convention.

[Cristian Maglie]

In data lunedì 10 febbraio 2014 23:53:18, Paul Stoffregen ha scritto:
> I've submitted a pull request to address the conflict that can occur
> when 2 libraries have the same header.

This sound pretty reasonable to me. This patch define some behaviour that was
previously undefined so I'm for merging it for the next release (of both 1.0
and 1.5) if there are no concerns.

C

[Matthijs Kooijman]

Hey Paul,

> Obviously the user intends to include the Bar library. Today, the
> Arduino IDE will attempt to use the Foo library, because it contains
> the file Bar.h. Foo is (usually) found after Bar, causing the IDE
> to overwrite its finding that Bar.h is in the Bar library.
>
> This simple patch improves Arduino library list building, so a
> library with matching directory is given preference.

This looks reasonable to me. I'd still think that something more
advanced, that involves user interaction, would be the best way to fix
this, but that is non-trivial to implement. For now, this approach will
already make things better and it should not cause any problems with
implementing an interactive solution later on.

I'd like to see this one merged.

Thanks,

Matthijs

[Paul Stoffregen]

On 02/11/2014 02:46 AM, Matthijs Kooijman wrote:
> This looks reasonable to me. I'd still think that something more
> advanced, that involves user interaction, would be the best way to fix
> this, but that is non-trivial to implement.

Yup, I agree, it could be so much better. I resisted the urge to
implement more than the simplest fix.

Another simple improvement might be well-worded warning message when 2
libraries are found with the same .h file, but neither matches the
library directory name. But there are currently a large number of those
cases.

On the documentation side, it might be worthwhile to advise library
authors to place additional headers, which are not intended to be
included by the user's sketch, in the "utility" or other subdirectories.

[Bill Perry]

Paul,
What about the case where a user creates a local version/copy of the SDK/IDE library in his sketchbook/libraries
folder to create some local modifications?
And also suppose he re-names it something different than the original name.
Then what?

Under the current behavior, the users sketchbook/libraries library would be selected rather than
the one that shipped with the library.
I'm not sure what would happen with your patch.
i.e.
I create a library directory called "fmLiquidCrystal", under my sketchbook/libraries.
This is a library that is designed to override the LiquidCrystal library
that ships with the IDE. Inside that directory is a file called "LiquidCrystal.h"
Currently the user's local library will override the system library even though
the directory name for the library does not match the header file name.
But it sounds like this might nott work anymore.

The reason that I bring this up, is that I use the method for development & testing.
If you don't rename the local library directory when you put it into your local sketchbook/libraries
then things can get confusing as you will see the library in the list of libraries twice.

I know I'm not a typical Arduino user. But I'd ask that you consider this scenario.
In the case of local collisions, I agree that using the one with the matching directory name
should win, but I would think that a local library should still override a system library, even if the
local directory doesn't match.
So the directory match preference would only be done if there are multiple local libraries to pick from
and a local library would always be preferred over the system library even if the local library didn't
have a matching directory name.

On a related note,
one of things that I'd really like to see would be a change the include path.
Currently, the include path is based on the where the include files used in the sketch are found.
This is what is creating this very issue.
The thing I'd like to see would be to add two directories to the front of the include path:
- The path to the users sketchbook/libraries directory
- the path to the IDE libraries directory for the current board.

This would have  benefits not only for sketches but also for libraries.
If these two directories were added to the path, then sketches and libraries could specify
which library to use. i.e:
#include <Wire/Wire.h>

#include <fmLiquidCrystal/LiquidCrystal.h>

etc...

This allows the sketch or library specify with certainty which library & header file
they want and not have to worry about collisions.

--- bill

[Paul Stoffregen]

On 02/12/2014 10:45 PM, Bill Perry wrote:

Paul,
What about the case where a user creates a local version/copy of the SDK/IDE library in his sketchbook/libraries
folder to create some local modifications?
And also suppose he re-names it something different than the original name.
Then what?

Under the current behavior, the users sketchbook/libraries library would be selected rather than
the one that shipped with the library.
I'm not sure what would happen with your patch.
i.e.
I create a library directory called "fmLiquidCrystal", under my sketchbook/libraries.
This is a library that is designed to override the LiquidCrystal library
that ships with the IDE. Inside that directory is a file called "LiquidCrystal.h"
Currently the user's local library will override the system library even though
the directory name for the library does not match the header file name.
But it sounds like this might nott work anymore.

That's correct.  If you create a copy of the library and name its directory "fmLiquidCrystal", but the .h file inside that directory is "LiquidCrystal.h", then this patch will cause the original copy which is in a directory named "LiquidCrystal" to be used.

Of course, if you also change the header file to "fmLiquidCrystal.h", then your copy will be used.

The reason that I bring this up, is that I use the method for development & testing.
If you don't rename the local library directory when you put it into your local sketchbook/libraries
then things can get confusing as you will see the library in the list of libraries twice.

I know I'm not a typical Arduino user. But I'd ask that you consider this scenario.

Let's see if a clear consensus grows out of this discussion?

If so, I'll be happy to develop another patch that implements whatever algorithm is collectively agreed upon.  Or you, or Matthijs or anyone else could do it if you like.  That code which builds the hash of headers to libraries is pretty simple.

Just to be 100% clear, I do not plan to touch the code again unless there is clear consensus with Cristian's approval on a well specified algorithm.

In the case of local collisions, I agree that using the one with the matching directory name
should win, but I would think that a local library should still override a system library, even if the
local directory doesn't match.
So the directory match preference would only be done if there are multiple local libraries to pick from
and a local library would always be preferred over the system library even if the local library didn't
have a matching directory name.

On a related note,
one of things that I'd really like to see would be a change the include path.

I'm currently working on an audio library, so I can not get more involved right now.  I'm not saying the include path should or should not be changed, but only that I just can't take on more Arduino IDE work at this time.

Currently, the include path is based on the where the include files used in the sketch are found.
This is what is creating this very issue.
The thing I'd like to see would be to add two directories to the front of the include path:
- The path to the users sketchbook/libraries directory
- the path to the IDE libraries directory for the current board.

This would have  benefits not only for sketches but also for libraries.
If these two directories were added to the path, then sketches and libraries could specify
which library to use. i.e:
#include <Wire/Wire.h>

#include <fmLiquidCrystal/LiquidCrystal.h>

etc...

This allows the sketch or library specify with certainty which library & header file
they want and not have to worry about collisions.

--- bill

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Bill Perry]

On 02/13/2014 08:20 AM, Paul Stoffregen wrote:

On 02/12/2014 10:45 PM, Bill Perry wrote:

Paul,
What about the case where a user creates a local version/copy of the SDK/IDE library in his sketchbook/libraries
folder to create some local modifications?
And also suppose he re-names it something different than the original name.
Then what?

Under the current behavior, the users sketchbook/libraries library would be selected rather than
the one that shipped with the library.
I'm not sure what would happen with your patch.
i.e.
I create a library directory called "fmLiquidCrystal", under my sketchbook/libraries.
This is a library that is designed to override the LiquidCrystal library
that ships with the IDE. Inside that directory is a file called "LiquidCrystal.h"
Currently the user's local library will override the system library even though
the directory name for the library does not match the header file name.
But it sounds like this might nott work anymore.

That's correct.  If you create a copy of the library and name its directory "fmLiquidCrystal", but the .h file inside that directory is "LiquidCrystal.h", then this patch will cause the original copy which is in a directory named "LiquidCrystal" to be used.

Of course, if you also change the header file to "fmLiquidCrystal.h", then your copy will be used.

But what about if I changed the directory under my sketchbook/libraries to "LiquidCrystal"?
i.e.
Does a local library have precedence over the system library if both are the same name and have
a header file by the same name?
I would expect this to be the case.

This allows users to have a local library that can override the system library.

What does the current patch do?


--- bill

[Cristian Maglie]

In data domenica 16 febbraio 2014 02:05:53, Bill Perry ha scritto:
> But what about if I changed the directory under my sketchbook/libraries to
> "LiquidCrystal"? i.e.
> Does a local library have precedence over the system library if both are
> the same name and have a header file by the same name?
> I would expect this to be the case.

Using the IDE 1.0, in this case the library is showed twice (once within the
system libraries and once within the local libraries). From the example menu
you can open examples from both library, and this behaviour is consistent with
the filesystem status. When you try to import, even if you have two menu
entries in the "Import" menu only one library can be used, because both
libraries tries to include "LiquidCrystal.h". Now I agree with you that the
local library should have priority over system library, and Paul's patch
breaks this behaviour, I'm going to fix this.

C

[Paul Stoffregen]

I was also planning to submit another patch to fix this. Let me know if
you want me to look into it?


>
> C
>

[Cristian Maglie]

In data lunedì 17 febbraio 2014 14:05:23, Paul Stoffregen ha scritto:
> I was also planning to submit another patch to fix this. Let me know if
> you want me to look into it?

no worries it's a trivial fix I already did it, just review the patch:

https://github.com/arduino/Arduino/pull/1853
https://github.com/arduino/Arduino/commit/57bee97d7be07023bc94460cde54c96e4ef3de1d

thanks!

C

[Paul Stoffregen]

On 02/17/2014 05:24 AM, Cristian Maglie wrote:

> In data lunedě 17 febbraio 2014 14:05:23, Paul Stoffregen ha scritto:
>> I was also planning to submit another patch to fix this. Let me know if
>> you want me to look into it?
> no worries it's a trivial fix I already did it, just review the patch:

> https://github.com/arduino/Arduino/commit/57bee97d7be07023bc94460cde54c96e4ef3de1d

Yes, that looks reasonable. The new variable names make it more
readable too. :-)

How would you feel about adding 2 info/warning messages, each inside the
2 endsWith(name) checks?


>
> thanks!
>
> C
>

[Bill Perry]

I couldn't quite follow the patch.
Will this most recent patch make duplicate libraries go away in the various lists (sketches, import, etc...)?
I like the idea of a warning as Paul suggested.
Kind of like the warning that currently issues if you use a "period"/dot in the directory name of the library.
"Ignoring bad library name"
i.e. it issues a similar warning as it discovers and/or ignores the duplicate libraries.

The current patch may not handle that but it sounds very useful.
That way users would see any library conflict issues as the IDE starts up then either be
told which one is used/ignored and then potentially only seeing the actual libraries that will end up being used/available
in the various lists.

This would solve one of the things that keeps coming up in the forums.
Specifically multiple LiquidCrytal_I2C libraries installed, but there are others as well.


In the larger picture, I think that if a library is ever ignored, the user should
get some sort of warning.



--- bll

[Cristian Maglie]

Hey, sorry for the delayed answer...

In data lunedì 17 febbraio 2014 19:47:32, Bill Perry ha scritto:
> > How would you feel about adding 2 info/warning messages, each inside the
> > 2 endsWith(name) checks?

Something like:
"Using library /path/to/user/Arduino/libraries/LiquidCrystal instead of
/path/to/sketchkbook/libraries/MyLiquidCrystal (both provides
LiquidCrystal.h)"
?

The 1.5 IDE has a similar feature already, if you build the sketch in verbose
mode at the beginning it prints:

Using library SPI in folder:
/home/cmaglie/git/arduino/build/linux/work/hardware/arduino/avr/libraries/SPI
(legacy)
Using library Ethernet in folder:
/home/cmaglie/git/arduino/build/linux/work/libraries/Ethernet

Maybe backporting this messages in 1.0 may be a better replacement of the
warning?

> Will this most recent patch make duplicate libraries go away in the various
> lists (sketches, import, etc...)?

No it doesn't, this requires some more work on the 1.0.x IDE, while the 1.5.x
has a better libraries handling that removes duplicate libraries from menus.
I'd like to not push too much changes in the 1.0.x branch and reserve
structural changes for the 1.5.x only because that branch has the focus of
development now.

C

[Vasilis Georgitzikis]

So, what's the consensus here?

as far as i understand, things settled down to:
* if there is a library with the same name (folder name) as the .h file, the IDE uses that library
* otherwise it uses whatever it finds
* builtin libraries take precedence over custom ones

In the interest of conversation, we run into the same issue at codebender, which is even more prevalent because we support nearly all of the well-know Arduino libraries builtin, which is about 380 libraries. we decided to check headers with the same library name. this is the same idea that Paul started, only a bit more strict, basically without the second bullet above.

since my english is not my native language, i'm going to give an example to describe how it works:

if you have too libs:
foo:
* foo.h
* bar.h

foobar:
* foobar.h
* bar.h

if you want to use bar.h you'll be doing
#include <foo.h>
#include <bar.h>

or

#include <foobar.h>
#include <bar.h>

and just #include <bar.h> will not work.

this, of course, breaks compatibility with some libraries, but of the nearly 380 libraries we support, there are only 8 which are broken using this algorithm. on the other side, it makes things more verbose and also acts as a good rule of thumb for the library developers. in our experience, this also helps new users understand the basics of #include better (by asking them to #include the library name, and any extra header files you want from inside it), but I would like to know Tom's opinion on this

Vasilis

[David Cuartielles]

wouldn't it make more sense that custom libs override the built in ones?

just like in every other good old object oriented language ...

/d
________________________________________
De: Vasilis Georgitzikis <tzi...@gmail.com>
Enviado: lunes, 10 de marzo de 2014 17:46
Para: Cristian Maglie; Tom Igoe
Cc: Arduino Developers
Asunto: Re: [Developers] Library conflicts

[Vasilis Georgitzikis]

that would also be my suggestion, but people already argued for and against it, and i decided not to press on the matter

[David Cuartielles]

then forget about my opinion, I haven't been part of the discussion

/d
________________________________________
De: Vasilis Georgitzikis <tzi...@gmail.com>

Enviado: lunes, 10 de marzo de 2014 18:20
Para: David Cuartielles
Cc: Cristian Maglie; Tom Igoe; Arduino Developers

[Paul Stoffregen]

On 03/10/2014 10:20 AM, Vasilis Georgitzikis wrote:
> that would also be my suggestion, but people already argued for and against it, and i decided not to press on the matter
>
> On Mar 10, 2014, at 6:49 PM, David Cuartielles <david.cu...@mah.se> wrote:
>
>> wouldn't it make more sense that custom libs override the built in ones?

I believe Chistian's change (committed on Feb 17) does exactly what
you're suggesting, at least if the library name and folder name match.

My understanding of the logic currently implemented in on the master
branch (aka 1.0.6) by Base.rebuildImportMenu() and Base.addLibraries()
is this:

1: When a the header file matches the library folder name in sketchbook,
the sketchbook copy is used, even if the same header name exists elsewhere.

2: Otherwise, when the header file matches the library folder name in
libraries, the libraries copy is used, even if the same header name
exists elsewhere.

3: When the header name does NOT match the library folder name, the
"first" library found is used. The libraries folder is scanned first,
then the sketchbook folder is scanned, so in the case where a header
does not match the library name, the sketchbook copy does not have
priority over the libraries directory.

It's also possible for more than one library in either of those
locations could match. Which comes "first" is undefined. Oracle's
File.list() documentation says "There is no guarantee that the name
strings in the resulting array will appear in any specific order; they
are not, in particular, guaranteed to appear in alphabetical order."
However, when I tested on Linux, the filename list was in alphabetical
order. I did not do detailed testing on Mac or Windows.


As for the 1.5.x branch, I do not know. The logic is quite different
and I'm not as familiar with the code and the new LibraryList type. My
initial impression is patch isn't on the 1.5.x branch yet, but there's
more layers and more complexity in 1.5.x to unravel, so it's entirely
possible I just missed it. I haven't done any testing of 1.5.x.

[David Cuartielles]

aha ... so this basically means that overriding "the traditional way" works and then there are other methods to cope with other cases, am I right?

/d
________________________________________
De: Paul Stoffregen <pa...@pjrc.com>
Enviado: lunes, 10 de marzo de 2014 19:24
Para: devel...@arduino.cc

[Paul Stoffregen]

On 03/10/2014 11:27 AM, David Cuartielles wrote:
> aha ... so this basically means that overriding "the traditional way" works and then there are other methods to cope with other cases, am I right?

Yes, on 1.0.6 I believe that is right.

I mis-spoke on the info below. When 2 libraries have the same .h file,
and neither library's folder name is the same as that header name, the
"last" library found is used, not the "first".

Since libraries (inside Arduino) is scanned before the sketchbook
folder, libraries you've copied into the sketchbook will override ones
that are inside Arduino.

However, in the case where 2 conflict within sketchbook, or 2 conflict
within libraries, which one gets used it not well defined. Often it will
be the one that comes last in alphabetical order, but that behavior is
not guaranteed.


I do not know what happens on 1.5.x. I could not find the code which
gives preference to same-named libraries on the 1.5.x branch.

[Vasilis Georgitzikis]

On Mar 10, 2014, at 8:47 PM, Paul Stoffregen <pa...@pjrc.com> wrote:

> On 03/10/2014 11:27 AM, David Cuartielles wrote:
>> aha ... so this basically means that overriding "the traditional way" works and then there are other methods to cope with other cases, am I right?
>
> Yes, on 1.0.6 I believe that is right.
>
> I mis-spoke on the info below. When 2 libraries have the same .h file, and neither library's folder name is the same as that header name, the "last" library found is used, not the "first".
>
> Since libraries (inside Arduino) is scanned before the sketchbook folder, libraries you've copied into the sketchbook will override ones that are inside Arduino.
>
> However, in the case where 2 conflict within sketchbook, or 2 conflict within libraries, which one gets used it not well defined. Often it will be the one that comes last in alphabetical order, but that behavior is not guaranteed.
>

that's exactly why we chose the solution i described. because if you have ~400 libraries in the users' folder (which is essentially what we do), you are guaranteed to have conflicts with no guaranteed behavior

[Bill Perry]

I've always thought that the all this header scanning seemed very odd
and not "normal".

So what about adding the library directories to the include path?
That way users could explicitly specify which library they wanted.
So you end up with:

#include <{libname}/{headername}>

.i.e.

#include <Wire/Wire.h>

or
#include <MyWire/Wire.h>

etc...
And then the users sketchbook library directory is ahead of the system library directory
on the include path.

This way users explicitly specify which library they are using and if there are two libraries with the same
name, the users sketchbook library overrides the system library.

It is very clean and simple.

As in this example the user was able to explicitly specify that he was using the "MyWire" library
and using the "Wire.h" from "MyWire" vs the system "Wire" library "Wire.h"
which is what would end up being used in the current methodology.

This specificity also ensures that you don't have to every worry about or deal with accidental library header
filename collisions in the case of where libraries are using more than just the typical {libname}.h header.


Maybe it could be added to what is there now
and then slowly migrate to the new methodology?

--- bill

[Visual Micro]

I agree that for a future major release paths in lib names is probably the correct solution. However I have tried out the following and it seems to work well enough within our existing structures.

For the current releases both 1.0 and 1.5, I would like to make a slightly different suggestion. 

99% of the time the support issues caused by duplicate headers in libraries result from a user wanting to use LibA and to ignore LibB. Normally users want to use either the <Foo> lib or the <Bar> lib but seldom both. 

In this case we can resolve a list of libraries using the ORDER of the #includes in the sketch. When we  resolve a library we use any other #includes of ancillary .h files found in the library folder. If we find <foo> is #included before <bar> then we grab the bar.h from the foo library. 

If the user #includes the <bar.h> library in the sketch before the <foo.h> library, then we use the bar lib for bar.h and only use the foo lib from bar if bar does not contain a foo.h

So, when we find an #included library during discovery we also find any other #included .h files that exist in the discovered library folder (even if their names match another library folder name)

Tests

1)

Using this system, I setup a library called "Test" with a Test.h and an SPI.h. Then #included <Test.h> and <SPI.h> in my sketch. This included the Test library and did NOT include the SPI library

2)

I then removed the <test.h> #include and the SPI library was used in the sketch compile

3)

Finally I reversed the #includes so that <SPI.h> was above <Test.h>. 

The result was that both the 'SPI' and the 'Test' libraries where included in the compilation

Summary & Notes

This demonstrates 3 clean and easy to document ways to achieve different results without users needing to remove library folders or rename header files.

This is an easy change to make in both 1.0.x and 1.5.x but I think 1.5 should output rev1/rev2 .o object files using a similar directory structure to pre15 libs

Other points of note

I question if libraries without matching header file names should ever have been allowed! However, with this system, any remaining stray #includes can be resolved in the "old way" (after the headers that can be matched to library names (along with the child/ancillary headers) have been processed) 

Regards,

Tim

[Bill Perry]

On 03/14/2014 02:09 PM, Visual Micro wrote:

Other points of note

I question if libraries without matching header file names should ever have been allowed! However, with this system, any remaining stray #includes can be resolved in the "old way" (after the headers that can be matched to library names (along with the child/ancillary headers) have been processed) 

Regards,

Tim

Not exactly sure what you meant by this but there are many cases where you don't want to have a single
header file that happens to match directory name of the library. - Assuming that is what you meant.

I have a very complex graphic library that has many dozens of header files for things like fonts and bitmaps
and to do many complex i/o mapping functions.
While the user doesn't see them or reference them directly by his sketch code, they are included
at compile time.

I think restricting things to a single header file would be an overly reaching restriction.

My preference would be to use dual include paths that point to the libraries directory for the user and the architecture
and then specify the desired library by referencing its directory name in the actual #include.
That way, there are no collisions or ambiguities.
i.e.

#include <foo/bar.h>

If that is what is desired.

--- bill

[Tim Leek]

Hi Bill,

Yes I agree. With the path your headers would no longer be "stray" because their owner is clearly defined.

#includes, with and without paths, can both be supported giving backward compatibility to existing sketches.

My comment related to the many Arduino sketches already in existence that do not #include with a path. 

In this case Pauls solution is to alter the final version(s) or Arduino 1.0.x to include all possible libraries. My variation on Pauls idea gives users more control and will cause less breakages. Support is a huge overhead in this area.

Out of interest Bill is this the library you mention? https://code.google.com/p/glcd-arduino/source/browse/#svn%2Ftrunk%2Fglcd

Tim

---

Date: Fri, 14 Mar 2014 21:12:26 -0500
From: bper...@gmail.com
To: devel...@arduino.cc
Subject: Re: [Developers] Re: Library conflicts

--
You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/m7sEkvANVyo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.

[Tim Leek]

Bill, 

I can see by all your projects why you need the path in the #includes! Wow, nice work. I see why you don't need a matching lib header name. 

Supporting a path and maintaining support for #includes without paths should be possible. 

It just leaves the question about using the "order of #includes" to control which "old style" library includes get priority/de-duped.

Tim

---

Date: Fri, 14 Mar 2014 21:12:26 -0500
From: bper...@gmail.com
To: devel...@arduino.cc
Subject: Re: [Developers] Re: Library conflicts

[Paul Stoffregen]

Just today I and others answered a forum question involving a confusing library conflict.  Still nobody knows what's really wrong, but the best theory is he's got multiple conflicting copies of SD and/or SdFat maybe in the Arduino libraries folder, or his sketchbook folder, or maybe even some files copied into his sketch.  On hearing advice about older versions, he may have copied SD from other Arduino versions into this copy, or his sketchbook.  Like many users, he's not perfectly clear about which locations are where on his computer.  He resorted to installing Arduino 1.0.5 on another computer, to discover the problem wasn't actually in his sketch, but "something messed up" on his machine.  These library conflicts can be a really, really frustrating!

1.5.6 now prints info about which libraries it's really using, but only if verbose info while compiling is turned on (not the default).  The info is printed early and tends to scroll out of sight.

This recent conversation about library conflicts makes me wonder if we could detect conflicts and print very helpful messages only in those troublesome cases.  Really, I suppose this is 2 questions.

#1: Cristian, Tom, Massimo - How would you feel about Arduino printing a library conflict advisory message in the highly visible location right before the sketch size, even without verbose mode?

#2: What types of conflicts should be detected and presented?

Obviously, no message should appear when every #include line matches exactly 1 library.  Even if 2 libraries potentially conflict, the message should only appear if their sketch contains a #include that references the conflict.  A message is probably not worthwhile if there's a conflict that's resolved by one library name matching and the other not matching the header name.

The 2 cases I'd like to see put in front of ALL Arduino users are when their sketch uses a library that's both in their sketchbook and Arduino's libraries, and the case where a #include matches to 2 libraries but it is the neither library's name.  These 2 are the most serious problems that cause tremendous user confusion.

A third useful case might be when the sketch contains a .h file that happens to be the same as any .h file from any of the libraries matched.  Use of .h, .c and .cpp files in sketches is rare, so perhaps this isn't worth the trouble?

A fourth controversial case might be a message when their sketch contains a #include that causes a library to be used, but no #include matching that library name was in their sketch.  For example, #include <EthernetUDP.h> without #include <Ethernet.h>.


This will not be an easy or small patch, since the IDE builds a 1:1 mapping of filenames to libraries at startup.  Information about conflicts is long gone when the sketch is compiled.  So before working on this, since I know IDE changes visible to normal users are a big deal, I'd like to talk about whether this is something you'd accept into Arduino, and if it's welcome, how you feel it should be designed?

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Bill Perry]

On 03/17/2014 06:40 PM, Paul Stoffregen wrote:

Just today I and others answered a forum question involving a confusing library conflict.  Still nobody knows what's really wrong, but the best theory is he's got multiple conflicting copies of SD and/or SdFat maybe in the Arduino libraries folder, or his sketchbook folder, or maybe even some files copied into his sketch.  On hearing advice about older versions, he may have copied SD from other Arduino versions into this copy, or his sketchbook.  Like many users, he's not perfectly clear about which locations are where on his computer.  He resorted to installing Arduino 1.0.5 on another computer, to discover the problem wasn't actually in his sketch, but "something messed up" on his machine.  These library conflicts can be a really, really frustrating!

1.5.6 now prints info about which libraries it's really using, but only if verbose info while compiling is turned on (not the default).  The info is printed early and tends to scroll out of sight.

This recent conversation about library conflicts makes me wonder if we could detect conflicts and print very helpful messages only in those troublesome cases.  Really, I suppose this is 2 questions.

#1: Cristian, Tom, Massimo - How would you feel about Arduino printing a library conflict advisory message in the highly visible location right before the sketch size, even without verbose mode?

Why not report this errors/issues when the IDE starts rather than on each and every compile?
i.e. the IDE can look for it as the IDE parses through all the libraries, which it already has to do anyway
and simply report any issues then, rather than defer them until compile time.
It already is reporting some library issues like ones it is ignoring because of the directory name,
so just add additional checks and warnings/alerts.

This keeps it out of the compile time and having to deal with all the verbose/non-verbose
output issues.


In the bigger picture, is it potentially time to starting considering changing the way includes are done
so that the directory name of the library is also specified. i.e.

#include <foo/foo.h>


It doesn't help existing sketches/libraries if backwards compatibility is preserved,
but moving forward, it solves several of the ambiguity & collision problems.

To me other things like trying to detect which headers the user might not have specified
seems beyond the scope of what the IDE should be trying to detect.
In other words, to me,  that is a user sketch programming error vs a potential build directory/environment
issue.
I think the IDE could be helpful in detecting build directory/environment issues - which it could do at startup,
but should stop short of trying to predict user programming errors.

--- bill

[Peter Feerick]

fwiw, I'm in favour of a warning either at compile time or during startup that informs the user that the #include (compile)/ .h (startup scan) is ambiguous as it matches more than one library / header file. I also like the idea of a warning that a user library may be conflicting with an arduino standard library.

Also, I thing the suggestion as to including the library folder as an additional validation as to which library the  header belongs to is an excellent way to resolve the 'which foo.h' issue, whilst still maintaining compatibility  with old scripts.

Regards,
Peter

[Tom Igoe]

On Mar 17, 2014, at 7:40 PM, Paul Stoffregen <pa...@pjrc.com> wrote:

#1: Cristian, Tom, Massimo - How would you feel about Arduino printing a library conflict advisory message in the highly visible location right before the sketch size, even without verbose mode?

Seems like a good thing.

#2: What types of conflicts should be detected and presented?

How about “You’ve got two copies of XXX included with your sketch and they conflict. Remove one of them.”?

“You included XXX with your sketch and it conflicts with the older XXX version that’s included with Arduino. Upgrade to a newer version of Arduino or remove XXX”

In general, I would like to see messages that recommend what to do, when possible.

Obviously, no message should appear when every #include line matches exactly 1 library.  Even if 2 libraries potentially conflict, the message should only appear if their sketch contains a #include that references the conflict.  A message is probably not worthwhile if there's a conflict that's resolved by one library name matching and the other not matching the header name.

Agreed.

The 2 cases I'd like to see put in front of ALL Arduino users are when their sketch uses a library that's both in their sketchbook and Arduino's libraries, and the case where a #include matches to 2 libraries but it is the neither library's name.  These 2 are the most serious problems that cause tremendous user confusion.

I forgot that second one, but yea, that makes sense too.

A third useful case might be when the sketch contains a .h file that happens to be the same as any .h file from any of the libraries matched.  Use of .h, .c and .cpp files in sketches is rare, so perhaps this isn't worth the trouble?

See “pitches.h” in the tone library. I’ve seen many cases where even relative neophytes use .h files to include things they don’t want in the main file, like long lists of constants, or password constants. So I’d suggest trapping for that somehow.

A fourth controversial case might be a message when their sketch contains a #include that causes a library to be used, but no #include matching that library name was in their sketch.  For example, #include <EthernetUDP.h> without #include <Ethernet.h>.

I wouldn’t do that. People name things all kinds of casual ways, having nothing to do with functionality. In the same way as I’m okay with non-professional writers not knowing how to use the Oxford comma, I’m okay with non-professional programmers using weird naming schemes.

This will not be an easy or small patch, since the IDE builds a 1:1 mapping of filenames to libraries at startup.  Information about conflicts is long gone when the sketch is compiled.  So before working on this, since I know IDE changes visible to normal users are a big deal, I'd like to talk about whether this is something you'd accept into Arduino, and if it's welcome, how you feel it should be designed?

Beyond what I’ve said here, I’ll defer to Cristian as usual.

t.

[Cristian Maglie]

Hi,

On 03/17/2014 06:40 PM, Paul Stoffregen wrote:

> ...Still nobody knows what's really wrong, but the best
> theory is he's got multiple conflicting copies of SD and/or SdFat...

That's frustrating indeed.

> #1: Cristian, Tom, Massimo - How would you feel about Arduino printing a
> library conflict advisory message in the highly visible location right
> before the sketch size, even without verbose mode?

Seems reasonable to me.

> The 2 cases I'd like to see put in front of ALL Arduino users are when their
> sketch uses a library that's both in their sketchbook and Arduino's
> libraries

ok, so we should warn the user that the IDE is using the library in the
sketckbook instead of the one bundled with the IDE.

> and the case where a #include matches to 2 libraries but it is the neither
> library's name.

Right, the behaviour here is undefined too, the IDE may choose randomly one of
the two libraries (or peraphs the last one in alphabetic order). What's the
supposed user's action to solve this conflict? removing one of the two
libraries?

> A third useful case might be when the sketch contains a .h file that happens
> to be the same as any .h file from any of the libraries matched. Use of .h,
> .c and .cpp files in sketches is rare, so perhaps this isn't worth the
> trouble?

Never seen this happen, BTW I guess it doesn't hurts having that case handled
too.

> A fourth controversial case might be a message when their sketch contains a
> #include that causes a library to be used, but no #include matching that
> library name was in their sketch. For example, #include <EthernetUDP.h>
> without #include <Ethernet.h>.

I don't know, has this caused issues until now? it seems that this fourth case
is the less troublesome, I'll avoid to add checks also for this one.

> This will not be an easy or small patch, since the IDE builds a 1:1 mapping
> of filenames to libraries at startup. Information about conflicts is long
> gone when the sketch is compiled. So before working on this, since I know
> IDE changes visible to normal users are a big deal, I'd like to talk about
> whether this is something you'd accept into Arduino, and if it's welcome,
> how you feel it should be designed?

When there are patch that changes the GUI or some visual aspect of the IDE, I
usually hear the opinion of the Arduino Team before taking a decision. In this
case I've seen no strong objections until now so let's go with it.

About how to design this patch, it's hard to say in advance, maybe building a
structure that maps every file to a list of libraries ordered by priority?
Currently the mapping file:library is 1:1, this way it becomes 1:N.
BTW if you are trying that, peraphs it would be useful to send a pull request
of the work in progress to see how it is going?

C

[Cristian Maglie]

In data sabato 15 marzo 2014 03:12:26, Bill Perry ha scritto:
> Why not report this errors/issues when the IDE starts rather than on each
> and every compile?

> This keeps it out of the compile time and having to deal with all the
> verbose/non-verbose output issues.

This may be an option, but most of the cases listed by Paul are triggered by
the user sketch, and maybe the warning is best noticed from the user only when
it's following an error message, instead of a message at startup time that is
easily forget.
The verbose/non-verbose is not a problem here: if I understand right the
proposal, the warning is always displayed, regardless the verbosity.

> My preference would be to use dual include paths that point to the
> libraries directory for the user and the architecture and then specify the
> desired library by referencing its directory name in the actual #include.
> That way, there are no collisions or ambiguities.
> i.e.
>
> #include <foo/bar.h>

This sounds interesting, of course we can't remove inclusions did in the "old
way" for compatibility, but specifying the path may be useful to resolve some
library conflicts, like the second example exposed by Paul, where an #include
matches to 2 libraries.

C