Library include path order (again!)

[Peter Feerick]

Hi all,

A quick question on a topic I remember being the subject of a rather long email discussion (which now cannot find!!), regarding library matching / search order, so my apologies in advance for raising this thorn again... I am using a ESP8266-derived board (Digistump Oak), which includes custom versions of some libraries. I also have the normal versions of some of theselibraries in my sketchbook library folder. And unless I move (delete) the normal library, the IDE is choosing the wrong version of the library...

i.e. 

Multiple libraries were found for "Adafruit_GFX.h"
 Used: E:\Dropbox\Arduino\libraries\Adafruit_GFX
 Not used: C:\Users\Peter\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.1\libraries\Adafruit_GFX

​​

This has been bugging me (pun intended) for the last few versions of the IDE, and I am at a loss as to why the sketchbook library is given priority over the version bundled with the core? Is that by design, or a bug?

Regards,

Peter

Peter Feerick
BIT, BLDes CQU

[Paul Stoffregen]

On 05/14/2016 03:39 AM, Peter Feerick wrote:

Is that by design, or a bug?

It's by design.  Definitely not a bug.

.... as to why the sketchbook library is given priority over the version bundled with the core?

So you, the end user, can override whatever libs came with the 3rd party core, or bundled with the Arduino IDE.  If it weren't this way, you'd be stuck with whatever the platform/core provides, without any way to override it.  Arduino is very much a system about learning and DIY experimenting, so the ability to override anything with your own hacks is a pretty fundamental part of the design.

However, the sketchbook library override is all-or-none with respect to platforms, cores, boards & variants.  Your copy in the sketchbook overrides them all.  Perhaps some day it will become more fine-grained, adding even more complexity to Arduino's build system that's grown far more complex in recent years, but how that could even be done in a way that's usable and easy to learn for users is a very good question.

This has been bugging me (pun intended) for the last few versions of the IDE

Library overrides have existed this way in Arduino since well before version 1.0.  But widespread use of the Arduino IDE for more than very similar AVR-based boards is relatively new (very few people used the "beta" 1.5 versions).  Those messages about multiple copies found are even newer.  For that matter, widespread library forking and fragmentation within the Arduino ecosystem is also relatively new, to the point high-level libs not tightly tied to hardware, like Adafruit_GFX, are the sort of thing you're overriding.

Currently there isn't any provision in the software to override a library for just one platform or core or board or variant.  Well, other than the message does tell you the full pathname, so you could just navigate to that folder and put your customized library there.  Even if not ideal, hopefully the message at least makes the situation clear for you?  (I'm the guy who contributed those messages, starting with Arduino 1.6.3.)

[Paul Stoffregen]

On 05/14/2016 04:14 AM, Paul Stoffregen wrote:

If it weren't this way, you'd be stuck with whatever the platform/core provides, without any way to override it.

Perhaps I mis-spoke (mis-typed) somewhat on this point?  Or course you can override anything if you know its full path.  There's always *some* way.

But the sketchbook folder is your only long-term safe location for storing your own work, and have the Arduino IDE automatically make use of it.

Suppose you undertake an ambitious rewrite of the SD library for many-sector caching and thread safety.  That's a LOT of work.  Even if your edits are small, they may be based on many hours of experimentation.  If you customize the SD lib in C:\Program Files (x86)\Arduino\libraries\SD, you almost certainly will lose all your hard work the next time you upgrade Arduino.  Likewise, customizing the copies installed by the boards manager isn't a safe place to store your work.

That's the reason *why* the sketchbook libraries folder automatically overrides the other library locations.  It's meant to enable you to edit the libraries and have your edits override anything else, and keep your work safe from loss when you upgrade the IDE or boards packages.

[Tom Igoe]

I have to say, the library matching came up on some sketches I was working on recently, and it did its job humbly and flawlessly, letting me know what it had done without getting in the way. I didnt even think about it, it just worked. Thank you for that.

t.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Tom Igoe]

(this bounced, so sorry if it comes to any of you twice)

I have to say, the library matching came up on some sketches I was working on recently, and it did its job humbly and flawlessly, letting me know what it had done without getting in the way. I didnt even think about it, it just worked. Thank you for that.

t.

On May 14, 2016, at 7:31 AM, Paul Stoffregen <pa...@pjrc.com> wrote:

[Peter Feerick]

Ok, that makes sense!

Seriously though, Paul, thank you for your detailed response - that makes perfect sense now! I remember when you added the library message, and that was indeed an much needed improvement... no more scratching your head going WTF when an error keeps coming up when you know you changed that code... only to later realise the IDE had been including a different library the whole time! And yes, the build system is very much becoming a bigger than ben hurr quagmire... so I can understand that making changes that might improve things further adds more opportunities for things to go wrong! 

So, a further question, with a workaround in mind (other than patching the sketchbook version of the library, thus making it non-standard, which then raises the question of 'if I patch the library files in the sketchbook, what will stop the library manager from overwriting the changes when the library updates?'). In the scenario here, when compiling for the Oak, and trying to use an Oak customised library, is there a better way to explicitly tell the IDE to use that library - even if I need to give the full path - maybe with a macro that points to the "...\digistump\hardware\oak" folder so it isn't so path dependent? I don't seem to be able to able to specify libraries by absolute path anymore... IDE still automatically grabs the wrong library. 

I take it I only have two options at the moment - 

a) patch the adafruit library with conditional compilation code for the Oak (doable, and I would need to submit a patch so it doesn't come back to bite me - dependent on how idiot-proof/safe library updating is); or

b) keep moving the adafruit library out of the libraries folder whenever working on the Oak (or have a separate sketchbook folder with Oak-specific code/libraries) - doably, just annoying

c) something else?

Cheers,

Peter

Peter Feerick
BIT, BLDes CQU

[Jeff Hoefs]

There was a change to the logic a couple of months ago that should be available as of Arduino IDE v1.6.9. In this change, as I understand it, a core library will have a higher priority than the sketchbook library (of the same name) if the core library's library.properties file defines an architectures param that differs from that specified in the sketchbook library's library.properties file.

https://github.com/arduino/arduino-builder/pull/123

So in the library.properties files of core libraries included with the digistump oak package, make sure the architectures parameter is specified as ESP8266 or whatever the decided value is for that architecture. I've never tried this myself, but based on the referenced PR that should work.

[Peter Feerick]

Thanks for that Jeff... as soon as you said library.properties the penny dropped. duh!

I was running an older version of the Adafruit_GFX lib (pre-library manager), so no library properties file, and wasn't up-to-date anyway... only two years old, and consequently pre-ESP8266 also. Installing the latest version and nuking my older copy has since removed the compile-time error. Interestingly enough, the multiple-library message isn't coming up anymore either - even though there are still two copies of the library present - one in the sketchboook, and one in the core folder. They both indicate that they support all architectures, (architectures=*), so maybe that has something to do with it? 

Peter Feerick
BIT, BLDes CQU

[Paul Stoffregen]

On 05/14/2016 08:05 PM, Peter Feerick wrote:

Interestingly enough, the multiple-library message isn't coming up anymore either - even though there are still two copies of the library present - one in the sketchboook, and one in the core folder.

Oh, that's so not good.  Please open a new issue.

Hopefully this can get fixed and 1.6.10 might arrive soon.  I'm really, really not wanting to go back to the bad-old-days of lengthy forum threads unable to resolve mysterious compile problems due to library conflicts!

[Matthijs Kooijman]

Hey Peter,

> Interestingly enough, the multiple-library message isn't coming up
> anymore either - even though there are still two copies of the library
> present - one in the sketchboook, and one in the core folder. They both
> indicate that they support all architectures, (architectures=*), so maybe
> that has something to do with it?

I just tried reproducing this problem (using a copy of the AVR Wire
library and changing the architectures line to *), but the message still
appeared for me.

Could you tell what ESP core version you are using and what library
version you have in your sketchbook, preferably with links so I can
recreate your setup? Alternatively, you could zip up your Sketchbook's
hardware and libraries folders and send them to me privately, so I can
recreate your exact setup here.

Gr.

Matthijs

[Matthijs Kooijman]

Hey Peter,

> Could you tell what ESP core version you are using and what library
> version you have in your sketchbook, preferably with links so I can
> recreate your setup? Alternatively, you could zip up your Sketchbook's
> hardware and libraries folders and send them to me privately, so I can
> recreate your exact setup here.

Never mind that. I tried the Digistump Oak platform 1.0.1 and the
Adafruit GFX library 1.1.5 and managed to reproduce the problem. I'll
make sure this is fixed in the next Arduino IDE release.

Thanks for pointing this out :-)

Gr.

Matthijs

[Matthijs Kooijman]

Hey Peter (& Paul),

> Interestingly enough, the multiple-library message isn't coming up
> anymore either - even though there are still two copies of the library
> present - one in the sketchboook, and one in the core folder. They both
> indicate that they support all architectures, (architectures=*), so maybe
> that has something to do with it?

This is fixed by https://github.com/arduino/arduino-builder/pull/151 see
the PR description for a more details.

Gr.

Matthijs

[Peter Feerick]

Hi Matthijs,

That was essentially my setup when that error occurred - i.e. Oak 1.0.1
and GFX 1.1.5 (just not a fresh-install copy). It was going to be a few
days before I could look into that further to see if it was a one-off or
I was missing something, so thanks for looking into it... and patching
it too! :D

Cheers,
Pete