Arduino Dependency Management?

51 views
Skip to first unread message

Alexander Christian

unread,
Apr 18, 2018, 5:17:41 AM4/18/18
to Developers
Hey there,

I already added a comment here, because the context looks similar -> https://github.com/arduino/Arduino/pull/6004#issuecomment-379227768

but I think finally this developer mailinglist targets the better audience ...

So, I'll again post here:


If I download a sketch/project from somewhere on the web and want to build the sketch, I first need to manually check the dependencies. I have to check the "includes" and maybe also the documentation. Then I need to search for the right library in the library manager and finally install it. THEN I can try to build it. And if I found the correct library with the correct version it will hopefully compile successfully.


What about having a kind of "dependency-config" for a sketch or a kind of meta-information IN the sketch that lets the Arduino IDE detect which version the build would require and finally download the correct version.
But then of course different sketches might use different version of same library. So arduino IDE would need to use a dedicated lib-folder for each sketch/project.

This would for sure improve the situation where you want to rebuild a certain project that you actually do not own, f.i. projects you found on Github etc ...


Of course this would add another level of complexity to a sketch. But it should be possible to have this kind of backward compatible: if dependency-information is available, use it. If it's not available, proceed as before: let the user find the dependencies on his own.


Have there been any thoughts about this kind of sketch-to-lib-dependency-management? If so: What's the progress?


kind regards,

Alex

Alexander Christian

unread,
Apr 18, 2018, 12:46:07 PM4/18/18
to Developers
I searched a bit, and found... nothing. Seems that a lot of people are facing this problem, but noone has a solution for this.

To precise my idea:
I'm thinking of something like this

/*
 * @LibraryDependency(name="FooLib", version="1.0.0")
 * @LibraryDependency(name="BarLib", version="1.2.3")
 */

on the top of the sketch. A comment with kind of java-like annotation, which describes what libraries in which versiont he sketch needs.

If the Arduino IDE detects this kind of comment, it could create a "libraries" (or temp or whatever) folder for THIS sketch, downloads the dependant libraries in the matching version into the sketch-specific libraries/temp/whatever folder and when it compiles, it compiles against those dependant libraries instead of the general sketchbook-libraries folder.

If there is no such kind of annotated comment, the Arduino IDE will use the sketchbook-libraries-folder as usual. So it's backward compatible and the advanced user can set up the required dependencies directly, instead of just documenting: you need to download this and that in this version before compiling..

I'm not bound to exactly this kind of "annotated comment". It could be also f.i. a properties-file like:

sketchdependencies.properties:
dependency_1=FooLib@1.0.0
dependency_2
=BarLib@1.2.3

or something similar.
All I request is, that the sketch-writer can tell the Arduino IDE: I need this and that library in this an that version. And Arduino IDE is already able to download libraries in specific version. So it's just about:

  • detecting the dependencies (comment, extra file, whatever)
  • create sketch specific lib/temp folder
  • download the dependant libs
  • compile against this folder instead of sketchbook-libraries folder

I'm sure this would ease the procss of "downloading a 3rd pary sketch and bring it to live" a lot ...

Open Questions:

* I'm not sure if this really needs to be tied to the board type. Typically a sketch is build a special type of board or, if the sketch works with several boards, the sketch has kind of #ifdef to decide what to include for which board.


Any comments in this idea in general? Any supporters out there?


best regards,
Alex

Nathan Seidle

unread,
Apr 18, 2018, 1:25:56 PM4/18/18
to devel...@arduino.cc
As the generator of a lot of libraries: I love this idea and we very much need something to make getting time-to-first-blink as short as possible. 

This is perhaps scope creep but it would be helpful if the Arduino IDE supported links to the library manager.

    #include "SparkFun_UHF_RFID_Reader.h" //Library for controlling the M6E Nano module

The user sees the above code in a sketch or example code but it's unclear where to obtain this dependency. Should I search the internet? Download a repo? Or try searching some terms in the library manger? In a perfect world a user would click a link within the sketch obtaining the library or at least opening the library manager at the requisite entry. For example:

    #include "SparkFun_UHF_RFID_Reader.h" //lib:"SparkFun Simultaneous RFID"@"v1.0.2"

With the lib:... being clicked to open the library manager.

The library.properties file has support for inputting platform information for that particular library. Perhaps the library link could include platform information as well.

Thanks for starting the discussion!
-Nathan




--
Nathan Seidle, Founder


--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+unsubscribe@arduino.cc.

Mike

unread,
Apr 18, 2018, 1:31:25 PM4/18/18
to devel...@arduino.cc
A question - perhaps a library has jumped from 1.0.2 to 1.0.3 (let alone 2.0.0)

This may or may not have broken the library IDE (hopefully not if the library author is careful).

How does the dependency manager square the code asking for 1.0.2 and the GitHub or Library Manager version being higher? If it is the Library manager, it could look for the lower level version number. If it's an outside library, older versions might not be found or a newer user might not know enough Git to get an older version out of a repo.

Just a consideration that I had. I agree, good discussion.

Ian Katz

unread,
Apr 18, 2018, 2:50:29 PM4/18/18
to devel...@arduino.cc
Is a package management paradigm like Ruby's bundler / Node's npm / etc out of the question here?  Those support both dependency fetching and semantic versioning, as well as a system for understanding whether the dependencies exist locally, in a package repository, or in a git repository (of a specific revision).

In practice, this would mean specifying your library dependencies in a file with an agreed-upon name -- not in the sketch itself.  Libraries would link to their own dependencies.  

Is that something that previously ended in failure?

per1234

unread,
Apr 18, 2018, 6:17:21 PM4/18/18
to Developers
I think this issue and PR is relevant to this discussion:
https://github.com/arduino/Arduino/issues/1481
https://github.com/arduino/Arduino/pull/7349
since it is proposing adding metadata to the sketch which

https://github.com/arduino/Arduino/pull/6004 is a bit different since it's about library dependencies rather than sketch dependencies and libraries already have an existing metadata file (library.properties).


This is perhaps scope creep but it would be helpful if the Arduino IDE supported links to the library manager.

It does already:
https://github.com/arduino/Arduino/commit/d1516f60b8d65662f533d69bf8e0493a6853a341
 
#include "SparkFun_UHF_RFID_Reader.h" // http://librarymanager/All#SparkFun_Simultaneous_RFID_Tag_Reader_Library

One of several useful features of the Arduino IDE that are largely unused because the developers didn't bother to properly document them. I think we really need an Arduino Sketch Specification page added to the arduino/Arduino GitHub wiki for this sort of information, especially if sketch metadata is added.


How does the dependency manager square the code asking for 1.0.2 and the GitHub or Library Manager version being higher?

I think it will only be practical to handle sketch dependencies for libraries that are in the Library Manager index. That index contains all releases of the library that meet the Library Manager requirements. So I don't think there's any issue with specifying the dependency version.

Nathan Seidle

unread,
Apr 18, 2018, 6:25:24 PM4/18/18
to devel...@arduino.cc
@per1234 - Thank you very much! I did not know that feature existed and will help with our examples a lot.

--
Nathan Seidle, Founder


--

Alexander Christian

unread,
Apr 19, 2018, 2:43:38 AM4/19/18
to Developers


Am Donnerstag, 19. April 2018 00:17:21 UTC+2 schrieb per1234:
I think this issue and PR is relevant to this discussion:
https://github.com/arduino/Arduino/issues/1481
https://github.com/arduino/Arduino/pull/7349
since it is proposing adding metadata to the sketch which
I ws not aware of this. But it sounds very interesting. Looking forward to get this PR merged into next release. *thumbsup*
 

https://github.com/arduino/Arduino/pull/6004 is a bit different since it's about library dependencies rather than sketch dependencies and libraries already have an existing metadata file (library.properties).
That was the PR is was refering to. And that also lead to this discussion. 

 

This is perhaps scope creep but it would be helpful if the Arduino IDE supported links to the library manager.

This is very interesting and already helps to navigate the user to the right library.  If one could also add/specify the version, this would be even better.

But for the automated lib dependency management, I would go for the sketch metadata and extend them with lib information.

I downloaded the Arduino IDEs source and had a look at it. In general is does not seem to be too complicated to add a sketch specific lib folder. But I have no idea about the indexer. 


"The direction is to remove as much logic as possible from the Java IDE, making it more or less an interface to cli tools. "

I agree with him. This sketch library dependency thing should be also available with the command line tools. 




One of several useful features of the Arduino IDE that are largely unused because the developers didn't bother to properly document them. I think we really need an Arduino Sketch Specification page added to the arduino/Arduino GitHub wiki for this sort of information, especially if sketch metadata is added.

I would love to see an improved documentation about all those "hidden" features.
 

How does the dependency manager square the code asking for 1.0.2 and the GitHub or Library Manager version being higher?

I think it will only be practical to handle sketch dependencies for libraries that are in the Library Manager index. That index contains all releases of the library that meet the Library Manager requirements. So I don't think there's any issue with specifying the dependency version.

*FullACK* the lib manager knows about the current and the past version. And the sketch author knows, whoch version fits to his sketch. So the sketch library dependency solver should go for matching lib version. Otherwise it will break things. 


Reply all
Reply to author
Forward
0 new messages