1.5 Lib: How to specify library dependencies

[Cristian Maglie]

Paul expressed his concerns:

I have three areas of concern about this 1.5 library spec.

1: Impact on end user feedback & community participation

2: Backwards compatibility with existing libraries

3: Effectiveness in describing required architecture

to solve this problem he propose:

Replace "architectures=" and "core-dependencies="

with "board-feature-required=" and "core-feature-required=".

Add feature lines in boards.txt and add features.txt in the core folder.

Paul may you elaborate a bit on your idea?

Why "features" doesn't lead to the same problem?

C

[Peter Feerick]

Cristian,

I'm not Paul, but I see this as allowing cores and boards to 'publish' what features they offer, and libraries to list which features they 'require'  - thus allowing the IDE to dynamically show/hide/use libraries based on boards/cores fulfilling library requirements.

Just my two cents :)

Pete

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Cristian Maglie]

In data lunedì 28 ottobre 2013 11:46:53, Peter Feerick ha scritto:
> I'm not Paul, but I see this as allowing cores and boards to 'publish' what
> features they offer, and libraries to list which features they 'require' -
> thus allowing the IDE to dynamically show/hide/use libraries based on
> boards/cores fulfilling library requirements.

May you elaborate a real-world example?

C

[Paul Stoffregen]

> Replace "architectures=" and "core-dependencies="
>
> with "board-feature-required=" and "core-feature-required=".
>
> Add feature lines in boards.txt and add features.txt in the core folder.
>
> Paul may you elaborate a bit on your idea?
>
> Why "features" doesn't lead to the same problem?
>

As Bill explained, "architectures=" is far too coarse. But listing
features (hopefully) can provide the necessary level of detail. Feature
lists should also be much less prone to human error.

Here are some quick examples.

1: The Esplora library requires numerous off-chip peripherals.
Architecture AVR says it would work on Uno, Leonardo, Mega. Listing
board-specific features in boards.txt would allow Esplora require those
features in its library.properties.

2: Bridge also claims to work on all AVR boards, but obviously Yun is
currently the only compatible board. But requiring AVR for Bridge
hardly seems like a good idea! Any architecture that implements a
serial object could work with bridge, if a Linux system+software is
connected. In fact, Bridge appears to use the Stream class properly, so
theoretically an EthernetClient stream could be used by Bridge. Like
Esplora, specific features in boards.txt can specify whether a Linux
system is connected

3: Temboo & SpacebrewYun are currently specified as AVR only, even
though they only depend on Bridge, which illustrates another shortcoming
of the architectures= approach: many library authors will just
copy-and-paste "architectures=avr". A specific list of
hardware/software features is much less likely to be incorrectly copied
than the very generic "architectures=avr".

4: The SPI and Wire libraries are specified as "architectures=avr,sam",
which is true for all official Arduino boards. But 3rd party boards
like Digispark and Adafruit Trinket will match to AVR, even though the
tiny chips they use lack the SPI and TWI peripherals. Listing every
hardware feature in boards.txt probably isn't feasible, but referencing
a file that lists all the on-chip features for that board's particular
chip would probably work. There are vast numbers of different AVR chips
with different capabilities, which can be matched by feature lists.

[Tom Igoe]

That was a really helpful and succinct explanation Paul. Thank you.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Matthijs Kooijman]

Hi folks,

regarding this issue, a few observations.

First, it seems this discussion is really about hardware dependencies,
rather than software dependencies. With software dependencies I mean
that a library might depend on the Wire library and needs to declare
this to the IDE so it can set up the appropriate include and source
paths. I think the software dependencies part is already covered and in
any case it's not the topic of this thread.

This thread is about hardware dependencies. In general, the question the
IDE wants to answer is "will library X work on board Y?".

Now, this is really not an easy question. In fact, in a lot of cases,
the answer really should be 'I don't know' instead of a plain yes or no.
In particular, a library author can not easily predict on what boards a
library will work.

For this reason, it might be useful to consider implementing something
less black and what. Note that below when I say "Specify boards" I mean
that in a general manner, without defining how to specify them (more on
that below).

It is easy for the board author to:
- Specify boards that the library will certainly work on. These can be
the boards the author tested on, or perhaps some bigger list if the
author knows a bit more about hardware features and can predict some
other boards that "should" work as well.
- Specify boards that the library will certainly _not_ work on. The
most obvious criterium for this is of course the architecture: A
library containing (only) AVR-specific will certainly not work on
other architectures.
All boards not falling in either of these two categories "might work"
and should probably be offered to the user to try (probably with a
message stating the unknown support status).

We can expect that in practice, a lot of boards will have the "might
work" status for a given library. If we allow multiple versions of a
library to be installed (which I somewhat proposed in the "Library
Layout" thread), where each version might support different boards,
there is a big chance that for a given board there will be multiple
library versions with "might work" status (e.g. one library version that
works on the uno, one that was modified to work on the leonardo, but
neither of them say anything about working on the mega).

In this case, it seems like a good idea for the IDE to simply ask the
user which of these libraries he wants to use (and make it easy to
switch afterwards).

Furthermore, if we have a good mechanism like this for dealing with
this "might work" state, I think we can afford to make the
"board-specifying-mechanism" less advanced and specific. Even if you
cannot specify _exactly_ on which boards your library will work, that's
ok: Just be conservative in the specification and the user can sort out
the rest.



Now, how to specify this list of boards? The most naive way to implement
this, is to do just that: specify an explicit list of boards. Very easy
to implement, but also very limited: a library author often only has one
or two boards to test on and would have to consider each board
individually to predict if the library would work. Any boards that are
built after releasing the library, or any boards from a 3rd party the
library author did not consider are never specified in the list. From a
"conservative" perspective, this is of course ok, just let the user sort
it out. However, in practice this will probably mean the user has to
sort it out in pretty much every case, which isn't so good.


Limiting on architectures makes things a lot better, though probably
only to say "These architectures might be supported, all others are
certainly not". For example, specifying architectures=avr should not
cause all avr-boards to be marked as supported, only to mark all non-avr
boards as not-supported. This implies that there is another mechanism
needed to mark boards as supported. Combining a list of architectures
with an explicit list of supported boards might be better, but it might
still be too limited.


Previously, something like "board features" was suggested. Each board
lists a number of features it has, each library specifies which features
it needs. However, I'm afraid that it will be extremely hard to properly
define these features.

For example, we could have a board feature "INT0", for boards that have
an "external interrupt 0". However, for this board feature to be
actually useful, we'd have to also specify the API implied by the INT0
feature: the ISC00 and ISC01 bits in EICRA specify the type, the INT0
bit in EIMSK masks the interrupt. This is how things work on a atmega328.

However, an attiny13 also supports the INT0 interrupt. However, it uses
different registers to control it: ISC00 and ISC01 live in MCUCR and
INT0 lives in GIMSK. Note that I'm using attiny13 here just because I've
been working with it, I'm sure there are similar differences between
atmega processors.

The point here is that small differences between chips might cause code
to not work on both, which would require a _lot_ of very fine-grained
board features to be specified. The advantage could be that you can
pretty reliably predict wether a library will work on a board. However,
I'm afraid that the extra complexity will cause both board and library
authors to make mistakes when choosing the features, which would defeat
the purpose.

Alternatively, we could just ignore these differences and API stuff and
just say "INT0" means the board has an INT0 interrupt. If a library
needs INT0, it should take care of #ifdeffing the right registers to
use, or perhaps just only use the attachInterrupt function.

In this case, specifying the library requires "INT0" doesn't really
guarantee it will work on any board that has INT0 (the code might not
have the right #ifdefs for the board), but it does guarantee things will
_not_ work on any board that does not have INT0.


So, in summary, I think:
- We should focus on making a library specify on which boards it
_might_ work (instead of trying to define on which boards it will
certainly work). Any boards which aren't specified as "might work"
are automatically marked as "will not work".
- The current architectures= property is fine to specify the
architectures on which the library "might work".
- We can add a "board features required" property to specify additional
requirements on the board. I think "board features" and
"architecture" are complementary here (especially since a board has
to fullfil the requirements of both to be marked as "might work").
- We could add an explicit list of board names that are supported
and/or the library was tested on. Perhaps also an explicit list of
board names that the library will certainly not work on?
- The IDE should find all versions of a given library and then:
1. If the user explicitly specified a version, use that one.
2. Otherwise, find out all versions that certainly support the
current board.
3. If this is just one version, use that.
4. If this is more than one version, allow the user to choose the library to use.
5. If there are no versions that certainly support the current
board, find out all versions that "might" support the board.
6. If this is just one version, use that.
7. If this is more than one version, allow the user to choose the library to use.
8. If there are no versions that "might" support the current board,
show an error.

How does this sound?

Note that this proposal is pretty perpendical to the discussion in the
Library Layout thread: The above works fine both with libraries
supporting a single architecture, as well as multi-arch libraries. The
above does also not define where (in the directory layout) multiple
versions of one library should be saved, but I proposed something for
this in the other thread as well.

Gr.

Matthijs

[John Plocher]

As a library author, board developer and shield developer, I see this confusion from all three ends, so to speak.

Paul's suggestion for a feature test matrix in the software/library realm sounds like it has the potential to make sense of this mess - but it is only part of the battle. As Paul, Matthijs and Bill mentioned, there still needs to be some support for abstracting the register bittwiddling and pin mapping needed to make it all work.  

The recent attachInterruptToPin() discussion is a step, but without comprehensive core support for exposing this type of information, it is difficult to write a library that will ever be more than a one-off.

My internalization of Paul's suggestion (at a high level) is:

The Variant="328p" and Variant ="32u4" might both say "I'm AVR", "I have TIMER", "I have DigitalPins" and "I have AnalogPins"

The Uno Board would say"I'm a '328P" as well as "Digital on pins 1-18", "PWM is supported on 3,5,6,9,10,11,  "Analog pins are 14,15,16,17,18,19" and  "InputCompare is on pin 8", 

The Leonardo would say  "I'm a '32u4" as well as "Digital on pins 1-18", "PWM is supported on 3,5,6,9,10,11,12,13" "Analog pins are 4,6,8,9,10,11,14,15,16,17,18,19" and  "InputCompare is on pin 4

 

My library would say "I need AVR", "I can use Uno", "I can use Leonardo", "I need InputCompare", "I need DigitalOutput" and "I need TIMER"

The GUI/IDE view is easy - the places where there are matches are shown in bold, the others greyed out and the user gets to choose.  

What is less clear to me is how this looks from the library developer's perspective.

How does this all come together in my public library API?  My initialization/binding call/constructor is:

         Loconet.init(TxPin, RxPin) 

Somehow it needs to associate "RxPin" with "InputCompare" and "TxPin" with "Digital", while the implementation itself depends on "TIMER"

Inside my implementation, I need well defined feature test macros or utility macros/functions or well known global symbol bindings for lookup tables (or...):

     If I'm on a Uno, RxPin must be 8, I need to install ISRs ICP1 and TIMER1 and set up the on-chip registers yadda yadda

     if on Leonardo, RxPin must be 4, ISRs differ this way, TIMERs that way...

     and on a tiny or tweensy or a nano or ...

     while on a Mega or Due it could be xx or xx or xx

(whew) and that's only the popular AVR variations...

This is why I say that Paul's board-feature-required= vision is only a start.

Some comments on Matthijs' post, which prompted mine:

- We should focus on making a library specify on which boards it
   _might_ work (instead of trying to define on which boards it will
   certainly work). Any boards which aren't specified as "might work"
   are automatically marked as "will not work".

As a library author, all I can tell you is that whatever value I would give for this property, it would be wrong and misleading, simply because the overlap between the "I don't have that board/ignorance" and "yes/no" is so overwhelming.  I can say "I tested this on that and it worked", but unless I have first hand experience with the differences, I am more likely to assert "Yeah, it should work with all the Arduinos that are shield compatible", without realizing that the '328 (Uno) and the 32u4 (Leonardo) are really different beasts under the covers and WON'T work at all.  Either I will lie by omission or commission - and either will result in dissatisfied end users.

 

 - The current architectures= property is fine to specify the
   architectures on which the library "might work".

I don't find it so in practice:

My I2C expander library depends on Wire, but not on AVR architecture.  While I have never tested it, there is no reason why my library wouldn't work just fine on a chipKIT or a Due, presuming that they supported Wire.  Same for my Loconet library - it depends on some timer, interrupt and ICP juju, all of which certainly *could* exist on other architectures, and (given the right core macro config/provisioning support/abstractions) might "just work".

This is exactly what I am hoping the new Arduino Library model provides - the framework and support for non-trivial libraries to somehow "just work" (or "just work with a bit of developer community hand holding") across ALL the various architectures and variants that provide the required features.

 

 - We could add an explicit list of board names

There should (IMHO) be a defined taxonomy of processor defines for these variations such that my library can use them as part of its feature-testing...

I like the idea of starting with a comprehensive example library that shows off all the cross-architecture and cross-variant tricks, tips and tools.  Right now, all the 3rd party libraries out there look like clones of morse(), and none of them are very agile across the variations - because there is no good example to work from.

  -John

 

[Matthijs Kooijman]

Hey John,

thanks for your thoughts, they look insightful! I'll reply to some of
your remarks.

> - We should focus on making a library specify on which boards it
> > _might_ work (instead of trying to define on which boards it will
> > certainly work). Any boards which aren't specified as "might work"
> > are automatically marked as "will not work".
> >
>
> As a library author, all I can tell you is that whatever value I would give
> for this property, it would be wrong and misleading, simply because the
> overlap between the "I don't have that board/ignorance" and "yes/no" is so
> overwhelming. I can say "I tested this on that and it worked", but unless
> I have first hand experience with the differences, I am more likely to
> assert "Yeah, it should work with all the Arduinos that are shield
> compatible", without realizing that the '328 (Uno) and the 32u4 (Leonardo)
> are really different beasts under the covers and WON'T work at all. Either
> I will lie by omission or commission - and either will result in
> dissatisfied end users.

That's exactly the point I was trying to make, but I think you worded it
a bit more explicitly and cleary.

> > - The current architectures= property is fine to specify the
> > architectures on which the library "might work".
> >
>
> I don't find it so in practice:
>
> My I2C expander library depends on Wire, but not on AVR architecture.
> While I have never tested it, there is no reason why my library wouldn't
> work just fine on a chipKIT or a Due, presuming that they supported Wire.

If this is the case, you're library would specify "architectures=*",
meaning it might work on any architecture. Another part of the library
metadata would specify it needs the Wire library, so the IDE knows it
won't work on boards/architectures without Wire (if any).

More specifically, I'm saying the architectures= property could be
well-suited for libraries that actually do some hardware-specific
stuff (such as direct port register manipulation), since they're certain
that the might work work on some architectures and not on others.
Libraries without any such hardware-specific code will just list that
they might work on all architectures, possibly further constrained by a
"required board features" entry as proposed by paul.

So, I guess I was mostly saying that the architectures= property could
very well co-exist with a required board features mechanism (and both
would solve a different part of the problem). Of course, we could
probably define a "uses AVR" and "uses ARM" board feature and express
the architecture constraints in terms of board features, which might
also be a fine solution (depends on the details of the board feature
mechanism, I guess).

> This is exactly what I am hoping the new Arduino Library model provides -
> the framework and support for non-trivial libraries to somehow "just work"
> (or "just work with a bit of developer community hand holding") across ALL
> the various architectures and variants that provide the required features.

I actually think that this might be orthogonal to the topic of this
thread: Making libraries work on different architectures requires
improved APIs and/or easy ways to #ifdef in the code to make it work
across various boards. AFAIU the topic of this thread is more about how
to declare / predict on what hardware a board will (not) work, given
some implementation.

Gr.

Matthijs

[Paul Stoffregen]

On 11/04/2013 12:29 AM, Matthijs Kooijman wrote:
> More specifically, I'm saying the architectures= property could be
> well-suited for libraries that actually do some hardware-specific
> stuff (such as direct port register manipulation), since they're
> certain that the might work work on some architectures and not on others.

Actually, this example (direct port register manipulation) is precisely
why architectures= is a bad idea.

Teensy3 uses compile-time software tricks to translate a few of the most
popular AVR registers/syntax into native Kinetis code. A library author
using "PORTD |= (1<<4)" would rightly specify architectures=avr in
library.properties, making it incompatible with any non-AVR boards which
actually do support that syntax.

1.5.X's architectures= prevents compatibility layers!

> So, I guess I was mostly saying that the architectures= property could
> very well co-exist with a required board features mechanism

I will simply be forced to mis-label Teensy3 as AVR for the sake of
compatibility with libraries that specify "architectures=avr". Since
some libraries will be mistakenly published with "architectures=avr"
(eg, a couple in Arduino 1.5.4 do), the odds of ANY 3rd party board
specifying anything other than "avr" or "sam" are very low. Why would they?

Is everyone specifying AVR, whether it's true or not, how you envision
this system working in practice?

[Matthijs Kooijman]

Hey Paul,

On Mon, Nov 04, 2013 at 02:38:40AM -0800, Paul Stoffregen wrote:
> On 11/04/2013 12:29 AM, Matthijs Kooijman wrote:
> >More specifically, I'm saying the architectures= property could be
> >well-suited for libraries that actually do some hardware-specific
> >stuff (such as direct port register manipulation), since they're
> >certain that the might work work on some architectures and not on
> >others.
>
> Actually, this example (direct port register manipulation) is
> precisely why architectures= is a bad idea.
>
> Teensy3 uses compile-time software tricks to translate a few of the
> most popular AVR registers/syntax into native Kinetis code. A
> library author using "PORTD |= (1<<4)" would rightly specify
> architectures=avr in library.properties, making it incompatible with
> any non-AVR boards which actually do support that syntax.
>
> 1.5.X's architectures= prevents compatibility layers!

Hmm, adding compatibility layers like these adds another level of
complexity to the problem, I guess. In addition to high-level APIs that
a board/core can or cannot support, now you also have low-level
(nominally architecture-specific) "APIs" that might be supported by
different boards on different architures...

In essence, the problem thus is that the architectures= specification is
too coarse. If youre teensy3 core would emulate _all_ of the
AVR-specific APIs, then it would be ok for the teensy core to specify it
(also) supports/has the avr architecture. In practice, this is of course
impossible, and the teensy core will only support a subset of the
AVR-specific APIs. This makes the architectures= specification too
coarse, since it does not allow library authors to specify what they
need exactly.

This can probably be solved better by including these kinds of
architecture-specific APIs in the list of "board features" like you
proposed. This should allow to make them more fine-grained (though it is
still a challenge to get the right level of granularity, of course. In
particular when faced with different variants of the same API between
different AVR chips).

Summarizing, an architectures= property is not expressive enough. This
could be ok if it causes not-supported boards to be marked as
(potentially) supported, but your example shows that it can cause
supported boards to be marked as not-supported, which is a real problem.
I'd even say that it would be better to just leave out the
architectures= specification entirely and let the IDE have no idea at
all about support (e.g., let the user just try and select a working
library version themselves), than implement architectures= which
advertises the wrong things...

> >So, I guess I was mostly saying that the architectures= property
> >could very well co-exist with a required board features mechanism
>
> I will simply be forced to mis-label Teensy3 as AVR for the sake of
> compatibility with libraries that specify "architectures=avr". Since
> some libraries will be mistakenly published with "architectures=avr"
> (eg, a couple in Arduino 1.5.4 do), the odds of ANY 3rd party board
> specifying anything other than "avr" or "sam" are very low. Why
> would they?

I think it is critical that we prevent things getting mislabeled. Of
course, library authors will make mistakes, but we should do our best to
provide guidance and documentation to prevent any errors (and/or make
them err on the side of caution). However, we should really try to make
sure that there is never a reason to _intentionally_ mislabel a library
or core, since once that happens, the entire system falls down...

Gr.

Matthijs

[Paul Stoffregen]

On 11/04/2013 03:09 AM, Matthijs Kooijman wrote:
> In essence, the problem thus is that the architectures= specification
> is too coarse.

Exactly.

> If youre teensy3 core would emulate _all_ of the AVR-specific APIs,

Sadly, I'm not that good.

> Summarizing, an architectures= property is not expressive enough. This
> could be ok if it causes not-supported boards to be marked as
> (potentially) supported, but your example shows that it can cause
> supported boards to be marked as not-supported, which is a real problem.

Yes, that's exactly my concern.

Trust me, one way or another, I *will* make things work as seamlessly
for end users as I can. You may be surprised by the ugliness of hacks
I'm willing to deploy, if necessary. Indeed I do have some rather ugly
workarounds planned. In a way, that scenario might not work out so
badly for me personally, since few (if any) of the other 3rd party board
makers are willing to do to such lengths.

But I would much rather collaborate early-on to shape a cleaner
solution, for everyone's benefit.

[Gmail]

I may be completely missing the point here but I have been thinking
about this problem and I feel I have a different approach that may be
helpful.
It looks like we are getting to a consensus that architectures= is to
course. I also have the impression that "board features" has
complexity/maintenance concerns .

My view is that all these proposals are from the ground up (like in the
hardware components being used on the board/shield) and try to
"guaranteed correctness".
Why not go from a top down approach? The end user buys "hardware
products" (as in boards, shields, breakout boards, chips) and the
library "guarantees support" for this set of "hardware products".

Practically: If the end user not only selects his Arduino board but also
his other "hardware products" used in a project. The "library provider"
can "guarantee support" for this collection of "hardware products"

To visualize it: In the GUI when selecting a library only the
"guaranteed libraries" could be shown and a sub-menu could contain the
"This is not guaranteed to work" libraries.

The properties file of the library could state the "hardware products"
boards, shields, breakout boards, chips it guarantees it works on. (this
is similar to the idea behind boards.txt)
These can be used to feed into the gui to allow to select the "hardware
products".

I think this could work perfectly for all "hardware products" delivered
by a provider who also delivers a library. This because he provides a
"hardware id". For others there may be a set of hardware id's.
Best regards
Jantje

[Matthijs Kooijman]

Hi Jantje,

> It looks like we are getting to a consensus that architectures= is
> to course. I also have the impression that "board features" has
> complexity/maintenance concerns .

Agreed.

> My view is that all these proposals are from the ground up (like in
> the hardware components being used on the board/shield) and try to
> "guaranteed correctness".
> Why not go from a top down approach? The end user buys "hardware
> products" (as in boards, shields, breakout boards, chips) and the
> library "guarantees support" for this set of "hardware products".

Ok, if I understand you correctly, you basically say two thing:
1. Don't try to predict support for given library/hardware combination,
but only specify what's explicitly supported (e.g., the hardware the
library author, which might be the hardware vendor, has tested).

2. Don't just include the main board as the "hardware", but also
include any shields and other hardware in the compatibility check.


As for 2., I feel that including other hardware will only make an
already complicated problem even more complicated. For declaring the
board you use there is already infrastructure, but something completely
new should be added for the add-on hardware, with all the associated
maintainance burden. Hence, I do not think that doing this will be worth
the trouble.

However, once (possible) conclusion you could draw from this is: If
specifying support for just the board is really only half of the
solution and the entire solution is too hard to implement, should we
perhaps not try at all?

As for 1., since library authors will be testing on a handful of boards
at best, usually just one, most libraries will just declare support for
one or two boards, even though they might work on nearly every
Arduino-compatible board.

If so, is declaring compatibility still useful?


In the dicussion, I think we've found that providing a proper solution
for dependencies has a lot of problems. In particular, problems are:
- A system that precisely specifies dependencies must be very
fine-grained and thus complex
- Libary authors want to write a library, not spend much time on
figuring out a complex system.
- Library authors will be making mistakes.
- A system that specifies only specific supported boards will be
useless in most cases where users have different boards.
- A perfect system should really include shields and other hardware as
well.

This is just from the top of my head, the problems we're running into.
Now, it would be totally awesome if we could have a library browser in
the IDE or on the website that allows the user to just enter his
hardware and then automatically downloads the library the user needs.
However, more and more I get the impression that implementing something
like that simply won't be possible. We could have something that works
in some cases, but I'm certain it is impossible to make it work in _all_
cases. Furthermore, I suspect that at best the system would simply be
unable to given any useful advice in _most_ cases and at worst it will
be giving _wrong_ advice in a significant number of cases.

If this is so, we might be better off simply not trying to specify any
hardware dependencies and leave it to the user to decide and to library
author to provide guidance in the library's description.

Note that I'm not convinced that we cannot figure something that will
work, but I haven't been able to think of anything that really works so
far...

Gr.

Matthijs

[avenue33]

I've given up.