1.5 Lib: Library layout

[Cristian Maglie]

avenue33 and Jan Bayens, maintainers of embedXcode and Arduino Eclipse plugin, expressed some concerns about the "arch" folder in the 1.5 library layout Just to be clear, we are talking about libraries created and contributed by users the ones that we put under SKETCHBOOK/libraries

Now, unless I made some errors on the recap, I want to hear the opinion of the others member of the list too:

- is the "arch" folder an "overengineering"?

- it may be useful, or is something that nobody is going to use?

Follows a recap of what the arch folder is, the PRO and the CONS already expressed. (warning: very long email)

The reasons behind the "arch" folder:

it was conceived to store architecture specific code ONLY, instead of writing a singe huge file where code is selected through #ifdefs.

For example, an hypothethical "probe" library written without the arch folder may looks like this:

Probe/src/probe.c

int probe() {
#ifdef ARDUINO_ARCH_AVR

return PORTA & 0x01; // AVR Only

#elif ARDUINO_ARCH_SAM
return REG_PORTA_DR & 0x01; // SAM Only
#else
#error The hardware is not supported
#endif

}

Instead using the arch folder the library author provides one file per architecture:

Probe/arch/avr/probe.c:

int probe() {
return PORTA & 0x01; // AVR Only

}

Probe/arch/sam/probe.c

int probe() {
return REG_PORTA_DR & 0x01; // SAM Only
}

IMHO the PRO of this approach are:

- Much more clear structure of the code

- Adding another platform is much easier (just add the arch/xxxx/probe.c file)

- Code that is not platform specific is common for every architecture can be placed in Probe/src, it may be updated and all architectures will benefit from the update

- The same structure is used by many real world projects:

https://github.com/mirrors/linux/tree/master (notice the arch folder)

http://stackoverflow.com/a/3627208/1655275

http://stackoverflow.com/a/14592359/1655275

- The linux kernel uses the same structure and it actually use make and the gnu-gcc toolchains and its still not clear to me why changing the Makefiles accordingly is so hard, but I understand that this may be my limit.

The CONS expressed by janjte are:

1) There are plenty of non architectural dependent libraries who do not

need to cope with the complexity multiple architectures bring.

2) The "average starting Arduino library builder" will get scared when

he sees an overly complex folder/file structure which will damage the

"Arduino simple and easy advantage".

3) I expect that most ide's won't have lots of trouble to implement it.

4) And 3 has to be implemented anyway during transition.

The CONS expressed by avenue33 are:

Last remark: the inner structure of the libraries should be friendly with standard C++. This implies understanding that Arduino is two things: an IDE and a framework.

The Arduino IDE performs many hidden jobs for compilation. It provides basic features and there are many other more powerful IDEs.

The Arduino framework should be compatible with the C++ tools —make— that can't provide the fine granularity of the hidden work performed by the Arduino IDE.

C

[Matthijs Kooijman]

Hey Christian,

> Now, unless I made some errors on the recap, I want to hear the opinion of the
> others member of the list too:
> - is the "arch" folder an "overengineering"?

To me, it seems like a useful feature. Having code littered with #ifdefs
gets messy real quick, so separating the code per architecture can
result in a nicer library layout.

However, do realize that you're making the architecture property of the
build a bit of a first-class citizen now. What I mean by this is that
there a lot of things that you might want to check for in #ifdefs in the
code: Architecture, board, endianness, presence of a timer0,
who-knows-what. Each of these will make the code a bit more messy, but
you're fixing this for architecture, but not for any other build
parameters.

Note that I'm not saying this is a bad thing - IMHO the architecture can
potentially influence so much of your code, that having arch subdirs is
probably a good compromise between library structure complexity and code
complexity.

> Instead using the arch folder the library author provides one file per
> architecture:
>
> Probe/arch/avr/probe.c:
>
> int probe() {
> return PORTA & 0x01; // AVR Only
> }
>
> Probe/arch/sam/probe.c
>
> int probe() {
> return REG_PORTA_DR & 0x01; // SAM Only
> }

Note that you could implement also implement this in a library without
explicit support for library subdirs in the IDE. In this case, you'd
just have a Probe/probe.c that looks like:

#ifdef ARDUINO_ARCH_AVR
#include "../arch/avr/probe.c"
#elif ARDUINO_ARCH_SAM
#include "../arch/sam/probe.c"

#else
#error The hardware is not supported
#endif

While writing this I realized that you need to #include .c files, which
isn't usually considered a good idea. Also, with this approach you lose
the feature of dropping in support for a specific architecture into an
existing library.

> - Much more clear structure of the code
> - Adding another platform is much easier (just add the arch/xxxx/probe.c file)
> - Code that is not platform specific is common for every architecture can be
> placed in Probe/src, it may be updated and all architectures will benefit from
> the update

Agreed!

> 1) There are plenty of non architectural dependent libraries who do not
> need to cope with the complexity multiple architectures bring.
> 2) The "average starting Arduino library builder" will get scared when
> he sees an overly complex folder/file structure which will damage the
> "Arduino simple and easy advantage".

Firstly, for "non architectural dependent libraries" (also referred to
as Vanilla libraries in another thread), which I take to mean libraries
that only contain plain C code and only use architecture independent
API's (digitalWrite for pin access, Wire library for I2C, etc.), there
wouldn't be any complexity. They can just implement everything in src/
and don't bother with the arch/ subdirectory. I'd expect most of the
libraries from "average starting Arduino library builders" to fall in
this category.

Secondly, libraries that only support a single architecture, could also leave out
the arch/ subdirectory and just put the arch-specific code inside src/.
The library.properties should of course note that only a single
architecture is supported in this case.

The downside of this approach is that it's harder to add support for a
second architecture, but in practice it's not always easy to predict
what code should go into arch/ when you still only support a single
architecture (especially for novice programmers), so postponing this
separation until a second architecture is supported probaly makes sense.

Thirdly, for library writers that do not like the arch subdirectories,
or libraries where there is only a very small amount of
architecture-specific code, using the #ifdef approach in code inside
src/ and omitting arch/ is still possible. The library.properties allows
to explicitely list the architectures supported in this way.

I'm not entirely sure if what I say above is allowed with the current
library specification, but looking at the spec I think it'll just work.
It might be useful to formalize the above approaches in the spec as
well, to explicitly make them allowed and supported.

> 3) I expect that most ide's won't have lots of trouble to implement it.
> 4) And 3 has to be implemented anyway during transition.

Reading jantje's original mail, it seems he was not expressing CONs per
se, but saying that supporting 1.0-style libraries should continue
forever, since the 1.0 format is easy to implement in other IDEs and the
official IDE needs to implement both during a transition period anyway.
However:

- Above, I suggested some ways which make a 1.5 library simpler and
more like a 1.0 library.
- Converting from 1.5 to 1.0 minimally requires moving some files
around and adding a library.properties file, which I think we can
expect for library authors.
- The 1.0 format is easy for other IDEs to implement, but other IDEs
should implement the 1.5 format anyway. Libraries will start use the
1.5 format, even when the official IDE infinitely keeps supporting
1.0 libraries.
- Supporting the 1.0 format infinitely will encourage people to keep
using it. I'd rather see they switch to the new format and at least
include a library.properties file. Dropping support for 1.0 libraries
in, say, Arduino 2.0 will help to get people to adopt the new format.

> Last remark: the inner structure of the libraries should be friendly with
> standard C++. This implies understanding that Arduino is two things: an IDE
> and a framework.
>
> The Arduino IDE performs many hidden jobs for compilation. It provides basic
> features and there are many other more powerful IDEs.
>
> The Arduino framework should be compatible with the C++ tools —make— that
> can't provide the fine granularity of the hidden work performed by the Arduino
> IDE.

I agree with this, but I don't think the current format is very
make-unfriendly. AFAICS, all that's needed is:
- Set an ARCH variable
- if arch/$ARCH/ exists, include the sources in that directory in the
build
- otherwise, if arch/default exists, include that in the build.
Additionally, when compiling the src/ folder, the include path should
contain the arch-specific (or default) folder and vice versa.

Optionally, it might need to parse library.properties to find out if the
current library format is supported. However, if it doesn't and just
compiles, it'll probably just get compilation failures, just like with
the current 1.0 format.

It'll also need to parse library.properties to find out the dependencies
of the library, but this should be a lot easier than with the 1.0 format
where it has to look for #includes in the source files instead.

I just also read the entire forum thread, which helps to understand the
source of various concerns.

Regarding IDE-friendlyness, the forum raised a point about the eclipse
plugin having troubles with the arch/ folders. However, IMHO the
specification is quite sane and the problems that emerge seem to be due
to limitations in eclips more than because of weirdness in the library
spec. Having said that, a solution with symlinks was suggested on the
forum and it isn't clear to me why that solution would _not_ work to me.


While reading the forum, I also noticed another important concern
Christian did not include in his summary:
- How does one add support for a new architecture or board to a core
library?
- How does one add support for a new architecture or board to a library
without cooperation of the original library author?

Regarding the latter point, I totally support Christian's vision that
library authors should cooperate to get a single official library that
supports as much architectures and boards as possible. Even though I
think some things could be improved in this area, we probably won't
reach this ideal world and people will continue to port libraries
without properly collaborating, and I guess we need to easily support
this somehow. "easily" here I guess means "without having to modify
library.properties".

The most obvious approach here is that a porter completely duplicates
the library and makes it work on his new architecture or board. An
end-user can just drop the library into his Sketchbook/libraries folder,
but only if there isn't already a version of the same library for a
different architecture (since they'd have the same name).

The current 1.5 library spec could be used to solve this, by dropping a
subdirectory into Sketchbook/libraries/Foo/arch/ for the new
architecture. However, this is isn't ideal because:
- It requires modifying library.properties as well (though this could
be remedied by automatically appending any architectures that are
inside arch/ to the list of supported architectures).
- Since the src/ and arch/$ARCH/ directories are compiled together,
there is a strong coupling between those, so somebody developing
support for a different architecture will need to target a specific
revision of the original library.
- This only works when porting to a different architecture. For
supporting a different board on the same architecture, this approach
is useless.
- Reading what Christian wrote, the arch/ subfolder is not actually
_intended_ for this usecase, but instead is meant to allow cleaner
multi-arch libraries.

In other words: The current 1.5 spec seems adequate for the "multiple
porters work together on a single library" usecase. However, the
"multiple porters independently work on the same library" usecase, is
not solved at all. Without thorougly thinking them through, here's a few
options that come to mind:
- Put libraries for different hardware in the
Sketchbook/hardware/*/libraries (or perhaps inside the core) folder.
I think this is somewhat supported already, but there's only a
limited amount of places to put these libraries, and spreading them
out for no real reason is not very elegant.
- Allow naming the directory a library is in differently from the
actual library. E.g., you could have:
Sketchbook/libraries/Foo/ -- For the regular avr/sam architectures
Sketchbook/libraries/Foo-pic32 -- For the pic32 architecture
Or perhaps even:
Sketchbook/libraries/Foo/ -- For the regular avr/sam architectures
Sketchbook/libraries/Pic32/Foo -- For the pic32 architecture
The latter might already be (partly?) supported I think. Important in
any case is that you can refer to the library using the name "Foo"
(which is easy if library.properties exists, than the actual dirname
might not be so important anymore).

In both cases, some sane way of specifying for which boards a specific
library can be used is needed (so the library can check each library
against the current board). However, this is of course a non-trivial
problem, which might be only solveable by involving the user to make the
final selection (and then storing that for the current sketch, or
better, board).


Apologies for my long post, I hope it might be enlightening in some
areas. As you probably noticed, I'm not sure what the best approach is
either and I've read so many things about this throughout the day that I
probably missed some possibly important bits (my head is a bit full of
it now ;-p), but I hope it's useful still.

Also, after reading the forum I found that a lot of the things I'm
pointing out in this mail were already pointed out there, but perhaps it
helps to re-state them in a possibly different context :-)

Gr.

Matthijs

[Bill Perry]

There are too many threads on this 1.5 library subject.
I have no idea which one to respond to as they are all overlapping.
Can't we delete some of these can go back to a single thread where
we can discuss the merits of the new library system and figure out
the desired wants/needs and issues to be solved before we start
jumping in and trying to solve things?
And why do we have to use GoogleGroups?
I'm guessing that the Arduino Forum gets better visibilty.

===============================================
A few comments on Matthijs, previous post.
Even with a "vanilla" library there are still potential issues that crop up.
The biggest one being due to AVRisms creeping into the code.
For example, PROGMEM.
This is a proprietary AVR hack to gcc. It does not exist in the other
processor gcc version tools. Some core providers go to great lengths to
try to provide compatibility with this but it is VERY difficult and in some
cases impossible to provide 100% compatibility.
I finally had to create my own "const" data types that get remapped depending on
the target processor to avoid all the AVR progmem issues
with c++ and to ensure compatibility across other platforms like arm and pic32.

And while I don't want to talk about specific implementations, one
thing I have seen floating around is the use of links or symlinks.
My concern with their use is how they work across OS platforms and how
they get saved and restored in backups or when copied or used
across network mount points.
While I have no doubt that I could make it all work in a *nix environment I'm
not so sure how all that works in the Windows or Mac environments or what
happens when things are shared between them, perhaps with network
mounts and during backups and restorations on those platforms.
Overall, I don't know that there are issues on Windows,
but having to rely on links and symlinks sure feels kind of kludgey.
--- bill

[avenue33]

Sorry guys, I'm lost: now there are 3 different threads!

Comments on Arduino 1.5 Specifications

1.5 Lib: Library layout (3)

Arduino IDE 1.5: Library Specification Discussion (7)

> Now, unless I made some errors on the recap, I want to hear the opinion of the 

> others member of the list too: 

> - is the "arch" folder an "overengineering"? 

Have a look at the UTFT library: it includes a limited number of #ifdef to manage different platforms and a long list of screens.

Actually, I also care about core and application libraries.

Why not to stick with plain .h and .cpp files and why add a .properties file a standard compiler is going to ignore?

If Arduino wants to depart from C++ and create its own dialect of C++, feel free!

—cross-posted with the other Arduino developer thread

[Paul Stoffregen]

I have 3 areas of concern: #1: impact on user feedback, #2:
compatibility, #3: effectiveness.

In this message, I'd like to talk about #2, compatibility.

Breaking library compatibility is bad. Very bad. End users suffer.

For example, consider the Time library. Months after Arduino 1.0's
release, Michael Margolis updated it with Arduino.h and other fixes to
allow it to compile. But several of the examples did not work, like the
NTP time server query, because of other changes like the Ethernet
library. There's a lengthy thread on the Arduino forum about the Time
library with dozens of unanswered requests for help with compatibility
issues. It was only 7 months ago (well over a year after Arduino 1.0's
release) that Michael and I worked on fully updating and retesting all
the Time library examples on Arduino 1.0.X and several different boards
(I personally tested on Uno, Due and Mega, as well as all Teensy
boards). 3rd party libraries often lag far behind Arduino, especially
for their less commonly used examples and features, and during that time
end users experience frustrating errors that they're powerless to fix.

Another problem is users have older versions of the software. A common
scenario involves learning about Arduino at a workshop or class. Then
it sits unused for months or years, until one day there's a project.
Maybe it's as simple as blinking LEDs inside a Halloween costume, but
maybe it involves something like a temperature sensor with a 3rd party
library. Usually such projects have time pressure, but users feel
empowered because Arduino seemed easy to learn in the class and they
still have the hardware in a box on a shelf and the software still
installed on their computer. So they use google, find some library that
does 90% of what they need. But it doesn't work. Maybe they have
Arduino 0022 and Arduino.h is not found? Whatever the cause, any
compatibility problem turns what was going to be a fun project into a
disappointment. Maybe they do more searching and download a new
version, or maybe they just build their costume without an Arduino? The
non-Arduino path is probably leaves a permanent impression that Arduino
seemed easy in the class, but was actually hard to use for a real project.

If 1.5.X introduces a new library format that's not compatible with
1.0.X, then these compatibility troubles Arduino users face today will
multiply. Today, many issues are "old vs new": a 2-way split between
pre-1.0 and post-1.0. But it could easily become a 3-way split, "very
old vs somewhat old vs new", pre-1.0 vs 1.0.X vs 1.5.X, which is far
more confusing. User success rates at resolving compatibility problems
will become much lower when the problems don't so neatly fall into an
old vs new paradigm.

It's easy to assign responsibility (and blame) to library authors, but
the end result is novice users who stand the most to benefit from
Arduino's simplicity are the ones of suffer. The cold hard reality is
many 3rd party Arduino libraries are not well maintained. But even in
the official libraries, I believe only recently have some pre-1.0
flush() functions finally been updated to the post-1.0 behavior, or at
least their pre-1.0 behavior removed.

Incompatible changes on a widely used platform like Arduino will
continue to cause end uper pain for a very long time.

Michael Shiloh said it very well earlier in this conversation:

>
> Earlier when teaching Arduino, almost all libraries seemed to be
> compatible with the current version of Arduino, and in general
> students experienced a high degree of satisfaction: Download, install
> (this was tricky sometimes, especially on Mac OS), and it just works.
>
> Lately, it seems very likely that libraries will not work. Sometimes
> names must be changed (no hyphens!), sometimes dependencies on other
> libraries that have changed must be researched and fixed.

If 1.5.X moves to a new library format that is incompatible with every
non-beta version of Arduino that's been in widespread use for the last 2
years, the change is likely to cause a lot of novice user pain and
suffering who try to use new format libraries on the
previously-installed old versions.

Another big concern is keeping 1.5.X compatible with old libraries meant
to 1.0.X. It's one thing to intend to maintain compatibility, but
another to actually do so reliably. My main concern is ALL the
libraries shipping with 1.5.X have converted to the new format. If 2 or
3 of the wide used libraries were permanently kept in the old format,
future changes that break compatibility would likely be discovered and
fixed sooner.

Arduino as a platform has already lost some of it's simplicity with the
expansion from 1, then 2, and now 5 different hardware plaforms, and 1.0
compatibility issues that STILL impact some users. Adding yet another
layer of compatibility issues seems like a risk to Arduino's main value
proposition.

As a 3rd party who makes products and software for the Arduino platform,
I want to see the platform continue to make things as simple as possible
for users. In recent years much of the original simplicity has been
lost. Please, I beg of you, carefully consider backwards and forwards
compatibility.

[Cristian Maglie]

In data mercoledì 30 ottobre 2013 11:17:19, Paul Stoffregen ha scritto:
> Breaking library compatibility is bad. Very bad. End users suffer.

got it, thanks Paul.

C

[David Mellis]

Paul, you're right that backwards compatibility is important, but so is the future. If we never change anything, it's hard to make progress. I think you need to make it possible for people to provide a good experience for users during the transition, but that it's okay to impose some burden on developers in order to get to something better. You're right, though, to point out that the current 1.5.x specification would make it infeasible to provide a single library download that would work on both 1.0.x and 1.5.x.

My suggestion is to modify the 1.5.x library format to be compatible with 1.0.x, but keep the addition of the arch/ folders, a sub-folder of which is automatically included in the compilation process by the IDE. That is, you could take an existing 1.0.x library and it would work as-is on 1.5.x (as long its architecture-independent or you're using an AVR board). If you want to modify the library to support multiple architectures, you could add bunch of #ifdefs (which I think will be very messy in the long-run) or you could use the arch/ folder. This latter option requires a transitional hack to allow the library to continue to work on Arduino 1.0.x: i.e. you'd need to use an #ifdef to selectively #include the AVR-specific files (in arch/avr) for ARDUINO < 150 (from within a file in the library's root folder). This is a hack, yes, but it would (I believe) allow someone to transition to the new, cleaner multiple-architecture support (w/ the arch/ folder) while still providing a single download for use on 1.0.x and 1.5.x. I think this is cleaner than requiring an #ifdef for every single supported architecture -- and, furthermore, it's only necessary for the transition from 1.0.x to 1.5.x, which, while it will probably be long, will not be infinite.

As an analogy, consider the change from WProgram.h to Arduino.h. Now, you might not have felt this was an important change -- but let's leave that aside for the moment. The point is that someone could modify a library to work with both by including the appropriate #ifdef. Yes, this is a hack and yes, it took a while for libraries to get updated. Still, I think it allowed us to make the change and to provide libraries that would work on both 0022 and 1.0.x in the meantime.  At this point, I see few enough people using 0022 that I don't feel the need to support it. Other people may have longer horizons for backwards compatibility, but, again, I don't believe in supporting old versions forever, because it makes progress far more difficult.

Now, again, you might not have thought the change from WProgram.h to Arduino.h was worth it -- but I think it shows that its possible to make a transition. (There are numerous analogies from the Apple world; I, for one, am glad that they're not still on PPC even if it means that I can't run Design By Numbers or Boxer anymore.)  I think support for an arch/ directory is worth it, given that people are now trying to support the Arduino language and software on a number of architectures and this should allow them to do it in a much cleaner way. 

You could imagine a similar transition for eliminating the specialness of the utility/ folder: start recursively compiling everything (except the arch/ folder, examples/ folder, and extras/ folder) in a library, and change #include "foo.h" to #include "utility/foo.h", which should (I believe) work on 1.0.x and in this potential 1.5.x format. As long as you want your library to work on 1.0.x as well as 1.5.x, you'd have to put everything in utility (that's not in the root) but if you only needed to support 1.5.x, you could include source in any folder. 

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+unsubscribe@arduino.cc.

[Matthijs Kooijman]

Hey folks,

> Paul, you're right that backwards compatibility is important, but so is the
> future. If we never change anything, it's hard to make progress. I think
> you need to make it possible for people to provide a good experience for
> users during the transition, but that it's okay to impose some burden on
> developers in order to get to something better. You're right, though, to
> point out that the current 1.5.x specification would make it infeasible to
> provide a single library download that would work on both 1.0.x and 1.5.x.

Agreed on both points!

> My suggestion is to modify the 1.5.x library format to be compatible with
> 1.0.x, but keep the addition of the arch/ folders, a sub-folder of which is
> automatically included in the compilation process by the IDE. That is, you
> could take an existing 1.0.x library and it would work as-is on 1.5.x (as
> long its architecture-independent or you're using an AVR board). If you
> want to modify the library to support multiple architectures, you could add
> bunch of #ifdefs (which I think will be very messy in the long-run) or you
> could use the arch/ folder. This latter option requires a transitional hack
> to allow the library to continue to work on Arduino 1.0.x: i.e. you'd need
> to use an #ifdef to selectively #include the AVR-specific files (in
> arch/avr) for ARDUINO < 150 (from within a file in the library's root
> folder). This is a hack, yes, but it would (I believe) allow someone to
> transition to the new, cleaner multiple-architecture support (w/ the arch/
> folder) while still providing a single download for use on 1.0.x and 1.5.x.
> I think this is cleaner than requiring an #ifdef for every single supported
> architecture -- and, furthermore, it's only necessary for the transition
> from 1.0.x to 1.5.x, which, while it will probably be long, will not be
> infinite.

This seems good, but I'd perhaps consider adding an "ARDUINO_LIB_15"
define (or perhaps something with a better name) in the IDE and use that
instead of reqlying on ARDUINO < 150. This way, you won't into trouble
later if you decide to backport the new library format to the 1.0.x
branch and 3rd party tools can explicitely indicate support for the
library format separate of supporting the Arduino 1.5 core framework.


Also, note that I think what you propose is already possible with the
current specification and implementation. If I have a Foo library, it
could look like:

Foo/library.properties
Foo/src/Foo.h
Foo/src/Foo.cpp
Foo/arch/avr/Optimized.cpp
Foo/arch/default/Default.cpp

Above is the "regular" 1.5-style library files. To enable compatibility,
you'd add:

Foo/Foo.h:
#include "src/Foo.h"

Foo/Foo.cpp:
#include "src/Foo.cpp"
#include "arch/avr/Optimized.cpp"

These files could also include any additional .h and .cpp files if they
would be there. Also, I'm assuming that the "1.0 compatible" version of
the 1.0 library only supports AVR. Taking into account 3rd party forks
(like Paul's teensy version) invalidates this, but I guess you could add
some #ifdefs to include the right files from arch/).


AFAIU, the current 1.5 implementation will ignore Foo/Foo.cpp (since it
only looks at Foo/src and Foo/arch, right?), so the ARDUINO < 150 check
suggested above isn't even needed. If it doesn't, I'd suggest we add
that to the IDE (ignore files outside of src/ and arch/), but of course
only if library.properties is present (so we can still support real
1.0-style libraries which don't have library.properties).

The above solution with includes is potentially a bit messy, but for any
library authors that are serious about compatibility, this shouldn't be
much of a burden?

Does this make sense?


Halfway through writing this, another option occured to me: We could
make the "1.5 library format" support multiple layouts. Then, inside
library.properties, you'd have either
layout = 1.0
or
layout = 1.5

Doing this, we can have _all_ libraries add library.properties now, and
library authors can choose themselves if they want maximum simplicity
and 1.0 compatibility on one hand, or a more flexible and clear layout
on the other hand. There is no barrier to adding a library.properties
file, so we'd get the advantages offered by that the fastest.

The advantages of the new layout will probably come slower, but since
this is mostly an advantage to the library authors themselves, not the
IDE writers, that isn't so bad. Once the 1.5 IDE version becomes more
standard, I'd expect more library authors to switch to the 1.5 layout as
well.

Thinking on this a bit more, it actually seems like a fairly elegant way
to do things. You can even do a very simple "If library.properties is
not present, assume layout = 1.0".

Gr.

Matthijs

[Paul Stoffregen]

On 10/30/2013 08:43 AM, David Mellis wrote:

Paul, you're right that backwards compatibility is important, but so is the future. If we never change anything, it's hard to make progress.

Progress and backwards compatibility are not mutually exclusive.  Achieving both is more difficult and involves slow transition, but sparing end users from pain is makes it worth the extra effort.

Now, again, you might not have thought the change from WProgram.h to Arduino.h was worth it -- but I think it shows that its possible to make a transition.

David, it's pretty incredible you'd consider this a good example.  Two years ago, I and others begged you to put a 1-line backwards compatible WProgram.h into Arduno 1.0.  You refused.  Countless Arduino users suffered.

You could have added code to detect use of legacy WProgram.h and print a gentle warning.  Instead, you rejected backwards compatibility, arguing it would impede the rate of adoption.

Much like the directory structure parts of this new library spec, it was a change that has little (if any) benefit for Arduino's end users.

[Bill Perry]

On 10/30/2013 10:43 AM, David Mellis wrote:
>
>
> As an analogy, consider the change from WProgram.h to Arduino.h. Now, you might not have felt this was an important change -- but let's leave that aside for the moment. The point is that someone could modify a library to work with both by including the
> appropriate #ifdef. Yes, this is a hack and yes, it took a while for libraries to get updated. Still, I think it allowed us to make the change and to provide libraries that would work on both 0022 and 1.0.x in the meantime. At this point, I see few
> enough people using 0022 that I don't feel the need to support it. Other people may have longer horizons for backwards compatibility, but, again, I don't believe in supporting old versions forever, because it makes progress far more difficult.
>
> Now, again, you might not have thought the change from WProgram.h to Arduino.h was worth it -- but I think it shows that its possible to make a transition. (There are numerous analogies from the Apple world; I, for one, am glad that they're not still
> on PPC even if it means that I can't run Design By Numbers or Boxer anymore.) I think support for an arch/ directory is worth it, given that people are now trying to support the Arduino language and software on a number of architectures and this
> should allow them to do it in a much cleaner way.

Ok, so this kind of talk makes me feel that the Arduino team simply does not understand
how to do backward compatibility or how to handle a transition.
The Arduno 1.0 release is a great example of how NOT to handle a transition in that
the decision was made to intentionally break 100% of all the 3rd party libraries with no transition period.

We are not talking about changes that have limited impact on 3rd party libraries.
We are talking about breaking 100% of the libraries that don't come with the IDE.
Furthermore, there were some simple options available that would have allowed most of the existing 3rd party
libraries to continue to work and yet the Arduino team still decided to move forward
with a strategy that breaks every single library that does not come with the IDE,

For example, the Aduino IDE could have continued to provide a WProgram.h
and there were also backward compatible ways ways to handle things like write() vs send() functions etc...
In fact if you look at the Teensy core, Paul has done many of these
simple things to ensure that most of the pre 1.x libraries can still
continue to work on his core.

What really concerns me about the comments above is
that it seems to totally miss the point of backward compatibility.
Consider this statement:

> The point is that someone could modify a library to work with both by including the appropriate #ifdef. Yes, this is a hack and yes, it took a while for libraries to get updated. Still, I think it allowed us to make the change and to provide libraries
> that would work on both 0022 and 1.0.x in the meantime.

To me, that isn't backward compatibility at all.
That pushes any sort of backward compatibility fully out to and on to the
3rd party library writer.
The Arduino supplied environment is making no attempt to provide
any backward compatibility.
To me backward compatibility is when pre-existing unmodified code continues work.

And also this statement:

> Now, again, you might not have thought the change from WProgram.h to Arduino.h was worth it -- but I think it shows that its possible to make a transition

While the change from WProgram.h to Arduino.h is a change or a "transition" to
something new, the way it was handled was not backward compatible to existing 3rd party
libraries and also did not provide for a transition period for existing 3rd party code.

That "transition" could have been handled in way that still allowed most pre-existing code
to continue to work while allowing new code to use the new interfaces.
To me THAT is how to handle a transition when you have a large existing code base
that would break without backward compatibility.


From my perspective, these kinds of statements and views
re-enforce my deep concerns over a new 1.5 library spec
since in my view, one of the most important things to ensure is pre 1.5 library
compatibility which requires handling backward compatibility by doing things in the
IDE vs the library itself, which is not the way it was handled in the 1.x transition.

--- bill

[David Mellis]

Guys, the point is not to argue about the past, but to figure out the future. Any comments on the actual suggestion in my message?

My suggestion is to modify the 1.5.x library format to be compatible with 1.0.x, but keep the addition of the arch/ folders, a sub-folder of which is automatically included in the compilation process by the IDE. That is, you could take an existing 1.0.x library and it would work as-is on 1.5.x (as long its architecture-independent or you're using an AVR board). If you want to modify the library to support multiple architectures, you could add bunch of #ifdefs (which I think will be very messy in the long-run) or you could use the arch/ folder. This latter option requires a transitional hack to allow the library to continue to work on Arduino 1.0.x: i.e. you'd need to use an #ifdef to selectively #include the AVR-specific files (in arch/avr) for ARDUINO < 150 (from within a file in the library's root folder). This is a hack, yes, but it would (I believe) allow someone to transition to the new, cleaner multiple-architecture support (w/ the arch/ folder) while still providing a single download for use on 1.0.x and 1.5.x. I think this is cleaner than requiring an #ifdef for every single supported architecture -- and, furthermore, it's only necessary for the transition from 1.0.x to 1.5.x, which, while it will probably be long, will not be infinite.

[Bill Perry]

On 10/30/2013 02:05 PM, David Mellis wrote:

Guys, the point is not to argue about the past, but to figure out the future. Any comments on the actual suggestion in my message?

My suggestion is to modify the 1.5.x library format to be compatible with 1.0.x, but keep the addition of the arch/ folders, a sub-folder of which is automatically included in the compilation process by the IDE. That is, you could take an existing 1.0.x library and it would work as-is on 1.5.x (as long its architecture-independent or you're using an AVR board). If you want to modify the library to support multiple architectures, you could add bunch of #ifdefs (which I think will be very messy in the long-run) or you could use the arch/ folder. This latter option requires a transitional hack to allow the library to continue to work on Arduino 1.0.x: i.e. you'd need to use an #ifdef to selectively #include the AVR-specific files (in arch/avr) for ARDUINO < 150 (from within a file in the library's root folder). This is a hack, yes, but it would (I believe) allow someone to transition to the new, cleaner multiple-architecture support (w/ the arch/ folder) while still providing a single download for use on 1.0.x and 1.5.x. I think this is cleaner than requiring an #ifdef for every single supported architecture -- and, furthermore, it's only necessary for the transition from 1.0.x to 1.5.x, which, while it will probably be long, will not be infinite.

I don't think we are arguing at all about the past,
the non Arduino Team folks all seem to be pretty much in agreement
that we don't want to see any new library format break backward
compatibility the way 1.0 did.

As far as comments on your 1.5x library format suggestion:
My comment to it and all the others so far
would be that talking about specific implementation details
before defining and agreeing on the overall purpose of the new library format/spec
is putting the cart before the horse.

Unless I've missed it, I have not seen anything about the overall
purpose, goals, and issues that need to be solved and
more importantly which issues are issues that don't need to be or won't be solved.
So I would still like to have the discussion for the purpose of
of this new library format so we can all get on the same page
with respect to which issues are going to be solved and which
are not.


i.e. I think there needs to be discussion that results in
some sort of written documentation that
defines what this new library format needs to do before
we start going down the path of specific implementations.
This is a very high level list of requirements, goals and desires
that does not get into specific implementation details.

This type of document is crucial to having on going technical discussions about
the implementation details.
It also allows new people to jump into the conversation
and see what the stated goals are.
It prevents having to constantly re-hash the same things over and
over again.
It also puts the requirements and goals right in front of everyone
so everyone is all on the same page and working for a common solution.



--- bill

[Matthijs Kooijman]

Hey Bill,

> Unless I've missed it, I have not seen anything about the overall
> purpose, goals, and issues that need to be solved and
> more importantly which issues are issues that don't need to be or won't be solved.

Christian just started a thread about exactly this topic (subject "1.5
Lib: Goals"). In case you missed it, it's here:

https://groups.google.com/a/arduino.cc/d/msg/developers/ku32KnasSm0/iOZowtFLVJcJ

Gr.

Matthijs

[Paul Stoffregen]

On 10/30/2013 12:05 PM, David Mellis wrote:

Any comments on the actual suggestion in my message?

My suggestion is to modify the 1.5.x library format to be compatible with 1.0.x, but keep the addition of the arch/ folders, a sub-folder of which is automatically included in the compilation process by the IDE. That is, you could take an existing 1.0.x library and it would work as-is on 1.5.x (as long its architecture-independent or you're using an AVR board). If you want to modify the library to support multiple architectures, you could add bunch of #ifdefs (which I think will be very messy in the long-run) or you could use the arch/ folder.

I do agree #ifdefs are messy.

But the arch folder is not a good solution.  In fact, I think it's likely worse than just staying with the ugly 1.0 structure and #ifdef.  Here's 3 reasons why:

1: Duplicate copies of code is as difficult, or perhaps even more difficult to maintain than #ifdefs within a single file.

2: Architecture is too coarse (eg, Bridge for Uno, Esplora for Mega, Wire on Adafruit Trinket, etc).  #ifdefs are still needed.  A system with multiple files *AND* #ifdefs is definitely more complicated than only #ifdef.

3: Libraries with only arch/avr creates disincentive for 3rd parties to publish non-avr arch boards/cores.  If even 1 major library has only arch/avr with code that works on Teensy3, I absolutely will mis-label my kinetis core as "avr" for compatibility.  A theoretically flexible system that has content hard-coded for only 1 option isn't flexible in practice.

This latter option requires a transitional hack to allow the library to continue to work on Arduino 1.0.x: i.e. you'd need to use an #ifdef to selectively #include the AVR-specific files (in arch/avr) for ARDUINO < 150 (from within a file in the library's root folder). This is a hack, yes, but it would (I believe) allow someone to transition to the new, cleaner multiple-architecture support (w/ the arch/ folder) while still providing a single download for use on 1.0.x and 1.5.x.

The "transitional hack" should include an Arduino 1.0.6 release with support for compiling arch/avr.

[Matthijs Kooijman]

Hey Paul,

> But the arch folder is not a good solution. In fact, I think it's
> likely worse than just staying with the ugly 1.0 structure and
> #ifdef. Here's 3 reasons why:
>
> 1: Duplicate copies of code is as difficult, or perhaps even more
> difficult to maintain than #ifdefs within a single file.
>

> 3: Libraries with only arch/avr creates disincentive for 3rd parties
> to publish non-avr arch boards/cores. If even 1 major library has
> only arch/avr with code that works on Teensy3, I absolutely will
> mis-label my kinetis core as "avr" for compatibility. A
> theoretically flexible system that has content hard-coded for only 1
> option isn't flexible in practice.

You're description of these two suggest to me that you might have
misunderstood the proposal? AFAIU, the idea is that, for example on avr,
all files in src/ _and_ those in arch/avr/ is compiled. So any code that
is common to all architectures is put inside src/. It is written in such
a way that it calls functions or methods that contain just
architecture-specific code, whose implementation is stored in arch/.

So there should not be duplication of code between different archs, and
libraries that put architecture independent code in arch/avr and no code
in src/ are just wrong.

If you did already understand it this way, then I think I'm not
understanding your point properly.

> 2: Architecture is too coarse (eg, Bridge for Uno, Esplora for Mega,
> Wire on Adafruit Trinket, etc). #ifdefs are still needed. A system
> with multiple files *AND* #ifdefs is definitely more complicated
> than only #ifdef.

I agree that just architecture is too coarse to make the ifdefs
disappear completely. However, the point Christian makes is that when
comparing code for different architectures, there will often be large
pieces of code that are totally different. Doing these using big #ifdefs
is of course possible, but makes the code less easy to read and
maintain.

However, differences between boards and cpus within the same
architecture are likely to be a lot smaller and better suited to solve
with #ifdefs. I'd expect that having separate files for each board
and/or cpu would _hurt_ readability and maintainability (while having
them separate per architecture helps instead of hurts).

Having said that, I haven't seen enough multi-arch libs to say with
certainty if the above consideration fully applies in practice, but this
is how I understand the argument for arch/ subdirs (which makes sense to
me).

> The "transitional hack" should include an Arduino 1.0.6 release with
> support for compiling arch/avr.

That is most definately a good idea. However, won't that be mostly
pointless for people stuck on older versions (because their hardware
vendor forked the IDE, for example?). Or would it be acceptable to
expect those vendors to at least backport this feature into their fork
or update to 1.0.6?

Gr.

Matthijs

[Jan Baeyens]

Due to remarks which imply I'm aggressive and I have little hope
remaining the Arduino team is willing to change anything; I have not
been responsive lately on this subject.
However I have been reading most of the mails because this is something
that I feel is very important, and I wanted to state that I'm in
agreement with Paul, Bill.

Cold shivers run over my back reading some comments from David.
The Arduino approach on 1.0 with Arduino.h and the refusal to create the
one lined wProgram.h is an incredibly stupid idea. The fact the Arduino
team still defends this make it very clear they don't often frequent the
Arduino forum. If they would they be ashamed.

Jantje

[Jan Baeyens]

Using Pauls words:

As a 3rd party who makes products and software for the Arduino platform,
I want to see the platform continue to make things as simple as possible
for users. In recent years much of the original simplicity has been
lost. Please, I beg of you, carefully consider backwards and forwards
compatibility.

Jantje

[Gmail]

Matthijs
Paul understood the proposal very well. I'm completely with paul here.
The src folder is kind of useless and the default folder is still a very
big mystery to me.
The problem is as follows:
Shared code is in src. But what if I have a architecture that needs a
slightly modified version of the shared code?
I can include the code in my arch folder but that won't work because the
original method is still in the src folder that is on the build path.
So I need to copy, rename and modify the method in my arch folder or
remove the method from the shared folder and copy it to all existing
arch folders.
This is a consequence of having both src as shared on the build and
include path.
Look at 1.5.4 in the ethernet and wifi library the utility folder could
be in src but it is not. Why? This is "Duplicate copies of code"

Cmaglie referenced the linux build which is the example this is build
upon. There are however 2 big differences and both are nagging me
1) the linux uses [arch]/[libname] arduino uses [libname]/[arch]
2) Linux only has 1 folder for the build path.

jantje

[avenue33]

On Wednesday, October 30, 2013 8:05:43 PM UTC+1, David A. Mellis wrote:

Guys, the point is not to argue about the past, but to figure out the future. Any comments on the actual suggestion in my message?

Nice twist! Bella piroetta!

Ok, so let's talk about the future.

• Does Arduino want to remain compatible with standard C++ compilers?
  

or

• Does Arduino want to launch its own dialect and reduce its usage and acceptance? 
  

How hard to explain that 

• an antiquated and limited IDE shouldn't jeopardise a great framework?
  
• an IDE shouldn't perform under-the-hood jobs normally devoted to the toolchain?
  

My point of view isn't technical but functional. 

Whatever the best technical solution, if it is difficult to use, it wouldn't be used.

[Paul Stoffregen]

On 10/30/2013 02:12 PM, Matthijs Kooijman wrote:
> You're description of these two suggest to me that you might have
> misunderstood the proposal? AFAIU, the idea is that, for example on avr,
> all files in src/ _and_ those in arch/avr/ is compiled. So any code that
> is common to all architectures is put inside src/. It is written in such
> a way that it calls functions or methods that contain just
> architecture-specific code, whose implementation is stored in arch/.
>
> So there should not be duplication of code between different archs, and
> libraries that put architecture independent code in arch/avr and no code
> in src/ are just wrong.

Please allow me to provide an example. VirtualWire is a 3rd party
library for cheap one-way wireless modules. It implements AC signal
encoding (and equal number of 1's and 0's), checksums and other nice
features. Here's the website:

http://www.airspayce.com/mikem/arduino/

Like most libraries with platform-specific code, I have built the
hardware (ok, I bought cheap RF modules from Seeed), ported it to all
Teensy boards and personally tested my port. I'm familiar with the code
in nearly all widely used libraries with architecture specific requirements.

VirtualWire is currently a single .cpp and .h file, with one extra
"crc16.h" header, plus keywords.txt, several examples, and numerous
documentation files. Version 1.20 has 720 lines in VirtualWire.cpp.
There are 4 sections with #ifdef. The main two are on lines 163-453 and
lines 628-717. There's no substantial platform specific code in
VirtualWire's headers.

The first section, lines 163-453, sets up a timer to generate
interrupts. I hope you'll look at the actual code, but here's a brief
description. There are 3 copies of vw_setup() for 4 architectures. The
Launchpad and Maple versions looks like they might be fairly easy to
split out to separate files. A small amount of platform independent
code is duplicated in all 3 copies. Teensy 3.0 is supported by a tiny
#elif section inside the AVR stuff, plus 13 lines of special
definitions. Splitting that into 2 architectures will not be nearly as
simple.

The other section, lines 628-717, is the interrupt routine. This is
absolutely a case of substantial duplication of code. For the 4
platforms this library supports, there are 2 copies of the code. I must
confess, I do not fully understand how all 4 use these two. For
VirtualWire to use this arch folder approach, the interrupt routine
which implements a fair amount of code with a state machine and logic to
transmit the bits would become duplicated 4 times instead of only 2, and
in 4 different files instead of together in the same file.

My point is this seemingly clean and beautiful scheme with plaform
indepent source and arch specific folders doesn't necesarily match up
nicely to the reality of actual libraries.

[Paul Stoffregen]

On 10/30/2013 02:12 PM, Matthijs Kooijman wrote:

> AFAIU, the idea is that, for example on avr,
> all files in src/ _and_ those in arch/avr/ is compiled. So any code that
> is common to all architectures is put inside src/. It is written in such
> a way that it calls functions or methods that contain just
> architecture-specific code, whose implementation is stored in arch/.

Please allow me to provide one more quick example: the EEPROM library
currently in Arduino 1.5.4.

This one seems so very simple. The AVR specific code is all in the .cpp
file in arch/avr and the platform independent EEPROM.h is placed in
src. It's hardly any code at all. What could possibly go wrong?

Unlike AVR where the eeprom controller is always ready to go, Kinetis
needs hardware initialization (omitting complex issues here). Because
EEPROM has never used a begin() function, my EEPROM.h adds a constructor
to the object to initialize the hardware and remain compatible with all
Arduino sketches that use the EEPROM library.

The point here is porting libraries isn't always as neat and clean as
you might originally imagine when coding for a single architecture,
where you make decisions about what to place in src vs arch/avr. Often
other architectures have different requirements that aren't apparent
from experience only with AVR.

These issues are solvable. My point is only that partitioning code into
src and arch/avr is not so simple. It's very easy while coding for a
single architecture to overlook subtle architecture dependencies and
assume code you place in src is architecture independent. In theory,
separating code this way seems clean and beautiful. In practice, things
tend to be messy.

[Paul Stoffregen]

I'd also like to make one more point about the VirtualWire library example.

VirtualWire would not need ANY platform specific code if Arduino
provided an IntervalTimer API. Here is Teensy 3.0's platform specific
code inside VirtualWire:

#elif defined(__arm__) && defined(CORE_TEENSY)
// on Teensy 3.0 (32 bit ARM), use an interval timer
IntervalTimer *t = new IntervalTimer();
t->begin(TIMER1_COMPA_vect, 125000.0 / (float)(speed));

Well, there's also a few #defines to redefine some AVR macros, so the
AVR interrupt TIMER1_COMPA_vect becomes an ordinary function. The
"speed" variable is computed by a bunch of other code in VirtualWire.

VirtualWire isn't the only library like this. DmxSimple and SoftPWM are
2 others I've ported, by replacing piles of platform specific timer code
with this very simple IntervalTimer API. I recall doing one from
Adafruit recently too with only IntervalTimer, but I don't have the code
in my libraries directory at the moment.


My third and final point for this evening is smart APIs can replace the
need for a lot of the platform specific code currently in many Arduino
libraries. If you want a clean and beautiful solution, improving APIs
is the path. Separating code into more folders only moves the ugly mess
around.

[Bill Perry]

On 10/30/2013 08:32 PM, Paul Stoffregen wrote:
>
> My third and final point for this evening is smart APIs can replace the need for a lot of the platform specific code currently in many Arduino libraries. If you want a clean and beautiful solution, improving APIs is the path. Separating code into
> more folders only moves the ugly mess around.
>

I totally concur with this.
As is often the case, attacking the problem from a lower level where the root cause is
can make things substantially simpler in the higher layers.

In my glcd library, if there were faster pin i/o and sub microsecond delays available
in the Arduino API, I would not need any board, chip, or architecture specific code.

However, occasionally there are tool set differences that create the need for specific code.
For example, I need to interface to the stdio routines in the libC code.
The AVR librC does some crazy things and the arm tools used with Teensy3 need
an extra header included that no other implementation needs.
There are also some libC headers in AVR libC that are not C++ safe
so those have to be wrapped in cplusplus C wrappers before they are included
but can't be done that way on other platforms.

Another very common area that ends up creating many issues
is const data. The AVR's harvard architecture makes this VERY messy
while none of the other processors have this issue.
So in this case, it is a AVR vs the rest of the world type of thing.
While several platforms attempt to provide backward compatibility,
often it is not fully backward compatible given the sloppiness and
inconsistency of the the AVR progmem implementation and its
issues when used in C++.
My solution at this point for my library is to never use proprietary types or anymore
and declare wrapper types of functions to hide all this.
This is another area where an improved Arduino API might be able to help out.
In my case rather than using PROGMEM or PSTR or PGM_P, etc...
I use private type declarations that can be re mapped as needed by the toolset.

In my library to keep the code portable, anything that is processor or toolset specific
is wrapped in macros that are remapped depending on the environment all the way
down to the specific board.
None of this happens in the .cpp source code, it all happens in the header files.
There are 3 header files where this happens, and for me, I don't think
breaking them up across architectures would make things easier,
I think it would be more difficult to maintain.
Also, I'm not sure how it would work in certain cases.

For example, how would a library architecture type based split handle a case of
Use this this for AVR,
Use this for everything else?

I can easily do that with an ifdef/else in a header file
but how could that work if there are directories for each "architecture"?


--- bill

[Todd Krein]

Gentle Folk,
First off, let me say that I don't have a dog in this fight, and I will happily admit that I don't understand all of the subtleties in the compiler directives and defines that result. I'll work libraries into whatever structure results.

I would like to toss out one more viewpoint/datapoint.

I've only used the Arduino IDE for boards based on Atmel hardware, so I can't speak to the difficulties inherent in the blasphemy of porting to radically different architectures. But even within AVR, there are at least five different hardware implementations of I2C. If you want to work at the highest and slowest level, you can probably narrow it down to two, but that still requires a knowledge of the AVR family that you're using. If you want to run at anything like full speed, you need to know exactly which device in which family is being used.

I'll apologize here for not remembering the relevant thread from months ago, but it would be very helpful if a user could add a #define in their code that would enable non-default behavior in a library. I recall that this played great havoc with the make files, and I don't recall where it ended up. As an example, there is significant overhead required if you want to enable multiple masters in the I2C bus. My guess is that 99.9% of the Arduino world doesn't require that, and so the optimizations in removing it are justified. For people who need it, however, it seems unfortunate if we have to make them use a different library, or modify the library source code, or, worst of all, figure out the required changes themselves when someone else has already done that.

So, please consider that when better minds than mine determine the results of this discussion.

Cheers,
Todd

[Matthijs Kooijman]

Hey Todd,

> I'll apologize here for not remembering the relevant thread from
> months ago, but it would be very helpful if a user could add a
> #define in their code that would enable non-default behavior in
> a library. I recall that this played great havoc with the make
> files, and I don't recall where it ended up. As an example,
> there is significant overhead required if you want to enable
> multiple masters in the I2C bus. My guess is that 99.9% of the
> Arduino world doesn't require that, and so the optimizations in
> removing it are justified. For people who need it, however, it
> seems unfortunate if we have to make them use a different
> library, or modify the library source code, or, worst of all,
> figure out the required changes themselves when someone else has
> already done that.

thanks for sharing your experience, that might be helpful.

However, I think the request you are making is essentially this one:
https://github.com/arduino/Arduino/issues/421

It looks to me that it is perfectly possible to consider that request
separate from (or perhaps on top of) the library layout proposal, so I'd
rather go into it right now, to keep this layout discussion at least a
little bit structured :-)

Gr.

Matthijs

[Matthijs Kooijman]

Hi Paul,

> This one seems so very simple. The AVR specific code is all in the
> .cpp file in arch/avr and the platform independent EEPROM.h is
> placed in src. It's hardly any code at all. What could possibly go
> wrong?
>
> Unlike AVR where the eeprom controller is always ready to go,
> Kinetis needs hardware initialization (omitting complex issues
> here). Because EEPROM has never used a begin() function, my
> EEPROM.h adds a constructor to the object to initialize the hardware
> and remain compatible with all Arduino sketches that use the EEPROM
> library.

I understand your point. However, I think the arch/ subdirectories are
not intended to allow "drop-in" support of a new architecture. Even
though they _look_ like that, like you point out it is hardly ever
possible, since the architecture-independent code in src/ has to provide
the right "hooks" (e.g., call the right functions) in the
architecture-dependent code for things to work.

Where, AFAIU, the usecase for arch/ directories lies, is when a single
author, or a group of collaborating authors, builds a multi-arch
library. The arch/ subdirectories are intended just to make the layout
of such a library clearer, nothing else (you make valid points about
wether they are fit for that purpose in your other mail, which I'll
respond to separately).

> The point here is porting libraries isn't always as neat and clean
> as you might originally imagine when coding for a single
> architecture, where you make decisions about what to place in src vs
> arch/avr. Often other architectures have different requirements
> that aren't apparent from experience only with AVR.
>
> These issues are solvable. My point is only that partitioning code
> into src and arch/avr is not so simple. It's very easy while coding
> for a single architecture to overlook subtle architecture
> dependencies and assume code you place in src is architecture
> independent. In theory, separating code this way seems clean and
> beautiful. In practice, things tend to be messy.

In an earlier mail, I already argued that when writing a library that
(at first) supports just a single architecture, it makes sense to put
_all_ code in src/. It's not until you add a second architecture that
you can really properly decide what should go where, so I think we are
in agreement about this :-)

Gr.

Matthijs

[Matthijs Kooijman]

Hi Paul,

thanks for providing a detailed example, it also helps me to get a bit
more concrete.

I'll respond to both of your mails at once here.

Having multiple definitions of the vw_setup() function looks like it
could gain from separating each into a separate arch-specific file.
Given that each of them contain a bit of arch-independent duplicate
code, I'd keep vw_setup() in the main file and make it call a new
vw_setup_arch() function which is defined in each arch-specific file.

Now, you say that splitting the AVR and Teensy code into two files
is not simple, but I don't see that. The code that is shared between
these two is:
- The pinMode / digitalWrite calls, which I suggested to leave in
architecture-independent code above
- The _timer_calc function, which looks pretty architecture-independent
to me (the only architecture-dependent bit in there would be the
prescalers[] array, but that could even be moved into an
arch-dependent header file perhaps.

All other code is different between Teensy and AVR, so could just be
moved into the arch-dependent files.

> >The other section, lines 628-717, is the interrupt routine. This
> >is absolutely a case of substantial duplication of code. For the
> >4 platforms this library supports, there are 2 copies of the code.
> >I must confess, I do not fully understand how all 4 use these two.
> >For VirtualWire to use this arch folder approach, the interrupt
> >routine which implements a fair amount of code with a state
> >machine and logic to transmit the bits would become duplicated 4
> >times instead of only 2, and in 4 different files instead of
> >together in the same file.

It looks like the actual handling of the ISR is identical across
architectures, the only difference is the way the ISR is declared /
attached.

This is already handled in a sane way for MSP and STM, where a single
function is called from attachCompare1Interrupt and called from the real
ISR. The AVR ISR duplicates all code, presumably because calling a
function from the ISR greatly increases code size and decreases
performance, because the compiler resorts to preserving a dozen
registers that it could otherwise just leave untouched.

The obvious solution for this is of course to mark the vw_Int_Handler()
function as always_inline to remove the duplication. However, this
doesn't work when the vw_Int_Handler and ISR(TIMER_VECTOR) functions are
defined in different compilation units, since than the compiler cannot
inline. So, that suggests that moving ISR declarations into the arch/
subfolders isn't efficiently possible in all cases, which further
reduces the utility of those arch/ subfolders. You could of course
use defines in an arch-dependent header file and then define either of
these:

arch/avr/arch.h:
#define TIMER_ISR ISR(TIMER_VECTOR)
arch/stm/arch.h:
#define TIMER_ISR void vw_Int_Handler()
arch/msp/arch.h:
#define TIMER_ISR interrupt(TIMER0_A0_VECTOR) Timer_A_int(void)

and then in the main file:

TIMER_ISR
{
... code here ...
}

which is somewhat better, but not quite ideal.

I've left out the teensy code now, since that now calls
TIMER1_COMPA_vect, which would probably require some changes at the call
site as well.

> >My point is this seemingly clean and beautiful scheme with plaform
> >indepent source and arch specific folders doesn't necesarily match
> >up nicely to the reality of actual libraries.

Library authors are free to _not_ use the arch/ subfolders if it doesn't
help to improve their particular library, of course.

> VirtualWire would not need ANY platform specific code if Arduino
> provided an IntervalTimer API. Here is Teensy 3.0's platform
> specific code inside VirtualWire:

> My third and final point for this evening is smart APIs can replace
> the need for a lot of the platform specific code currently in many
> Arduino libraries. If you want a clean and beautiful solution,
> improving APIs is the path. Separating code into more folders only
> moves the ugly mess around.

I (still) agree with you on this point. However, I do wonder if we could
improve the API (fast) enough to completely make the arch/ subfolders
useless. But, perhaps we could, yes.

Also, remember that introducing arch/ folders does not in any way
prevent us from improving the API. If the API becomes so perfect that
arch/ folders indeed become useless, people will simply not use them
(and yes, I realize that if we expect this to happen (soon), it might be
better to leave out the arch/ folders in favor of simplicity for 3rd
party tool vendors).





Summarizing, I think we agree that:
- Providing arch/ subfolders _could_ improve readability and
maintainability of _some_ multi-arch libraries
- If _some_ above is very small, it's not worth the hassle of
supporting arch/ subfolders.

I think we disagree on the size of _some_ above. You argue that there
will be very few, if any libraries that would benefit. Christian
obviously thinks (at least when he proposed the spec) that a lot of
libraries would benefit.

Arguments that indicate that some is small are:
- Improving APIs will reduce the amount of architecture-dependent
libraries.
- Some code is hard to split up over architectures, especially when
multiple archs (partly) share the same code (which is easy to express
with #ifdefs, but not with arch/ subfolders).
- Still needing #ifdefs for different boards within the architecture
reduces the clarity gain of arch/ subfolders.

Did I miss any?

Furthermore, it seems that the arch/ subfolders complicate matters for
3rd party tool vendors. This does not belong in the above list of
arguments (since it doesn't make the arch/ subfolders more or less
useful for any given library), but does indicate that we should only
introduce the arch/ subfolders if the gain outweighs the cost on 3rd
party tool adoption.

I used to also think that most libraries would benefit, but you've been
succesful at making me doubt. You haven't entirely convinced me yet, but
that might happen automatically if I give this a bit more thought :-)

Gr.

Matthijs

[Paul Stoffregen]

On 10/31/2013 02:57 AM, Matthijs Kooijman wrote:
> Summarizing, I think we agree that:
> - Providing arch/ subfolders _could_ improve readability and
> maintainability of _some_ multi-arch libraries
> - If _some_ above is very small, it's not worth the hassle of
> supporting arch/ subfolders.
>
> I think we disagree on the size of _some_ above. You argue that there
> will be very few, if any libraries that would benefit. Christian
> obviously thinks (at least when he proposed the spec) that a lot of
> libraries would benefit.

Yes. You are also correct, the structure will be nice for some libraries.

However, I believe where we differ is assessing the _value_ and _cost_
of this feature.

My position is: #1) incompatible change is a tremendous cost, and #2)
tidy internal structure *may* be nice (a debatable point), but the value
is still low because it doesn't directly affect end users, only some
library authors.

Obviously there is an opposing viewpoint, that this structure has a lot
of value, enough to outweigh the cost of incompatibility. Or perhaps a
belief that incompatibility doesn't matter? There is no words that will
convince me incompatibility is ok, but I am interested to hear why
people feel this change has so much value?

[Matt Jadud]

There are clearly two classes of user (at least)---developers as supporters/ad-hoc extenders of the Arduino API, and users of the Arduino products of those developers---which makes identification of "user" needs tricky. It is easy to shift discussion back-and-forth, because both are important.

That said, is there an "evolution" vs. "revolution" question in this process?

For example: Paul has suggested a concrete API (TimerInterval) that would solve a real problem. There are certainly others (a SPI API, etc.). Instead of focusing on where files are, and the relative merits of those changes, is it instead valuable to incrementally move towards a cleaner, more easily supported structure?

1. Identify build features that need to be added to the system (parameters, metadata) to support existing boards better. (Can be done now.)

2. Identify APIs that existing code should be updated to, so that a critical abstraction layer (eg. TimerInterval) could be introduced, integrated into the existing codebases by a given date, and then relied on in the future. This suggests the notion of a "developer API" and an "end-user API," whereas currently there is only the latter; the goal is for the end-user API to be built on top of the former, for maximum portability. (Can be worked on sequentially, or in batches, over time.)

3. Use this API to design a mechanism for declaring what a board can and cannot do ("board" and "architecture" being an odd Venn diagram of features, in many cases) --- because it becomes a declaration of what and how the API is supported. (The API needs to co-evolve with these declarations.) This helps end-users immensely. (A revisitation/improvement of #1, in light of #2.)

4. Continue discussion/implementation of the Arduino API across an ever-increasing (and capable) range of hardware. 

This lets us move to a point where a package system can be built (much later) where the package system knows what board I'm trying to use, and what libraries I can and cannot leverage for a particular board. The package manager ultimately knows that the "Imponente 2020" board can use the GlobalRangeWiFi and Teleportation libraries, where as the "Povero 2019" board cannot.

From my point of view, I'd love to be able to throw students at #2, working with developers to improve libraries along known tracks. They're great entry-point open source opportunities, easily chunked along features, and (I currently imagine) very future-proofing for the community.

Cheers,

M

 

[Matthijs Kooijman]

Hey Matt,

On Thu, Oct 31, 2013 at 08:18:26AM -0400, Matt Jadud wrote:
> There are clearly two classes of user (at least)---developers as
> supporters/ad-hoc extenders of the Arduino API, and users of the Arduino
> products of those developers---which makes identification of "user" needs
> tricky. It is easy to shift discussion back-and-forth, because both are
> important.

Previously, a distinction between "users" (who write sketches) and
"library authors" (who write libraries) has been made. Is this the same
distinction you are making, or is yours subtly different?

> Instead of focusing on where files are, and the relative merits of
> those changes, is it instead valuable to incrementally move towards a
> cleaner, more easily supported structure?

I'm not sure if your "Instead" here is warranted. One does not exclude
the other, and better APIs do not solve the same problems as a different
library format, at least in the broader sense, looking at all the goals
Christian gave in the "Goals" thread. In the narrower sense, the arch/
subfolder, which seems to have become the subject of this thread, is
more related to API improvements, but I still argue that they solve
different problems and might both have merits. I certainly agree that
improving APIs is a good idea!

> 1. Identify build features that need to be added to the system (parameters,
> metadata) to support existing boards better. (Can be done now.)

If you mean adding metadata to the libaries, this is one of the goals
specified by Christian.

> 2. Identify APIs that existing code should be updated to, so that a
> critical abstraction layer (eg. TimerInterval) could be introduced,
> integrated into the existing codebases by a given date, and then relied on
> in the future. This suggests the notion of a "developer API" and an
> "end-user API," whereas currently there is only the latter; the goal is for
> the end-user API to be built on top of the former, for maximum portability.
> (Can be worked on sequentially, or in batches, over time.)

I'm not entirely sure what you mean here, it sounds like a useful
thought track. It sounds like you mean to have a "developer API", which
is an API that various architecture / core / board specific pieces of
code offer. It is a generalization of all supported hardware, but it can
still be very low-level (e.g., perhaps defines with register names or
pin mappings, etc.).

On top of this, you'd build the "end user API", that provides a more
usable and higher level API. The code for this is hardware-independent
and uses the "developer API".

Is this a bit what you meant? Or did you perhaps mean that the
"developer API" is used by libraries, whereas the "end-user API" is used
by sketches?

Also, you talk about "integrating by a given date", which suggests you
have an idea about a possible workflow, but I can't quite figure out
what you mean exactly.

> 3. Use this API to design a mechanism for declaring what a board can and
> cannot do ("board" and "architecture" being an odd Venn diagram of
> features, in many cases) --- because it becomes a declaration of what and
> how the API is supported. (The API needs to co-evolve with these
> declarations.) This helps end-users immensely. (A revisitation/improvement
> of #1, in light of #2.)

With "Use this API", I assume you mean "Express the declarations in
terms of support for this API" and not "Call functions in this API"?

This sounds useful, but I'm not so sure if one can find the proper
generalisations to make this work (without being overly pessimistic or
optimistic about which hardware is and isn't supported). I think it's
easy for any generalization to have shortcomings in "predicting" wether
a specific board will work.

I previously made some related observations here:
https://groups.google.com/a/arduino.cc/d/msg/developers/m2XEF4-uFxI/64B7JbwWKS8J

Gr.

Matthijs

[Cristian Maglie]

In data giovedì 31 ottobre 2013 01:56:53, Paul Stoffregen ha scritto:

> Please allow me to provide an example. VirtualWire is a 3rd party

These are a great test bench to check how good or bad the library specification works. Have you some other real world examples of multi-architecture libraries?

> My point is this seemingly clean and beautiful scheme with plaform

> indepent source and arch specific folders doesn't necesarily match up

> nicely to the reality of actual libraries.

Ok let's go and try it: I converted the library to the current 1.5 specification, here the process I followed:

1) first commit of the original library:

https://github.com/cmaglie/VirtualWire/tree/bc92ed3645dee3dca2b70eff9621138a87adbab6

2) I moved source files to follow 1.5 layout, in particular I moved the sources into the "src" folder and drop a properties files. (I know that probably we are going to remove the src folder to ensure forward compatibility with 1.0, but this is another topic ok? I'm just trying it :-))

Inside the library.properties I asserted that the library works with the architectures: "avr", "kinetis", "msp" and "stm32"

changeset:

https://github.com/cmaglie/VirtualWire/commit/459141f589dbe171bbb9da4da67bbdcc3c036397

code:

https://github.com/cmaglie/VirtualWire/tree/459141f589dbe171bbb9da4da67bbdcc3c036397

3) this is the most interesting part:

once everything is in place I moved all the arch dependent pieces into their respective arch folder.

If you notice every architecture is rounded by #ifdefs, something like:

#if defined(ARCHITECTURE_XXX_GUARD)

do stuff with XXX Architecture

#elif defined(ARCHITECTURE_YYY_GUARD)

do stuff with YYY Architecture

#else

....

basically I mentally followed what the C++ preprocessor would do and COPIED the corresponding piece of code through the various arch/xxx folders.

This allows me to remove all the architecture dependent code from src/.

Interesting also the Teensy case: it seems that Teensy use some #defines machinery to make the same ISR used by AVR to work without modifications. This lead to some partial code replication in the arch/kinetis folder that was not present in the original library.

changeset:

https://github.com/cmaglie/VirtualWire/commit/0b2e2c9c8c81dbd67b98a8cdf947e38b7e61afb8

code:

https://github.com/cmaglie/VirtualWire/tree/0b2e2c9c8c81dbd67b98a8cdf947e38b7e61afb8

4) Once that the various arch/xxx were in place I noticed that they share the same interrupt service routine. So the most reasonable thing to do is to refactor the subroutine: remove it from all architectures, put a single copy of it into the main src/ folder, and call it from the interrupt handler.

Someone may argue that this lead to inefficiency, but the compiler seems to be able to inline the call, at least on the AVR I see no code increase after the refactoring.

changeset:

https://github.com/cmaglie/VirtualWire/commit/ed144a173197a10f3d9536d6948df9c2315eb30b

code:

https://github.com/cmaglie/VirtualWire/tree/ed144a173197a10f3d9536d6948df9c2315eb30b

It doesn't took more than an hour, and here the final result:

https://github.com/cmaglie/VirtualWire

I tried to compile it on avr and it works. I haven't tried with other plaftforms since no one has yet their core ported to the 1.5 format (or I'm wrong?).

Now some observations:

- IMHO the library had an enormous benefit from this refactoring. Now its crystal clear (at least to me) how the architecture-specific stuff interact with the vanilla-core of the library. Its clear also that the only hardware specific part is the Timer setup and the ISR, while before it was absolutely not clear, and Paul had to explain in detail some of the library internals sometimes pointing directly to the code with line numbers.

- If I want to add support for another platform, say, sam or pic32, what should do I do? With the original library I should read through the code, understand where to hook, add my patch by adding whole new sections of #ifdefs, and submit the patch to the upstream authors. This is feasible yes, but simply dropping a file in arch/sam or arch/pic32 seems much more feasible to me, and requires basically no effort from the upstream author to accept the patch, because it doesn't interfere in any way with the other platforms.

- The original library has the interrupt handler replicated 3 times inside the big .cpp file. This means that every change to ISR should be propagated to *all* the 3 copies. The upstream authors failed to notice that the ISR can be declared as a vanilla function to be inlined inside the real ISR. This refactoring was pretty obvious to me during the step 4) of my porting because I wasn't distracted by all the #ifdefs crap. I want to recall the observation made by Matthijs Kooijman, that is very sensible here:

>libraries that only support a single architecture, could also leave out

>the arch/ subdirectory and just put the arch-specific code inside src/.

>The library.properties should of course note that only a single

>architecture is supported in this case.

>The downside of this approach is that it's harder to add support for a

>second architecture, but in practice it's not always easy to predict

>what code should go into arch/ when you still only support a single

>architecture (especially for novice programmers), so postponing this

>separation until a second architecture is supported probaly makes sense.

- Paul, I see that you setup the AVR-ISR-emulation to avoid the code duplication above but, if my refactoring is applied to the original library (even if it keep the 1.0 format), then you can optimize the Teensy version by removing the ugly AVR emulation and declare the interrupt handler directly.

- Another pro is that if I'm an expert of an architecture, I can safely ignore all the others. They are out of the way when I look at the core of the library, and doesn't distract me from my goal. I have no idea how the Energia or STM32 code works and it's nice to be able to ignore it.

- Let's look at the STM32 arch file (the Teensy one still uses some hardware dependent parts):

https://github.com/cmaglie/VirtualWire/blob/master/arch/stm32/VirtualWireInterrupts.h

we can see that by providing an HardwareTimer class we can further abstract the library and remove the architecture dependent parts at all, transforming the library into a "vanilla" library. This make me think that Paul statement about improving and extending the current low level APIs / Variants support is even more true (but we already know that, right? :)). I see also that all the complexity of the multi architecture timers will be transferred from the VirtualWrite library to the hypotetical HardwareTimer library, where the arch/ folder may still have sense.

C

[Cristian Maglie]

In data giovedì 31 ottobre 2013 17:39:33, Cristian Maglie ha scritto:
> These are a great test bench to check how good or bad the library
> specification works. Have you some other real world examples of
> multi-architecture libraries?

as a side note: this is just to provide some more elements to evaluate if the
"arch" folder is worth it or not. No more no less.

C

[Matt Jadud]

I'm always afraid of woodshedding in these kinds of conversations, so I'm trying to be brief, in the event that I'm adding no value.

On Thu, Oct 31, 2013 at 9:14 AM, Matthijs Kooijman <matt...@stdin.nl> wrote:

Previously, a distinction between "users" (who write sketches) and
"library authors" (who write libraries) has been made. Is this the same
distinction you are making, or is yours subtly different?

I believe I'm suggesting nothing radical: there are people using the Arduino, and people working to support hardware abstractions portably across multiple families/platforms. They're people with different needs. I like a lot of the things Paul has been saying (as well as others), and I'm just restating the obvious: breaking library compatibility can hurt the inexperienced user, and sometimes even the library maintainer/"developer."

 

> Instead of focusing on where files are, and the relative merits of
> those changes, is it instead valuable to incrementally move towards a
> cleaner, more easily supported structure? 

different problems and might both have merits. I certainly agree that

improving APIs is a good idea!

I'm suggesting that a challenging end goal (better APIs, better layout, etc.), with reasonable incremental changes that the community can keep up with (and even, possibly, grow the "developer" community in the process through hackathons and other outreach events to achieve those evolutionary goals) are good. So, again, nothing revolutionary/radical in notion, and no unnecessary "instead" or exclusions of process implied.

 

thought track. It sounds like you mean to have a "developer API", which
is an API that various architecture / core / board specific pieces of
code offer. It is a generalization of all supported hardware, but it can
still be very low-level (e.g., perhaps defines with register names or
pin mappings, etc.).

On top of this, you'd build the "end user API", that provides a more
usable and higher level API. The code for this is hardware-independent
and uses the "developer API".

Anyone can always work at any level. However, the notion that "digitalWrite()" is an "end-user" API makes sense, while "Stream" might be a developer abstraction. The end-user can always reach to the lower level (thus growing into the "developer" space over time), but that notion of glorious, fun, easy simplicity with the Arduino is where the "end-user API" lives. The "developer API" makes it possible to support more boards with more cool features, so the beginner can go from one board to another in a family (Uno to Due to Yun), or out-of-familly (Uno to Teensy).

Most of my time with the Arduino has been spent supporting a completely different language on the Arduino, and some of these ideas were integral to easing our own work.

> Also, you talk about "integrating by a given date", which suggests you
have an idea about a possible workflow, but I can't quite figure out
what you mean exactly.

If the community says "TimerInterval" is a great place to start (for example), then it is worth saying "can we do porting work in the core and in X,Y and Z libraries for a March release?" However, the community might not work that way. But, yes, evolution to me implies some kind of workflow.

Again, I'm trying to avoid woodshedding, but it seems like both a technical- and process-level discussion is happening, and I was reflecting it back as I saw things, for better or worse.

Cheers,

Matt

[Todd Krein]

Matthijs,
Yup, that nails it, exactly.

Now, just need an implementation. :-}


Thanks!

[Jan Baeyens]

Cristian
Now just imagine you have to do the same thing but someone else owns the src folder (like me 8-)  ).
I think your proposal is not good as someone else already did some porting where your proposal was not needed.

Depending on your persistence a new library will be created or we will get a very long mail thread; frustrations and no progress.
This is real world; you on your own and owner of all code is not real world.
Can you imagine sending a pull request to me?

Jantje

Also note that this example proves the drop a library for a arch will not work.
Also note

--
You received this message because you are subscribed to the Google Groups "Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Paul Stoffregen]

On 10/31/2013 09:39 AM, Cristian Maglie wrote:

These are a great test bench to check how good or bad the library specification works. Have you some other real world examples of multi-architecture libraries?

Sure, here's some more....

OneWire
IRremote
FastSPI_LED
Encoder
Adafruit_NeoPixel
Adafruit_CC3000   --- candidate for digitalPinToInterrupt  :-)

[Bill Perry]

-

I guess from my perspective of how it affects my openGLCD project,
I'd say given the way I have macro-ized everything to hide the tools, chip & board differences
from all the .cpp code,
an "architecture" level split really doesn't help much if at all.
My library has already been re-factored such that if it detects certain environments
it works better, but for the rest, it falls back to stock Arduino core routines to allow
it to limp along but still be usable.

To use a library structure that distributes out the files
I think it would be a bit more more work and harder to maintain and yet still require
ifdefs within the given architecture specific header files to deal with the
chip/board/variance differences.

One advantage of using ifdefs is that when there is an update/change,
there is one place to look and one file to fix vs having to fix it in all
the separate files.

So and the end of the day, I have to ask what would I be getting for my extra
work? And how will it improve the end user experience?

I think the answer to those questions goes back to my desire to have the
requirements/goals discussion first before we talk specifics
to make sure there is a clear understanding of what the new library format
must do and how it affects the end use experience when using the IDE.

-- bill

[Paul Stoffregen]

On 10/31/2013 02:57 PM, Bill Perry wrote:

I guess from my perspective of how it affects my openGLCD project,

Has there been a public openGLCD release yet?  Your approach involves far more sophisticated use of the C preprocessor than I've seen in any other Arduino library.

If your code openGLCD is available, I hope Cristian will take a look.

[Paul Stoffregen]

These 6 are examples where the author(s) went to some effort to make a single download work on multiple architectures.

Perhaps a better "test case" might be libraries that forked and didn't manage to consolidate into a single cross platform version.  One good example that comes to mind is the Open Music Labs Codec Shield, which has separate copies for Arduino & Maple.

http://www.openmusiclabs.com/projects/codec-shield/
http://wiki.openmusiclabs.com/wiki/AudioCodecShield

[Matthew Ford]

>>Obviously there is an opposing viewpoint, that this structure has a
lot of value, enough to outweigh the cost of incompatibility. Or
perhaps a belief that incompatibility doesn't >>matter? There is no
words that will convince me incompatibility is ok, but I am interested
to hear why people feel this change has so much value?

It is not that incompatibility is not important, just that is can be
approached via say a translation program to move the files, change
references etc
Not to say that the translation program would be trivial, but that it
could be used to provide a migration path to the new (better) layout

So lets first focus on what would be good going forward and then see how
we can migrate to there.
If the migration proves impractical then we can go back and re-examine
the goals

matthew

[Bill Perry]

On 10/31/2013 07:18 PM, Matthew Ford wrote:
> >>Obviously there is an opposing viewpoint, that this structure has a lot of value, enough to outweigh the cost of incompatibility. Or perhaps a belief that incompatibility doesn't >>matter? There is no words that will convince me incompatibility is
> ok, but I am interested to hear why people feel this change has so much value?
>
> It is not that incompatibility is not important, just that is can be approached via say a translation program to move the files, change references etc
> Not to say that the translation program would be trivial, but that it could be used to provide a migration path to the new (better) layout

Having done and been involved with many cross platform projects over a period of more than 3 decades,
I am not able to visualize a way to do that given that there are so many different ways
cross platform compatibility can be implemented.

In my view,
one thing that kind of comes close to that ideal of programatically doing a "translation"
would be to maintain backward compatibility by allowing existing 1.0 libraries to continue to work
"as is" and then extending the library format to enhance it for new features/capabilities
for the benefit of build systems and/or the end user GUI experience.

Then the question/argument comes down to, is the new extended library format
still compatible with older IDEs that have no knowledge of these extensions?

The ideal answer to that question would be yes.


--- bill

[Bill Perry]

While I still believe that most conversation about implementation details before
defining or agreeing to requirements is a bit futile, I'm contributing to this conversation
in hopes that it can be of educational value for the "goals"/"requirements"
discussion.

I have opened up the openGLCD repository to everyone.
Here is the link to the project:
https://bitbucket.org/bperrybap/openglcd

Keep in mind that the project is just now coming up and
is still what I call an alpha state.
Yes I've tested the code on many platforms, so it should
work for those want to actually try it out.

A bit of background.
This project creates a graphical lcd library, that is a fork of the glcd library (GLCDv3) which was a fork
of the ks0108 Arduino library.
My goal for the openGLCD project is to create a GPL v3 glcd library and toolset that is very portable and is
not tied to any given environment.
Another priority is high performance.
So while it currently creates an "Arduino" library what works with Arduino sketches, it was/is
not designed to be an "Arduino" library.
i.e. I'd like it to be portable enough
to be capable of being used for users that don't use any Arduino s/w or Arduino tools.
As such, much effort was taken to wrap all the low level i/o functions and resource
allocations with wrappers rather than use Arduino APIs directly in the main line code.
This is particularly helpful for boosting performance as the wrappers can auto detect
the specific environment and use environment specific optimizations.
When those optimizations are not available, the wrappers will automatically
fall back to less specific routines which under Arduino can mean dropping
all the way back to using the Arduino Core API functions.

Because of the AVR architecture, and the current Arduino digital i/o API routine design,
the only way to get high performance i/o on the AVR is to do raw port i/o on the AVR.
openGLCD has very sophisticated wrapper macros can automatically take arduino pin numbers
(or even AVR PORT/BIT values) and do not only individual raw bit/pin i/o but multi bit i/o.
This allows 8 bit data updates to the output pins to be converted into a single AVR instruction
if conditions allow it, i.e.
selected pins are all in the same register and are in the proper bit order.
If byte mode is not possible, nibble mode might be used, and then finally dropping back down to
individual bit/pin i/o.
This multi bit i/o allows an 8 bit register update to occur in 62.5ns vs around 40us if using
digitalWrite() for each of the 8 pins. A performance gain of over 600X
In real life glcd display updates for a typical mix of pixel updates,
the measured gains are in the 15 to 20 times gains.

In order to make all this "magic" work, the library doesn't use constructors for pin initialization.
The pins are configured based on configuration files that are external to the sketch.
I'm currently working on way to provide a way to move pin configuration back into to sketch,
for a slight performance penalty.
It will still allow raw port i/o for speed but in order to do it requires adding a single call/return
to each wrapper rather than doing it all inline.

If taking a look at the code, all the wrappers are in the include directory.
The main wrapper is glcd_io.h
glcd_io.h uses glcd_arduino_io.h to determine the specific Arduino board/chip environment
and pick up some of the pin mapping macros.
avrio.h performs all the AVR raw i/o mapping including the multi bit i/o.
glcd_delay.h figures out how to do sub microsecond delays for the given environment.

Obviously, my library is very different from the typical Arduino library.

My concern for any of these new split architecture library proposals I've seen up
to this point is that they are not capable of fully solving or eliminating having to use many ifdefs
because they are not capable of supporting differences all the way down to the
specific board or variant file since my belief is that libraries that tend to step
outside of the Arduino core i/o are often doing things at the chip/board level
rather than at the architecture/core level.

My fear is that there will be a system that can work and work very well for very simple libraries
but not very well for the more sophisticated libraries that need to share code
across different environments.
Do you end up with something that is more "different" than actually "better"?
And so my belief at this point is that there may be simpler solutions that
offer better "bang for the buck" while still solving
many of the bigger issues for a better user experience while
still assisting build systems, 3rd party tools developers, and library writers.

In other words, I believe that is likely to be possible to create something much simpler
that is easier to implement that can be used to enhance what is already in existence rather than
try to come up with some new all encompassing mega solution that tries
to solve everything.

But then that brings me right back to my desire to have the
"goals"/"requirements" conversation before talking about
specific solutions.
I'm still not understanding the desire to talk specific implementations
before there is even general agreement on which issues need to be solved
and which issues can be left unsolved.

--- bill

--
You received this message because you are subscribed to a topic in the Google Groups "Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/a/arduino.cc/d/topic/developers/4Jk4P0rsWPc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to developers+...@arduino.cc.

[Jan Baeyens]

Bill
I not fully understanding what you are saying here. You do look like a library builder to me. And so does Paul.
Both you and Paul have uttered concerns with the new library structure.
I can't recall 1 single library builder who has spoken in favor of this spec.
As this is supposed to make the life of library creators easier http://forum.arduino.cc//index.php?topic=165827.msg1429696#msg1429696

Can the library creators who think this is a good specification please stand up?

Jantje

You received this message because you are subscribed to the Google Groups "Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Bill Perry]

On 11/02/2013 02:34 PM, Jan Baeyens wrote:

Bill
I not fully understanding what you are saying here.

My summary is that without any defined requirements how can we evaluate implementations?
And depending on the final requirements, there may be some very simple solutions,
particularly if it doesn't have to solve certain types of issues.

You do look like a library builder to me. And so does Paul.
Both you and Paul have uttered concerns with the new library structure.

While I've done my share of multi platform library development and library porting,
of all the people around the Arduino world, I don't think there is another person
that has more experience dealing with porting and maintaining Arduino libraries,
3rd party tools and IDE features/issues across multiple platforms and architectures
than Paul - more so than even the Arduino team itself.
Just take a look at number of libraries he has taken in, it is close to 50.
Then there is all the enhancements he has done to make his cores better than the standard
Arduino core and make them more backward compatible with pre 1.x libraries.
He has had to create integration tools to install his core and patch the standard Arduino release
to make things work.

He makes his own boards and supports it all.
He is 1 guy.
There simply is no other person that has that level of experience.
So in my mind, his experience should not be ignored and should
be a big driver in defining the requirements for any new library format/specification.

I can't recall 1 single library builder who has spoken in favor of this spec.

Part of that may be that the audience may be small.
I think some people may be missing out on the discussions
by keeping the technical conversations here on the developers list
vs over on the forum.

As this is supposed to make the life of library creators easier http://forum.arduino.cc//index.php?topic=165827.msg1429696#msg1429696

Can the library creators who think this is a good specification please stand up?

Again, I'll come back to, there needs to a set of documented requirements in place
before any one could begin to truly evaluate any sort of spec or implementation.
My view is that a true "spec" must contain requirements not just implementation details.
It is the requirements that define the implementation details and that provide
a means to test an implementation for correctness and completeness.

For example, if the requirements specifically say that problem X is not to be solved,
then I can't say the implementation is poor because it doesn't solve problem X.

However, if there were requirements that X, Y, and Z must be solved/handled,
and the implementation only accommodates X, then I'd say it is a poor implementation.

In the absence of any requirements, I find it impossible to properly evaluate
any implementation since it means I have to guess as to what the requirements are and
the guesses I make are probably not going to be the same as someone else.

And this is why I keep harping on doing the requirements first.

--- bill

[Paul Stoffregen]

Thanks for the kind words Bill.  :-)

I've already written too many lengthy messages in this conversation.  It's hard to imagine more will help.  I'll be listening and available to answer specific questions.

I do intend to propose APIs to solve cross-platform compatibility issues, as time permits....

[Peter Feerick]

I would like to second Bill's comments re: Paul and experience. As a user of his products, I can say the commitment and support he provides is excellent. (However, that is off topic.) As Paul is a developer of a multi-architecture suite of libraries, he is member of the target audience. And luckily, he is committed to ensuring that the Arduino platform gets better as a whole, not just to continue working on his suite of libraries. This rare creature should be treasured less it ups roots and goes elsewhere ;)

-Pete

On Sun, Nov 3, 2013 at 6:29 AM, Bill Perry <bper...@gmail.com> wrote:

his experience should not be ignored and should
be a big driver in defining the requirements for any new library format/specification.

Peter Feerick
BIT, BLDes CQU

0413 431 284

174 Cummins Rd
Bundaberg Q 4670

peter....@gmail.com
peter....@cqumail.com

[avenue33]

I've given up.