crosstool-ng vs buildroot and the path to a NuttX toolchain

[ant...@vergeaero.com]

I am working on creating an improved NuttX toolchain that would make building software for NuttX easier.  What the nuttx-buildroot accomplishes, is the first step in what I'd like to accomplish. However, I'm conflicted about whether I'd like to start my work from nuttx-buildroot, or model the nuttx-buildroot in crosstool-ng, and then continue from there.  For those unfamiliar, NuttX is often built with a toolchain such as arm-none-eabi, where newlib is built in.  This can cause issues that conflict with NuttX libc.  Most can and are avoided using -nostdlib option.  However, as I will explain in a follow on post and as Greg has explained in other posts, there are other issues that really should be solved by having a custom toolchain for NuttX where NuttX's libc is built-in.

One topic I am introducing here, is the choice of upgrading to crosstool-ng, or sticking with nuttx buildroot.  Though I could probably be convinced to contribute my efforts on either tool, as I would rather have a unified community effort, I urge some consideration to using crosstool-ng as it does seem to be a very stable project requiring minimal changes to integrate NuttX so far.  It also seems to be a more highly regarded tool. 

Before continuing I should clarify that the rest of the steps to integrating the toolchain would be to build libc and libcxx into toolchain to create a hosted toolchain.  crosstool-ng and buildroot can be setup to do this.  crosstool-ng does this already by default. Additional effort is required for both tools.

I have started integration with crosstool-ng.  I will share a link to a repo in a follow-on post for others to use.  I am currently hitting build errors that I believe would happen regardless of my use of crosstool-ng or buildroot. I will post details in follow-on post as well.

I just want to get the conversation started about both help on this effort, and choice of tool to use moving forward for the NuttX toolchain.

Best,

Anthony

[ant...@vergeaero.com]

Adding some background:

My issues started when compiling a C++ library. Long story short this is the same issue Alan faced when integrating libcxx. The build errors seemed to suggest that somewhere in the NuttX code there was an unterminated extern "C" {... Alan's solution after not being able to find the unterminated extern "C" was to add extern "C++" around all libcxx headers, since the preprocessor supports nesting like this. This solved this issue, allowing libcxx code to build.

After hitting the same issue with different C++ code, I went digging for the unterminated extern "C" and without success of finding it, I stumbled upon the solution somewhere in a forum.  The gcc compiler is built in a way, that for certain targets, system headers are automatically wrapped in extern "C". I believe this was done to support backwards compatibility. On targets that support C++, a definition NO_IMPLICIT_EXTERN_C is added which disables this feature.  For the arm-none-eabi toolchain, this is not defined, and thus is the source of both my issue, and the libcxx issue Alan has fixed with the workaround.

This issue is one example of why a NuttX toolchain is needed

References:
Alan's issue: http://lists.llvm.org/pipermail/cfe-dev/2016-November/051535.html

https://bugs.launchpad.net/gcc-arm-embedded/+bug/1698539

https://gcc.gnu.org/ml/gcc-help/2011-09/msg00158.html

[patacongo]

I think the things that would be important would be:

1. Don't use the newlib library header files.  Those are a binary-incompatble time bomb that is just waiting to blow up in someone's face.

2. Use the the header files from the NuttX libc (but not the nuttx C library... the header files are enough.

1) and 2) basically build the tool chain that is build by builroot now.  Then if you add these:

3. Dissect out the newlib math libraries and build these separately agains the NuttX header files (plus possibly a custom math.h header file).  We have talked about this before.. about adding the newlib math libraries into the NuttX libc.  That can't be done because of source code licensing issues.  But there is absolutely no problem with building the newlib math libraries standalone in buildroot or crosstools-ng.  There is no licensing problem at all.

4. Bring in Alan's NuttX port of libxx.  Don't try to build libstdc++ (that is a possibilitiy, but challenging!).  You would probably still need libsup and possibly others from the GCC.

With these last two things we could have a complete, custom NuttX tool solution

[ant...@vergeaero.com]

Here is the repo that I am working out of for crosstool-ng to 

https://github.com/antmerlino/crosstool-ng

[ant...@vergeaero.com]

Greg,

From my understanding there are two types of environments the standard library can provide; freestanding, and hosted.  In freestanding, only basic libc header files are needed.  Whereas with hosted, libgcc is built and linked against the built-in libc.  Of course, to build the hosted toolchain, you must first build libc using a freestanding toolchain.  From my understanding, the nuttx buildroot builds the freestanding compiler. Is that correct?

I understand that in most cases for flat builds, libc.a is only linked in one place and that is taken care of by NuttX.  So you really only need a freestanding environment. Is that why you are suggesting not to build libc.a into the toolchain and only the headers?  Are you suggesting that there is no "hosted" toolchain?

If so, what does that mean for libcxx?  Is the goal just to add the include files and options into the compiler, but not actually to be building the standard library?

I think I understand how to go about many of these options, but not sure what the right solution is.  If we can clarify some of the vision and how it "should" work, I can probably take care of implementation and at least report the issues I face.

 

[ant...@vergeaero.com]

I should note that the goal of buildroot and crosstool-ng is usually to first build the freestanding compiler, then installing kernel headers and build libc and then using that to build final hosted compiler.  crosstool-ng breaks it down into these steps: (they are generally the same as buildroot)

CT_STEPS := \

            companion_tools_for_build  \

            companion_libs_for_build   \

            binutils_for_build         \

            companion_tools_for_host   \

            companion_libs_for_host    \

            binutils_for_host          \

            cc_core_pass_1             \

            kernel_headers             \

            libc_start_files           \

            cc_core_pass_2             \

            libc                       \

            cc_for_build               \

            cc_for_host                \

            libc_post_cc               \

            companion_libs_for_target  \

            binutils_for_target        \

            debug                      \

            test_suite                 \

            finish                     \

A good reference on this whole topic is here: 

https://crosstool-ng.github.io/docs/toolchain-construction/

[patacongo]

I understand that in most cases for flat builds, libc.a is only linked in one place and that is taken care of by NuttX. So you really only need a freestanding environment. Is that why you are suggesting not to build libc.a into the toolchain and only the headers?  Are you suggesting that there is no "hosted" toolchain?

Yes.  But mostly because the NuttX C library is a part of NuttX and, at least currently, cannot be built separately.

If so, what does that mean for libcxx?  Is the goal just to add the include files and options into the compiler, but not actually to be building the standard library?

You would have to talk to Alan.  I am not 100% certain how is libxx implementation works.  But I imagine in is like uClibc++:  when you build uClibc++, the C++ header file get installed inside of the NuttX include/ directory.  So when you draw in the NuttX header files, you would bet the C++ library files as well.

Building a foreign C++ library into GCC would, I suspect, be pretty tough.  Getting the GCC libstdc++ would probably also be difficult becuase almost certainly depends on secret interfaces and definitions in GLibc or newlib.

[ant...@vergeaero.com]

Okay, then getting libcxx integrated from a header file perspective should be pretty easy to do, I was under the impression more needed to be done to replace the lack of newlib.  

[ant...@vergeaero.com]

I am back to using buildroot, at least for now, as at least I can start with something that mostly works (other than cxx). 

My current hurdle: I need to build libsupc++.  This library is a sub-set of libstdc++-v3 inside gcc. However, we don't want all that.

The osdev.org website has been down the past few days.... Which is rather unfortunate timing considering it's an invaluable resource for understanding this whole toolchain web and I'm in the thick of it. Luckliy, wayback machine has some cached copies:

https://web.archive.org/web/20180115185925/http://wiki.osdev.org:80/Libsupcxx

These are the only instructions I can find for building libsupc++ stand-alone.

Here is what I *think* needs to happen...  looking for feedback if possible

1. The gcc build that is happening now inside of buildroot is to build the freestanding toolchain.  This means that the "gcc configure" operation that is happening, passes the WRONG toolchain and thus building libsupc++ from within this environment would be WRONG......... Correct??
2. If 1. is correct, to build our stand-alone libsupc++ I must follow the instructions in the above link, passing the toolchain built in the previous step. Since the toolchain in step 1 is built using NuttX headers, libsupc++ would be built against this too.... correct?
3. Once build I just need to copy it out to the lib directory so that future -lsupc++ options are resolved correctly.

If this is correct, a similar procedure (building using the built toolchain) should be used for building the partial newlib libm against the Nuttx headers using the freestanding toolchain.

[ant...@vergeaero.com]

I should clarify, I do not think gcc is being configured incorrectly, for what it is currently used for when I say "WRONG".  What I am trying to figure out is whether or not I can use that version of gcc configuration to build the libsupc++ library from.  Or, must I create a new build directory and re-configure gcc with different settings, then selectively build libsupc++ and any other additional libraries. My thought is: don't we need the toolchain created in the current gcc step in order to compile libsupc++.a?

[ant...@vergeaero.com]

Greg,

Find attached 2 patch files. The patches add nuttx to the list of supported OS' in gcc to a greater extent then before.  Now, we have a nuttx.h inside of gcc where we can define things like #define NO_IMPLICIT_EXTERN_C or other things.  

However, I believe there is an issue here that will break all compatibility with old buildroot builds, therefore please do not merge this yet.  Here is the issue:

• Changing config.sub in the package/gnuconfig folder inside buildroot will change the canonical toolchain name for all versions of all builds.  
• My change makes it so that `./config.sub arm-nuttx-eabi` outputs `arm-unknown-nuttx-eabi` instead of `arm-nuttx-eabi`
• This is because the current buildroot uses nuttx not as the system name, but the vendor. Herein lies the problem.
• Because the target will be arm-unknown-nuttx-eabi for all builds, not just the versions in which we've added NuttX OS support to, I'm nearly certain the builds will fail because GCC won't know what a NuttX system is.
• I originally implemented the config.sub change as a patch to gcc/binutils, but NuttX buildroot kept copying over using the config.sub provided by packages/gnuconfig.    I mention this because we may be able to use this version specific control of config.sub to overcome the problem.

Please review the patches and let me know your thoughts on how we should handle this. Perhaps we need some conditional logic somewhere?  Or maybe we copy the gnuconfig config.sub into gcc and binutils before applying patches, allowing us to patch in the addition of the nuttx os. Though this seems to defeat the purpose of the shared gnuconfig package....in which case we may want to use the config.sub of the package directly and not copy the common one in.

-Anthony

[ant...@vergeaero.com]

One solution that comes  to mind.... Add a Kconfig option that when selected patches the config.sub file in gnuconfig prior to copying over. This option could be selected automatically by particular versions of gcc.

[ant...@vergeaero.com]

Please find attached revised patches. One of the patches for gcc included in this buildroot patch was missing a quote ( " )

[Gregory Nutt]

Hi, Anthony,

Find attached 2 patch files. The patches add nuttx to the list of supported OS' in gcc to a greater extent then before.  Now, we have a nuttx.h inside of gcc where we can define things like #define NO_IMPLICIT_EXTERN_C or other things.

This would be in analogy to the linux/linux.h header file, right?

There is already a nuttx/nuttx.h header file.  Can we not use that?  It currently contains only a single Linux-like definition.

However, I believe there is an issue here that will break all compatibility with old buildroot builds, therefore please do not merge this yet.  Here is the issue:

• Changing config.sub in the package/gnuconfig folder inside buildroot will change the canonical toolchain name for all versions of all builds.  
• My change makes it so that `./config.sub arm-nuttx-eabi` outputs `arm-unknown-nuttx-eabi` instead of `arm-nuttx-eabi`
• This is because the current buildroot uses nuttx not as the system name, but the vendor. Herein lies the problem.

I think the toolchains do have a configure option to force the generated PREFIX to be anything what.  I might be mistaken.  Look at ./configure --help.  I think there is one.  The configured PREFIX can override the canonical PREFIX. I believe.

• Because the target will be arm-unknown-nuttx-eabi for all builds, not just the versions in which we've added NuttX OS support to, I'm nearly certain the builds will fail because GCC won't know what a NuttX system is.

If you can't force the tool PREFIX to be backward comaptible, that is not a problem either.  The tool prefix is determined by common logic in one place and changing the tool prefix will not effect anything other than a signel file (per architecture).  Look in arch/*/*/Toolchain.defs.  That is where the conflicting PREFIX is defined.  You can always change it to match.

• I originally implemented the config.sub change as a patch to gcc/binutils, but NuttX buildroot kept copying over using the config.sub provided by packages/gnuconfig.    I mention this because we may be able to use this version specific control of config.sub to overcome the problem.

I am sure that there is some kind of naming trickery that you could do to work around that.

Please review the patches and let me know your thoughts on how we should handle this. Perhaps we need some conditional logic somewhere?  Or maybe we copy the gnuconfig config.sub into gcc and binutils before applying patches, allowing us to patch in the addition of the nuttx os. Though this seems to defeat the purpose of the shared gnuconfig package....in which case we may want to use the config.sub of the package directly and not copy the common one in.

I will not have a chance to do any detailed review.  I have a bunch of things I need to get done in the next few days.

I seldom spot technical issues when reviewing code.  Sometimes I do if they are localized, but looking a a bunch of differences out of context is usually not very useful.  It is painful to do and usually does not accomplish much.

I generally have to trust people who submit patches that they have studied the issues and verified the solutions.  Occasionally my trust is misplaced, but I do have complete faith in you.  You tell me when its good... I can do my usually superficial review to make sure it looks good.

Greg

[patacongo]

Now I see this error on every file that has C++ support enabled (but does not include libc++):

make [1]: *** libs/libxx/libxx:  No such file or directory.  Stop.

This is new (mis)behavior and, I assume, the result of these changes.

Greg

[patacongo]

Now I see this error on every file that has C++ support enabled (but does not include libc++):

make [1]: *** libs/libxx/libxx:  No such file or directory.  Stop.

This is new (mis)behavior and, I assume, the result of these changes.

Yes, you change was incorrect.  There is no directory libs/libxx/libxx  that should be libs/libxx as it was before.  I have reverted your change.  I don't understand the libcxx build since I have never built it, but I assume that it is installed at libx/libxx/libcxx.  Please review this commit to see if it resolves the path errors:

commit a04d2eeb10b4af8fd9c3ab88d55d32416c20a291
Author: Gregory Nutt <gn...@nuttx.org>
Date:   Thu Sep 13 14:03:09 2018 -0600

    tools/Directories.mk:  Correct the path to the installed libcxx directory.

Greg

[ant...@vergeaero.com]

Greg,

Yes, that change looks correct.  This is unrelated to all buildroot efforts... It was introduced relatively recently when that restructuring happened with libs/ directory.  Oddly enough, I always see make complain about it, but it doesn't cause any build issues. However I got tired of seeing it and thought I resolved it with this change but see my mistake now.

Thanks for fixing it!

-Anthony

[Gregory Nutt]

> Yes, that change looks correct.  This is unrelated to all buildroot
> efforts... It was introduced relatively recently when that
> restructuring happened with libs/ directory.  Oddly enough, I always
> see make complain about it, but it doesn't cause any build issues.
> However I got tired of seeing it and thought I resolved it with this
> change but see my mistake now.

Actually,l it does cause a build problem but it is just not very visible
nor it is is catastrophic.  It would cause the create of dependencies to
fail in the libs/libxx directory.