Moving on from make for embedded cross compilation

[Bernhard Weller]

Hello,

we are facing some issues with our GNU make based build (running on Windows 10). The main issue is that it inexplicably runs extremely slow on our Jenkins build client but only when it runs in a Jenkins context. Not sure if this problem will resolve when we move to shake, but moving on doesn't seem to be the worst idea anyways. 

In the past days I tried to educate myself on build systems to move on from make. A lot of the build systems seem to move to magic (my opinion) and "just work" for specific targets and compilers. The thing I'm seeing critically is "How do I get it to work with my compiler and my static code analysis tool and other things?" because we are working with small embedded systems and our toolchain isn't very well supported in the open source community. So I was focusing more on build systems which require to write the rules as default (because I hope that the language would be easier if that's the main feature). A reddit post pointed me to Build systems à la carte, which is how I ended up here.

I was also really considering tup, but it looks like it isn't actively developed anymore, which is sad. The automatic dependency detection (although it sounds a bit whacky with the injection) is a feature which got me interested.

Let me explain a bit on how things work for us:

We have a codebase which contains a lot of software components (a bit like libraries, but they don't get linked as a lib) and software units, which linked together with the software components form a functional executable. In the last step we can combine two units to form a software segment - usually a boot part and a main part. The combination is done with the help of some batch scripts which is working fine as it is.

The way a software unit it built looks like this: we have (or should have) a makefile which contains all the rules which work universally across our range of units. For a specific unit we edit the list of directories containing the source to build the unit. Each unit can have different flags for the compiler and of course has a unique name to it. We have the ability to build either debug or release builds with different compiler and linker flags and to build everything (make all). Debug and Release get placed in different subdirectories in a binary directory for that unit.

During our build we can have different static code analysis tools running in conjunction with the compiler (currently they run sequentially, first compile then lint in one rule) generating additional output files in Lint-directories.

One of the evil quirks we have to use is a tool to format the output of our compiler (IAR Embedded Workbench) to give output which is understandable by Visual Studio (2013 for that matter). For that we use a bash shell to get the ability to pipe the output in a format tool. A similar approach is done for the linter, the output is written to a file (which is needed to have post-build) and then the relevant messages are filtered to the console output using gawk.

The build takes quite a while even though our projects are rather small. For example a complete build (debug and release with the linter running on the release build) where the boot part is around 70 files to compile and the main part is around 250 files it takes around 6 minutes to complete on my machine using -j8 (I have a small i7-7500U (dual core, 4 threads) but it doesn't use 100 % CPU with that setting...). Just for the fun of mentioning it, the same build takes 3 hours and 30 minutes on our Jenkins which is a way more capable machine.

Question time (note: no one has Haskell knowledge in my company, so that is a hurdle to take):

1) Is it possible to have build.exe which will read from configuration files a list of source directories and compiler flags to build different software units without each unit having an own build.exe?

So we have a 'BootPart/BuildBootPart.Config' and a 'MainPart/BuildMainPart.Config' and then can call 'Software/BootPart/> Build.exe -f BuildBootPart.Config all' or something similar to that.

2) I think I already read this somewhere, but having a release, debug and all target should be possible, as well as passing an option like DONT_LINT to prevent the linter from running?

3) Is it possible to capture the command output and manipulate it directly in the build.exe to format it in a way to make it understandable for Visual Studio (with the help of regular expressions usually)? This would remove the dependency on a Unix shell under Windows and reduce the number of processes needed to spawn and speed up everything (I think). Same for the linter.

4) Does the build.exe have any dependencies? We'd like to distribute the build system along with the source for the developers without having to install anything (compiler is an exception to that rule).

5) As a starting point I guess I should go have a look at shake-language-c and start from there?

This is a rather lengthy post, thanks to anyone reading it and for any answer or suggestion in advance.

Bernhard

[Raoul Duke]

to get OT momentarily sorry... i am a fan of shake...

yet sometimes i wonder why we can't just use our regular programming / scripting languages instead to magic baked into make/cmake/scons/etc. 

just like how we could roll our own simple dependency injection instead of using eg Spring. 

of course it is likely that after making a home grown system it would either be hell, or would turn out looking like other tools, with magic built in cf. <.> exe featuritis. :)

[Vanessa McHale]

Quirks like piping through bash and whatnot is definitely doable, though maybe not as most straightforward if you haven’t used Haskell or an ML before.

The build executable hopefully won’t have many dependencies, by default GHC links other Haskell libraries statically.

Cheers,

--
You received this message because you are subscribed to the Google Groups "Shake build system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shake-build-sys...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/shake-build-system/42d2cb45-a20a-4f42-b469-0bc20c16b8f9%40googlegroups.com.

[Neil Mitchell]

Hi Bernhard,

Interesting application! A few notes:

* Splitting lint and compile will probably give a nice speed up, as then your link can proceed earlier.

* Using things like gawk and bash scripts on Windows can be quite expensive. The advantage of Shake is that moving that stuff into Haskell is very natural, and if the pieces aren't that complex, not very hard to do. However, I'd port it as is, and then improve in a subsequent step.

* Having a custom toolchain is absolutely fine for Shake. We haven't yet built up standardised toolchains, so going custom is literally no additional effort.

* Have you checked for an antivirus or similar on your Jenkins server? Is it running from a network-mounted disk? Is the disk nearly full?

* The lack of Haskell knowledge makes things harder. A simple Shake script can be viewed as a weird language for builds, e.g. https://shakebuild.com/manual doesn't assume any Haskell knowledge. Going deeper you certainly benefit from Haskell. There are lots of resources on learning Haskell that are very good, so hopefully it's not too much of an impediment.

1) Yes. I gave a talk on that. https://ndmitchell.com/#shake_09_oct_2015. Unfortunately the videos are temporarily unavailable, but the slides should help.

2) Yep, debug/release is all fine.

3) With Shake you get a library that you use to build a program, where the library can provide 90% of the work. Alternatively, you can do a little more and specify exactly what it outputs, when, and how. Shouldn't be hard to get rid of the wrapper. At worst, just turn on Quiet and take control of the output, although a few careful putNormal lines might work.

4) No runtime dependencies at all. Statically linked. You may wish to distribute the .Config files as per Q1, but they can also be linked into the exe if you want.

5) I'd probably start from the manual at https://shakebuild.com/manual. If you are writing everything custom, and your C rules aren't that complex, doing it yourself from scratch might let you use all the moving parts yourself. If however you want to abstract over the complexities of Anroid vs Linux compilation, that's a great package to start from (but doesn't seem to be your use case).

Thanks, Neil

--

[Bernhard Weller]

Hi Neil and all the others,

thanks for the detailed response and reassurance that shake should work for my intentions.

It seems though, that someone is already using CMake in my company and that we might move to that approach, but I'm not supporting that approach.

So it's probably going to be a long way for me to learn Haskell and build a shake build system in my free time, but my complete misunderstanding about functional programming makes it a lot more interesting to learn.

Thanks for the pointers, I'll go check them out and work my way through.

Best regards and stay healthy,

Bernhard

To unsubscribe from this group and stop receiving emails from it, send an email to shake-build-system+unsub...@googlegroups.com.