30' build time with buildType == Release

[benjamin....@u-space.fr]

Hello BSK folks, 

I've noticed that it takes about 30 minutes to build Basilisk (and a plethora of ExternalModules) when buildType == Release is passed to the conanfile (or without passing it since it is the default value).

This shrinks to 9 minutes when buildType == Debug is used.

I'd curious to hear how this compares to build times experienced by other users.

Cheers, 
Ben

[Hanspeter Schaub]

Mm, curious.  I've not built with Debug in a long time.  On my M1 macBook a clean build (Release) takes about 3min.

[benjamin....@u-space.fr]

hi HP, thanks for your feedback. I've only ever tried to build BSK on Ubuntu systems. The most time consuming part of the build is the swig pass that occurs towards the end, not sure why it's taking forever ...

[benjamin....@u-space.fr]

Hello HP,

I've given another shot at compiling Basilisk alone (without external modules) and the full build completes in about 13 minutes. The desktop I'm using has a Intel® Xeon(R) W-2255 CPU @ 3.70GHz × 20 processor, 128 GB of RAM and runs Ubuntu 20.04.2 LTS. 

Do you know how long it takes for your continuous integration pipeline to build BSK ? Do you know how its HW specs compare to your mac or my desktop? 

Thanks, 
Ben

[Hanspeter Schaub]

Howdy Benjamin, this 13 min is to do a clean build?  Does this include conan building all the support packages? That can take a while, especially if you have `opNav` packages included that use `openCV`.  

If you are doing a basic build with `python3 conanfile.py` then I think it just builds it in a single threaded mode on Linux.  I would recommend looking over the section `Building The Framework Separately` on http://hanspeterschaub.info/basilisk/Install/installOnLinux.html.  Here we use the `buildProject` flag being set to false, you cd into `dist3`, and then invoke `make -j20` in your case as you have 20 cores.

I have a macBook with the M1 Pro chip in it.  It has a total of 10 cores (8 performance and 2 efficiency cores I think).  Assuming the conan support packages have already been built, a basic clean build in a multi-threaded manner takes about 3min.

[benjamin....@u-space.fr]

Hi HP, 

13 minutes is the time needed for 'python conanfile.py --clean' to complete, assuming support packages have already been compiled (so 13 minutes is the recurring compilation time to rebuild basilisk once the support packages have been installed).

'python conanfile.py' does make use of all available cores on a linux machine. There could be some XCode and/or ARM processor magic at play here, so I'm really curious about compilation times obtained by other users on different environments.

I don't have access to the workstation today but will give a shot to 'python conanfile.py --buildProject False' followed by 'make -j' next week. 

Thanks, 
Ben

[benjamin....@u-space.fr]

Good morning HP, 

I just gave a try to the above approach this morning ('python conanfile.py --buildProject False' followed by 'make -j') . Here are my findings:

1. The build still takes about 15 minutes to complete. This is similar to the build times I was previously achieving with buildProject == True
2. The above approach however requires a large amount of RAM (up to 20 GB). Trying it out on my laptop last week didn't go well for that reason

I would be extremely interested to get more feedback from the BSK community regarding the build times they have managed to achieve. Does the AVS lab have any feedback on that matter ? 

Cheers, 

Ben

[benjamin....@u-space.fr]

Here's another observation : the build time is halved if --buildType Debug is used

[Hanspeter Schaub]

I'll ask some of my students.  Nobody right now runs Linux, but macOS and Windows.  If you build with Debug the run-time performance suffers due to the Eigen library being a header only library.

[benjamin....@u-space.fr]

Hello HP, 

Assuming that you last built basilisk in Release mode, would you mind checking the CMakeCache.txt file under basilisk/dist3 and tell me what the flags CMAKE_CXX_FLAGS_RELEASE and CMAKE_C_FLAGS_RELEASE are set to ? 

Thanks, 
Ben

[Hanspeter Schaub]

I see

//Flags used by the CXX compiler during RELEASE builds.
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG

//Flags used by the C compiler during RELEASE builds.
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG

[Scott Carnahan]

Hey Ben and Dr. Schaub,

My team uses BSK in WSL2 with Ubuntu 20.04 and a variety of laptops. We struggle majorly with build times - well beyond what one expects from running in a virtual machine like WSL. In general, the first build takes 30-60 minutes. If you kill it during the final swig pass as Ben said it seems to always have been trying to swig messagingPYTHON_wrap.cxx. I always build with 6 cores on my 8 core i7.

As of last week one of my team members can't get a clean build to complete at all. We also have a handful of external modules we've built and are always building. we're exploring some of the RAM and external module issues that Ben brought up.

The hour long build has been true for me since I started using BSK more heavily around the holidays in December.

I'm hopeful, given Dr. Schaub's 3 minute build time.

-Scott

[benjamin....@u-space.fr]

Thanks HP. I was somehow thinking -O3 might explain the long build times, but this is clearly not the case. 

[Hanspeter Schaub]

Howdy Scott, thanks for the feedback.  Sounds like the Linux build is taking an undue amount of time to build.  On the CI servers we use the Linux build is taking far longer than the other builds on Windows and macOS, but we thought this was because the Linux CI computers were slower hardware.  This info makes me question this.

But, that said, I'm not in a position to dig into the Linux specific build scripts at the moment.  If someone sees something that is off and causing longer build time, please let us know.  There is no reason to believe the GNU compiler is that much slower than the clang macOS compiler or VisualStudio compilers.

[Clément Jonglez]

Hello folks,

I also have the same issue on my Ubuntu 20.04 machines (3 different computers). It takes for instance 20 minutes for my 8-thread Intel Xeon to compile Basilisk, and at some point of the compilation it takes more than 30% of the 16GB RAM. Until this is resolved (and I don't enough about Basilisk's build process to solve it myself yet), I will use the debug build.

Cheers

Clément

[Hanspeter Schaub]

Not sure why this is taking so much longer on Linux.  It appears linked to the use of gcc.  Others have used clang on Linux and gotten much faster build times.  We are close to pushing an update to the build system to the `develop` branch that has cut the Linux gcc build time in half.  Keep checking develop branch.