Quick profiling comparision

[Tully Foote]

To get a sense of what the real speedups are between the systems I did some very quick timings.  There's definitely more precise benchmarking which can be done, but I wanted to give people a sense of the scale when we're talking about speedups using a single workspace. 

I installed electric and fuerte ros-base variant (ros and ros_comm stacks) from source on my machine and ran equivalent commands on the two.  This seems like a reasonably representative workspace.  This is a test of 68 packages compiling.  The old rosbuild system is exactly a python script invoking cmake && make multiple times in different workspaces. 

One optimization to note in the old system is that 9 packages do not have a Makefile to avoid invoking cmake at all.  Thus in electric it's really only traversing 59 packages.  

The biggest difference I will note is that the noop build using a single cmake workspace is on the order of one second (less if you enable parallelization).  And if you tweak a single file, it climbs up to 2 seconds.  Whereas a noop build in rosmake takes 30 seconds.  Also the actual full compile step takes about the same amount of time, if the fuerte system is not threaded.  If it is threaded it takes one 5th the time.  

A random comparison I'd note is that a full configure and build with a single workspace takes the same amount of time as a noop build using rosbuild.  Also from profiling the rosmake builds, the minimum invocation time for cmake && make is about 0.3 to 0.5 seconds per package.  

Tully

Fuerte ros-base:

cmake .. 14.6s

make 106.4s

make clean 2.3s

make -j8 19.2s

make 1.7s

make -j8 0.2s

touch ros/core/roslib/src/package.cpp

make 2.1s

cmake .. 10.0s

Electric ros-base

rosmake -a 105.9s

rosmake -a 27.1s

touch ros/core/roslib/src/package.cpp

rosmake -a 26.8s

rosmake -a --threads=1 31.3s

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[tkruse]

Thanks Tully,

I tried to incorporate that information in the catkin review, as additional justification for how much catkin improves usability.

On that occasion, I though about whether some of the catkin lessons and Daniel's remarks could be used for rosbuild/rosmake, given those might be around for some more time.

So one idea was that rosmake could avoid the configure step when there is nothing to reconfigure.
Lorenz created a patch to the mk/cmake.mk which I added at:
https://code.ros.org/trac/ros/ticket/4035

With that, the noop rebuild of electric actionlib on my laptop dropped from 30s to 10s.
Still not like catkin, obviously (at 0.2s).

So this is not to say your benchmark is flawed or rosbuild could be kept, but while many people still work with rosbuild, maybe other speed improvements are possible to make rosmake/rosbuild more usable.

I also think about other improvements, such as limiting the scope of the cmake process to a set of packages, e.g. the workspace set (for professional users), so that people do not have to manually call make in several directories just to avoid rosmake delays.

regards,
  Thibault

[tkruse]

Looking some more into these dramatic speed differences, I got suspicious. After all, the bulk of the electric build time is spent on building roscpp, and a make call in there should not be faster in catkin than with rosmake.

I believe this is due to a bug in rosmake:
https://code.ros.org/trac/ros/ticket/4036
which cause rosmake never to run make with multiple jobs per package. If true, that would mean a lot of wasted waiting time.

I listed other possible causes for speed differences in the catkin review:
http://www.ros.org/wiki/catkin/Reviews/2012-08-01_API_Review#On_configuration_.26_build_speed

One that may also be interesting now is the lack of distinction between configure, runtime and build time dependencies.
E.g. consider the ROS package rosout, which just creates an executable. roslaunch depends on rosout at runtime, but not at build time. So rosmake could invoke make in those two in parallel, but does not because roslaunch manifest says it depends on rosout. The same goes for anything that depends on roslaunch, because roslaunch only contains python code. This chain of packages goes on.

I believe this is interesting even with catkin, as moving forward with catkin one day we might consider more isolated builds.
So what are other people's thoughts on extending the manifest to declare the lifecycle stages of dependencies?
I believe a backwards-compatible way would be like:
<depend package="roslaunch" at="runtime"/>, with default being depending in all phases.

I know n catkin's current design, that would not make a difference, so for catkin manifests that could be confusing (as an attribute without effect) but changing the catkin design one day is a distinct possibility.

cheers,
  Thibault