On Meta Build Tools and ROS
62 views
Skip to first unread message
Jonathan Bohren
Jul 7, 2015, 10:12:50 AM7/7/15
to ros-sig-ng-ros, ros-sig-buildsystem
All,
I've tried to summarize my thoughts on this topic that I feel like we should stop spending so much time talking about so that we can get back to making robotics happen. Take a look at the google doc linked below, in which I express both my interpretation of the facts and my own opinions:
-j
Brian Gerkey
Jul 8, 2015, 9:50:44 PM7/8/15
to ros-sig...@googlegroups.com, ros-sig-buildsystem
Thanks for the summary, Jon. Looks like there's a useful conversation
going on inside that doc.
I'd like to explain a few things, which I think either haven't been
said, or perhaps have gotten lost in the discussion. I'll borrow your
Socratic approach to cover the key concerns and questions that I've
heard so far:
1. I'm worried that ROS 2 development will break my ROS 1 system.
To maintain the stability of existing ROS 1 code and avoid the risk of
regressions in ROS 1, we don't intend to backport anything significant
from ROS 2. That's not to say that it can't happen, but our default
position is that ROS 1, including catkin, should remain as is, with
incremental improvements and bug fixes.
2. OSRF (and Willow Garage before it) wastes time working on build systems.
Form the beginning, we adopted for ROS a federated development model.
We made that choice in the hope that it would help to spur adoption,
participation, and contribution, and I think that we were right. The
tradeoff is that it's inherently complex to manage code in a federated
ecosystem, whether building a bunch of interdependent packages from
source or creating binary installation packages.
We mitigate that complexity in the build system with automation.
Wherever possible, we use existing, standard tools, namely CMake and
Python. Where those tools lack built-in support for what we want to
do, we build on top of them (as we've done since the rosbuild days).
Of course, you could agree with all the above, and still believe that
we spend more time on build systems than is justified, given the
resulting benefits. That's a question on which reasonable people can
disagree.
3. Why did OSRF create a yet another build system, instead of improving catkin?
ament is an evolution of catkin, not a new build system. You could
think of it as catkin 2.0.
Our goal with ament is to take the feedback that we've gotten over the
years about catkin, along with the great enhancements that have been
developed in auxiliary packages like catkin_simple and catkin_tools,
and provide the best possible user/developer experience. We are
intimately aware of the pain and suffering caused by the rollout of
catkin in Groovy, and we do not intend to repeat it.
ament started out as a fork of catkin, then accumulated a variety of
changes. The changes were substantial enough that we decided to keep
the two code bases separate. Down the road, as we identify common
chunks of code that should be shared, we can always refactor (merging
parts of catkin_tools and ament_tools into a third, shared Python
package is a good example).
4. Then, why change the name? Why not call it catkin 2.0, or catkin2?
To achieve the improvements that we want to make in catkin, we're not
committing to backward compatibility with the current version of
catkin (this could change, e.g., by adding a 'catkin_cmake' build type
to ament). We expect people to want to develop ROS 1 and ROS 2 code
on the same machine (and perhaps in the same workspace or even
package). So we need to support side-by-side installation and use of
catkin and ament.
So we need a different string identifier to use for naming
directories, CMake modules, etc. Or at least we think that we do.
We'd be happy to hear suggestions for how to keep the name catkin but
allow installation and selecting of different versions. We'd also
need to consider how to manage documentation for the different
versions in a way that's not too confusing.
Assuming that we do need a different name, 'catkin2' could easily be
used in place of 'ament.' We chose 'ament' as the package name and
the prefix for CMake function names because we felt that 'catkin2' has
the potential to confuse users. I'm happy to hear counter-arguments
on that point.
5. I don't want to have to port my build files from catkin to ament.
For a start, you only need to update CMakeLists.txt files for packages
that you're converting to use ROS 2, and at that point you're probably
touching quite a bit of code anyway (perhaps with the help of
semi-automatic upgrade tools). For most packages the CMake changes
will be pretty minor by comparison. No changes are needed in your
package.xml.
To give you an idea of what the CMake changes will look like, Dirk put
together some examples:
- catkin: https://gist.github.com/dirk-thomas/596a53bbb922c4d6988a
- ament: https://gist.github.com/dirk-thomas/83ae6bfbfc94e06cd519
- ament auto: https://gist.github.com/dirk-thomas/a76f952d05e7b21b0128
(These are the kinds of things that will make up part of the ROS 1->2
migration guide.)
ament auto is kind of like catkin_simple, but because we're able to
integrate it properly into the build system, as opposed to hanging it
on the outside, it can cover a lot more use cases. I would guess that
ament auto will handle the needs of most users, and it's pretty
concise.
6. catkin was / is too hard to use. ament will be the same or worse.
I certainly hope not. We learned a lot from catkin, and are doing our
best to make ament (and ament auto) easy to use while still offering
the automation that we need to manage the complexity of doing
federated development.
We're already working with ament on ROS 2 development and improving it
where needed as we go. We'd love to get your feedback, if you're
willing to try it out.
brian.
> --
> You received this message because you are subscribed to the Google Groups
> "ROS SIG NG ROS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ros-sig-ng-ro...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
going on inside that doc.
I'd like to explain a few things, which I think either haven't been
said, or perhaps have gotten lost in the discussion. I'll borrow your
Socratic approach to cover the key concerns and questions that I've
heard so far:
1. I'm worried that ROS 2 development will break my ROS 1 system.
To maintain the stability of existing ROS 1 code and avoid the risk of
regressions in ROS 1, we don't intend to backport anything significant
from ROS 2. That's not to say that it can't happen, but our default
position is that ROS 1, including catkin, should remain as is, with
incremental improvements and bug fixes.
2. OSRF (and Willow Garage before it) wastes time working on build systems.
Form the beginning, we adopted for ROS a federated development model.
We made that choice in the hope that it would help to spur adoption,
participation, and contribution, and I think that we were right. The
tradeoff is that it's inherently complex to manage code in a federated
ecosystem, whether building a bunch of interdependent packages from
source or creating binary installation packages.
We mitigate that complexity in the build system with automation.
Wherever possible, we use existing, standard tools, namely CMake and
Python. Where those tools lack built-in support for what we want to
do, we build on top of them (as we've done since the rosbuild days).
Of course, you could agree with all the above, and still believe that
we spend more time on build systems than is justified, given the
resulting benefits. That's a question on which reasonable people can
disagree.
3. Why did OSRF create a yet another build system, instead of improving catkin?
ament is an evolution of catkin, not a new build system. You could
think of it as catkin 2.0.
Our goal with ament is to take the feedback that we've gotten over the
years about catkin, along with the great enhancements that have been
developed in auxiliary packages like catkin_simple and catkin_tools,
and provide the best possible user/developer experience. We are
intimately aware of the pain and suffering caused by the rollout of
catkin in Groovy, and we do not intend to repeat it.
ament started out as a fork of catkin, then accumulated a variety of
changes. The changes were substantial enough that we decided to keep
the two code bases separate. Down the road, as we identify common
chunks of code that should be shared, we can always refactor (merging
parts of catkin_tools and ament_tools into a third, shared Python
package is a good example).
4. Then, why change the name? Why not call it catkin 2.0, or catkin2?
To achieve the improvements that we want to make in catkin, we're not
committing to backward compatibility with the current version of
catkin (this could change, e.g., by adding a 'catkin_cmake' build type
to ament). We expect people to want to develop ROS 1 and ROS 2 code
on the same machine (and perhaps in the same workspace or even
package). So we need to support side-by-side installation and use of
catkin and ament.
So we need a different string identifier to use for naming
directories, CMake modules, etc. Or at least we think that we do.
We'd be happy to hear suggestions for how to keep the name catkin but
allow installation and selecting of different versions. We'd also
need to consider how to manage documentation for the different
versions in a way that's not too confusing.
Assuming that we do need a different name, 'catkin2' could easily be
used in place of 'ament.' We chose 'ament' as the package name and
the prefix for CMake function names because we felt that 'catkin2' has
the potential to confuse users. I'm happy to hear counter-arguments
on that point.
5. I don't want to have to port my build files from catkin to ament.
For a start, you only need to update CMakeLists.txt files for packages
that you're converting to use ROS 2, and at that point you're probably
touching quite a bit of code anyway (perhaps with the help of
semi-automatic upgrade tools). For most packages the CMake changes
will be pretty minor by comparison. No changes are needed in your
package.xml.
To give you an idea of what the CMake changes will look like, Dirk put
together some examples:
- catkin: https://gist.github.com/dirk-thomas/596a53bbb922c4d6988a
- ament: https://gist.github.com/dirk-thomas/83ae6bfbfc94e06cd519
- ament auto: https://gist.github.com/dirk-thomas/a76f952d05e7b21b0128
(These are the kinds of things that will make up part of the ROS 1->2
migration guide.)
ament auto is kind of like catkin_simple, but because we're able to
integrate it properly into the build system, as opposed to hanging it
on the outside, it can cover a lot more use cases. I would guess that
ament auto will handle the needs of most users, and it's pretty
concise.
6. catkin was / is too hard to use. ament will be the same or worse.
I certainly hope not. We learned a lot from catkin, and are doing our
best to make ament (and ament auto) easy to use while still offering
the automation that we need to manage the complexity of doing
federated development.
We're already working with ament on ROS 2 development and improving it
where needed as we go. We'd love to get your feedback, if you're
willing to try it out.
brian.
> You received this message because you are subscribed to the Google Groups
> "ROS SIG NG ROS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ros-sig-ng-ro...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Thibault Kruse
Jul 9, 2015, 6:31:34 AM7/9/15
to ros-sig...@googlegroups.com, ros-sig-b...@googlegroups.com
Thanks Brian for giving the OSRF perspective.
It sill seems odd to me. Basically OSRF seems to be saying:
1. ament is *very* similar and close in source to catkin.
2. ament is vastly superior to catkin.
3. ROS1 users will not get ament, they are stuck with catkin until they migrate to ROS2. Tough luck.
That does not add up well. If ament is vastly superior yet similar to catkin, then it makes sense to allow building ROS1 packages with it as well.
It sill seems odd to me. Basically OSRF seems to be saying:
1. ament is *very* similar and close in source to catkin.
2. ament is vastly superior to catkin.
3. ROS1 users will not get ament, they are stuck with catkin until they migrate to ROS2. Tough luck.
That does not add up well. If ament is vastly superior yet similar to catkin, then it makes sense to allow building ROS1 packages with it as well.
On Thursday, July 9, 2015 at 3:50:44 AM UTC+2, Brian Gerkey wrote:
1. I'm worried that ROS 2 development will break my ROS 1 system.
To maintain the stability of existing ROS 1 code and avoid the risk of
regressions in ROS 1, we don't intend to backport anything significant
from ROS 2. That's not to say that it can't happen, but our default
position is that ROS 1, including catkin, should remain as is, with
incremental improvements and bug fixes.
The main worry should be that an inevitable future transition to ROS2 will break the robot.
That means if a team is working on a 5 year project, and halfway through some
packages will start to be released only for ROS2, not for ROS1, then that team will
have to spend vast amounts of time on migrations and bridges and not on their
actual research. And just sticking with ROS1 and old libraries and old linux versions
may not be an option.
3. Why did OSRF create a yet another build system, instead of improving catkin?
The main question here is why ament was not build to still support ROS1 builds from the start.
From what I understand, of all improvements made to ament, the only crucial one
was the restructuring to allow ROS2 message generation. Everything else are "nice-to-have's",
that would by themselves never have motivated OSRF to invest time in the buildsystem.
Given that, the strategy could have been as follows:
1. Create a workspace with catkin on an experimental branch/fork catkin2.
2. Add ROS1 core packages and some example ROS1 packages
3. Add ROS2 core packages
4. change ROS2 packages and catkin2
5. check ROS1 still builds + works
6. if ROS2 goals not achieved or ROS1 breaks, goto 4
7. Done
This process would not have broken catkin or disturbed ROS1, and the changes
could have been kept outside the main catkin branch, or even in separate sources
like ament. Catkin as a name would have been more viable to keep around where
possible, sharing and reusing code would have been more viable. Early adopters
working on ROS1 could have started using ament/catkin2 and provide early feedback,
while working realistically with ROS1.
This process would also made it more obvious to keep single message definition packages.
It seems gruelsome that there is now common_msgs and common_interfaces, and people
worrying about how to build bridges spanning both 2 communication layers and two buildsystems.
Instead, in the workspace used in the refactoring process above, common_msgs could have
been switched to a ROS2 branch with message generators for both ROS2 and ROS1. With mappings from ROS2
to ROS1, making sure wach incremental change to the IDL can be mapped to a ROS1 message.
That way, a package like common_msgs might even not just generate bindings for ROS2 and ROS1,
but also the bridging/transformation code, obviously also belonging close to both message definitions.
In the current layout, to make any change to a message definition, it is necessary to change the
ROS2 .msg file, and the ROS1, msg file, and the bridge, and then validate the change.and then those
3 packages must be updated simultaneously in every active workspaces in the world, which for most
users means updating an amend workspace and then updating the overlaying catkin workspace.
Did OSRF actually decide that this would be the optimal strategy for introducing ROS2?
regards,
Thibault
Geoffrey Biggs
Jul 9, 2015, 7:28:11 AM7/9/15
to 'Thibault Kruse' via ROS Buildsystem Special Interest Group
that take 5 years). They choose their tools at the start of the project such
that they will be supported (as in fixing problems) for the length of the
project, but they don't typically want major changes during the project's time
schedule. Industry traditionally doesn't like moving targets.
Granted, research projects are very different and by nature may have to track a
moving target. I think that most research projects are willing to pick a
particular point in time at which to invest their resources in upgrading their
infrastructure rather than research for a period of time.
I agree with the OSRF and Ingo's statements. There are so many things changing
in breaking ways, it makes sense to create a single clean break. Then users can
say "for the next X months, we will spend our time moving to the new
infrastructure (ROS 2)" rather than having to continually try and keep up with
constant new features being introduced. If it were *just* the build system or
*just* the transport or *just* how nodes are structured, then I would be of a
different opinion.
Geoff
Vincent Rabaud
Jul 9, 2015, 11:31:42 AM7/9/15
to ros-sig-b...@googlegroups.com
Isn't there an obvious technical reason for justifying the split ? (maybe OSRF forgot because it's so obvious to them). I believe:
- catkin needs to work on Ubuntu Trusty because Indigo is a LTS distro (cf http://www.ros.org/reps/rep-0003.html#catkin-rosbuild-support)--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/20150709112803.GA3047%40atlas.yamane.nz.
Dirk Thomas
Jul 9, 2015, 12:16:55 PM7/9/15
to ros-sig-b...@googlegroups.com
On Thu, Jul 9, 2015 at 8:31 AM, Vincent Rabaud <vincent...@gmail.com> wrote:
- ament is not finished yet and uses CMake 3.2 and Python 3.4, features which are not backportable to TrustyIsn't there an obvious technical reason for justifying the split ? (maybe OSRF forgot because it's so obvious to them). I believe:- catkin needs to work on Ubuntu Trusty because Indigo is a LTS distro (cf http://www.ros.org/reps/rep-0003.html#catkin-rosbuild-support)
This is not the fact.
We currently do all our ROS 2 development on Ubuntu Trusty and use the default packages for CMake (2.8.12.2) and Python3 (3.4.0).
- Dirk
Reply all
Reply to author
Forward
0 new messages