On Meta Build Tools and ROS

[Jonathan Bohren]

All,

I've tried to summarize my thoughts on this topic that I feel like we should stop spending so much time talking about so that we can get back to making robotics happen. Take a look at the google doc linked below, in which I express both my interpretation of the facts and my own opinions:

https://docs.google.com/document/d/1Tnst0T4LoDWnHZ5gwT-C-6F68A33_AkLUBozdP-hEso/edit?usp=sharing

-j

[Brian Gerkey]

Thanks for the summary, Jon. Looks like there's a useful conversation
going on inside that doc.

I'd like to explain a few things, which I think either haven't been
said, or perhaps have gotten lost in the discussion. I'll borrow your
Socratic approach to cover the key concerns and questions that I've
heard so far:

1. I'm worried that ROS 2 development will break my ROS 1 system.

To maintain the stability of existing ROS 1 code and avoid the risk of
regressions in ROS 1, we don't intend to backport anything significant
from ROS 2. That's not to say that it can't happen, but our default
position is that ROS 1, including catkin, should remain as is, with
incremental improvements and bug fixes.

2. OSRF (and Willow Garage before it) wastes time working on build systems.

Form the beginning, we adopted for ROS a federated development model.
We made that choice in the hope that it would help to spur adoption,
participation, and contribution, and I think that we were right. The
tradeoff is that it's inherently complex to manage code in a federated
ecosystem, whether building a bunch of interdependent packages from
source or creating binary installation packages.

We mitigate that complexity in the build system with automation.
Wherever possible, we use existing, standard tools, namely CMake and
Python. Where those tools lack built-in support for what we want to
do, we build on top of them (as we've done since the rosbuild days).

Of course, you could agree with all the above, and still believe that
we spend more time on build systems than is justified, given the
resulting benefits. That's a question on which reasonable people can
disagree.

3. Why did OSRF create a yet another build system, instead of improving catkin?

ament is an evolution of catkin, not a new build system. You could
think of it as catkin 2.0.

Our goal with ament is to take the feedback that we've gotten over the
years about catkin, along with the great enhancements that have been
developed in auxiliary packages like catkin_simple and catkin_tools,
and provide the best possible user/developer experience. We are
intimately aware of the pain and suffering caused by the rollout of
catkin in Groovy, and we do not intend to repeat it.

ament started out as a fork of catkin, then accumulated a variety of
changes. The changes were substantial enough that we decided to keep
the two code bases separate. Down the road, as we identify common
chunks of code that should be shared, we can always refactor (merging
parts of catkin_tools and ament_tools into a third, shared Python
package is a good example).

4. Then, why change the name? Why not call it catkin 2.0, or catkin2?

To achieve the improvements that we want to make in catkin, we're not
committing to backward compatibility with the current version of
catkin (this could change, e.g., by adding a 'catkin_cmake' build type
to ament). We expect people to want to develop ROS 1 and ROS 2 code
on the same machine (and perhaps in the same workspace or even
package). So we need to support side-by-side installation and use of
catkin and ament.

So we need a different string identifier to use for naming
directories, CMake modules, etc. Or at least we think that we do.
We'd be happy to hear suggestions for how to keep the name catkin but
allow installation and selecting of different versions. We'd also
need to consider how to manage documentation for the different
versions in a way that's not too confusing.

Assuming that we do need a different name, 'catkin2' could easily be
used in place of 'ament.' We chose 'ament' as the package name and
the prefix for CMake function names because we felt that 'catkin2' has
the potential to confuse users. I'm happy to hear counter-arguments
on that point.

5. I don't want to have to port my build files from catkin to ament.

For a start, you only need to update CMakeLists.txt files for packages
that you're converting to use ROS 2, and at that point you're probably
touching quite a bit of code anyway (perhaps with the help of
semi-automatic upgrade tools). For most packages the CMake changes
will be pretty minor by comparison. No changes are needed in your
package.xml.

To give you an idea of what the CMake changes will look like, Dirk put
together some examples:
- catkin: https://gist.github.com/dirk-thomas/596a53bbb922c4d6988a
- ament: https://gist.github.com/dirk-thomas/83ae6bfbfc94e06cd519
- ament auto: https://gist.github.com/dirk-thomas/a76f952d05e7b21b0128
(These are the kinds of things that will make up part of the ROS 1->2
migration guide.)

ament auto is kind of like catkin_simple, but because we're able to
integrate it properly into the build system, as opposed to hanging it
on the outside, it can cover a lot more use cases. I would guess that
ament auto will handle the needs of most users, and it's pretty
concise.

6. catkin was / is too hard to use. ament will be the same or worse.

I certainly hope not. We learned a lot from catkin, and are doing our
best to make ament (and ament auto) easy to use while still offering
the automation that we need to manage the complexity of doing
federated development.

We're already working with ament on ROS 2 development and improving it
where needed as we go. We'd love to get your feedback, if you're
willing to try it out.

brian.

> --
> You received this message because you are subscribed to the Google Groups
> "ROS SIG NG ROS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ros-sig-ng-ro...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Thibault Kruse]

Thanks Brian for giving the OSRF perspective.

It sill seems odd to me. Basically OSRF seems to be saying:

1. ament is *very* similar and close in source to catkin.
2. ament is vastly superior to catkin.
3. ROS1 users will not get ament, they are stuck with catkin until they migrate to ROS2. Tough luck.

That does not add up well. If ament is vastly superior yet similar to catkin, then it makes sense to allow building ROS1 packages with it as well.

On Thursday, July 9, 2015 at 3:50:44 AM UTC+2, Brian Gerkey wrote:

1. I'm worried that ROS 2 development will break my ROS 1 system.

To maintain the stability of existing ROS 1 code and avoid the risk of
regressions in ROS 1, we don't intend to backport anything significant
from ROS 2.  That's not to say that it can't happen, but our default
position is that ROS 1, including catkin, should remain as is, with
incremental improvements and bug fixes.

The main worry should be that an inevitable future transition to ROS2 will break the robot.
That means if a team is working on a 5 year project, and halfway through some
packages will start to be released only for ROS2, not for ROS1, then that team will
have to spend vast amounts of time on migrations and bridges and not on their
actual research. And just sticking with ROS1 and old libraries and old linux versions
may not be an option.
 

3. Why did OSRF create a yet another build system, instead of improving catkin?

The main question here is why ament was not build to still support ROS1 builds from the start.

From what I understand, of all improvements made to ament, the only crucial one
was the restructuring to allow ROS2 message generation. Everything else are "nice-to-have's",
that would by themselves never have motivated OSRF to invest time in the buildsystem.

Given that, the strategy could have been as follows:

1. Create a workspace with catkin on an experimental branch/fork catkin2.
2. Add ROS1 core packages and some example ROS1 packages
3. Add ROS2 core packages

4. change ROS2 packages and catkin2
5. check ROS1 still builds + works
6. if ROS2 goals not achieved or ROS1 breaks, goto 4
7. Done

This process would not have broken catkin or disturbed ROS1, and the changes
could have been kept outside the main catkin branch, or even in separate sources
like ament. Catkin as a name would have been more viable to keep around where
possible, sharing and reusing code would have been more viable. Early adopters
working on ROS1 could have started using ament/catkin2 and provide early feedback,
while working realistically with ROS1.

This process would also made it more obvious to keep single message definition packages.
It seems gruelsome that there is now common_msgs and common_interfaces, and people
worrying about how to build bridges spanning both 2 communication layers and two buildsystems.
Instead, in the workspace used in the refactoring process above, common_msgs could have
been switched to a ROS2 branch with message generators for both ROS2 and ROS1. With mappings from ROS2
to ROS1, making sure wach incremental change to the IDL can be mapped to a ROS1 message.
That way, a package like common_msgs might even not just generate bindings for ROS2 and ROS1,
but also the bridging/transformation code, obviously also belonging close to both message definitions.

In the current layout, to make any change to a message definition, it is necessary to change the
ROS2 .msg file, and the ROS1, msg file, and the bridge, and then validate the change.and then those
3 packages must be updated simultaneously in every active workspaces in the world, which for most
users means updating an amend workspace and then updating the overlaying catkin workspace.

Did OSRF actually decide that this would be the optimal strategy for introducing ROS2?

regards,
  Thibault

[Luetkebohle Ingo (CR/AEA2)]

Hi Thibault,

my interpretation is that this is less about technical concerns but about effort for the (small) team at OSRF. The following is just my interpretation, of course.

> That does not add up well. If ament is vastly superior yet similar to catkin, then it makes
> sense to allow building ROS1 packages with it as well.

If you do that, you have legacy to worry about. My interpretation of what Dirk and Brian wrote is that they don't want that. They want a clean cut, in *both* directions.

In this discussion, they emphasized one of the sides ("ROS1 should not be broken") but just consider the reason for doing ROS2 at all: It's breaking with legacy. *All* the features in ROS2 (caveat: that I know of) could have been done incremental to ROS1, but that would have been more effort. It's not completely clear how much more, but likely very much more.

From the OSRF perspective, it is not clear who would've taken on that effort, and if they would have taken it on, it would likely slow down ROS development, so they are pursuing the safe path for them, which is to get rid of legacy. That way, they can still be fast.

cheers,
Ingo

[Dirk Thomas]

On Thu, Jul 9, 2015 at 3:31 AM, Thibault Kruse <tibo...@googlemail.com> wrote:

Thanks Brian for giving the OSRF perspective.

It sill seems odd to me. Basically OSRF seems to be saying:

1. ament is *very* similar and close in source to catkin.
2. ament is vastly superior to catkin.
3. ROS1 users will not get ament, they are stuck with catkin until they migrate to ROS2. Tough luck.

That does not add up well. If ament is vastly superior yet similar to catkin, then it makes sense to allow building ROS1 packages with it as well.

The main question here is why ament was not build to still support ROS1 builds from the start.

From what I understand, of all improvements made to ament, the only crucial one
was the restructuring to allow ROS2 message generation. Everything else are "nice-to-have's",
that would by themselves never have motivated OSRF to invest time in the buildsystem.

You have brought up this concern multiple times in different threads in the last days.

And I have replied to it repeatedly.

Your misunderstanding is that ament is *not* only a restructuring to allow ROS message generation.

As mentioned before ament also has the following changes (which is not a complete list):

* it does not support the devel space anymore

* it provide ament_cmake_auto to achieve what catkin_simple was never able to deliver

These changes *require* breaking changes.

E.g. the function ament_package() has a different semantic meaning, has a different signature and is called in a different location every packages CMakeLists.txt file.

Since this would require to update every ROS 1 package in order to work and our policy is to *not* disrupt ROS 1 in such a way the feature has *not* been deployed in ROS 1 but exclusively in ROS 2.

Given that, the strategy could have been as follows:

1. Create a workspace with catkin on an experimental branch/fork catkin2.
2. Add ROS1 core packages and some example ROS1 packages
3. Add ROS2 core packages

4. change ROS2 packages and catkin2
5. check ROS1 still builds + works
6. if ROS2 goals not achieved or ROS1 breaks, goto 4
7. Done

This process would not have broken catkin or disturbed ROS1, and the changes
could have been kept outside the main catkin branch, or even in separate sources
like ament. Catkin as a name would have been more viable to keep around where
possible, sharing and reusing code would have been more viable. Early adopters
working on ROS1 could have started using ament/catkin2 and provide early feedback,
while working realistically with ROS1.

This process would also made it more obvious to keep single message definition packages.

And again you repeat your argument without considering the previous feedback.

As mentioned before:

Message packages in ROS 1 and ROS 2 require different message generators.

Those need to be listed as dependencies in the package.xml file.

Sine they must be different a single package can obviously not satisfy both ROS versions.

- Dirk

[Jack O'Quin]

On Thu, Jul 9, 2015 at 11:27 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

As mentioned before:

Message packages in ROS 1 and ROS 2 require different message generators.

Those need to be listed as dependencies in the package.xml file.

Sine they must be different a single package can obviously not satisfy both ROS versions.

This is apparently much more obvious to you than to some of us.

I can certainly imagine ways to deal with that problem. Perhaps you have already considered and rejected them all. 

The first thing that comes to mind is adding a ROS version designation to package.xml. The default would be "1". I expect that will be useful, and perhaps even necessary, for many reasons.

But, wanting to maintain the same message package names is clearly of great benefit to users. So, you should seriously consider various means of resolving that limitation. 

--

 joq

[Dirk Thomas]

On Thu, Jul 9, 2015 at 12:20 PM, Jack O'Quin <jack....@gmail.com> wrote:

On Thu, Jul 9, 2015 at 11:27 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

As mentioned before:

Message packages in ROS 1 and ROS 2 require different message generators.

Those need to be listed as dependencies in the package.xml file.

Sine they must be different a single package can obviously not satisfy both ROS versions.

This is apparently much more obvious to you than to some of us.

I can certainly imagine ways to deal with that problem. Perhaps you have already considered and rejected them all. 

I haven't considered or rejected anything I am not aware of.

Please share the ways you imagine to deal with that problem.

If you can describe you idea than I as well as others might be able to understand it and either consider it or provide feedback why it might be not possible.

 

The first thing that comes to mind is adding a ROS version designation to package.xml. The default would be "1". I expect that will be useful, and perhaps even necessary, for many reasons.

I don't understand how this helps to make a single package work with both ROS version.

How would annotating a ROS package with either a ROS 1 flag or a ROS 2 flag make a package work with both?

 

But, wanting to maintain the same message package names is clearly of great benefit to users. So, you should seriously consider various means of resolving that limitation. 

We have thought quite a bit about this.

And based on that we have describe our view of what we think is the technical limitation.

As Brian mentioned we would be happy to hear any idea how to overcome those.

- Dirk

[G.A. vd. Hoorn - 3ME]

Would it perhaps be an idea to provide some insight into which
alternatives you've already considered? Similar to how a REP (sometimes)
contains an 'Alternatives' section?

That might help increase understanding for the current approach, and
would help list members avoid coming up with things that you've already
rejected.


Gijs

[Jack O'Quin]

On Thu, Jul 9, 2015 at 2:34 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On Thu, Jul 9, 2015 at 12:20 PM, Jack O'Quin <jack....@gmail.com> wrote:

The first thing that comes to mind is adding a ROS version designation to package.xml. The default would be "1". I expect that will be useful, and perhaps even necessary, for many reasons.

I don't understand how this helps to make a single package work with both ROS version.

How would annotating a ROS package with either a ROS 1 flag or a ROS 2 flag make a package work with both?

It would permit package resolution tools to find either the ROS 1 and ROS 2 versions of a package name.

When building a ROS 2 package, the ROS 2 message packages would be used. When building special-purpose "bridge" packages, there would need to be specific CMake functions to resolve the ROS 1 message packages differently. 

I am not claiming this to be the "best" or "right" solution, just pointing out that solving that problem is not impossible.

--

 joq

[Dirk Thomas]

We are mixing two different topics in this discussion:

(1) it was requested that a single package with a single code base should be a ROS 1 as well as a ROS 2 package

(2) two message packages (one for ROS 1 and one for ROS 2) should be able to have the same name

Regarding (1):

ROS 2 want's to make breaking changes.

Some examples are:

* consider removing primitive types (char, byte)

* treat Duration / Time as messages rather then primitive types

* change the "seq" field from the Header message

Any of these changes should not be introduced into ROS 1 as it implies a breakage.

For obvious reasons the message generator will be different which are listed as dependencies in the package manifest.

Under these assumptions how should a package like "std_msgs" with a single code base be part of ROS 1 and ROS 2 at the same time?

I simply don't see a way to achieve that.

Regarding (2):

In short: finding two packages in CMake having the same name is not possible.

In long: lets go through a simple example.

We have an example package "foo_msgs" which exists in ROS 1 and ROS 2.

In CMake each package individually can found using the function "find_package()" and signals through whatever mechanism its ROS version.

The function will either use the CMAKE_PREFIX_PATH to search for the package or a custom provided list of locations.

It would be possible to write a Python script which looks for *all* packages which have that name and return a mapping like this:

* ROS 1 foo_msgs was found in: /op/ros/indigo/share/foo_msgs

* ROS 2 foo_msgs was found in: /op/ros2/xxx/share/foo_msgs

Even with that knowledge it is not possible in CMake to "find_package" both.

The first invocation does what it is supposed to do:

* find_package(foo_msgs PATHS known-path-for-ros1)

But the second invocation does not:

* find_package(foo_msgs PATHS known-path-for-ros2)

This wont find the ROS 2 version of package because CMake will reuse the information of the previously defined ROS 1 package "foo_msgs".

The order is arbitrary here - it is always the second find which fails.

Resetting individual variable in the CMake cache before the second find call to retrigger a search is also not a viable option.

It is unknown how many other packages "foo_msgs" has find_package()-ed recursively and what side effect it had.

- Dirk

--

[Thibault Kruse]

On Thursday, July 9, 2015 at 6:27:06 PM UTC+2, Dirk Thomas wrote:

On Thu, Jul 9, 2015 at 3:31 AM, Thibault Kruse <tibo...@googlemail.com> wrote:

Thanks Brian for giving the OSRF perspective.

It sill seems odd to me. Basically OSRF seems to be saying:

1. ament is *very* similar and close in source to catkin.
2. ament is vastly superior to catkin.
3. ROS1 users will not get ament, they are stuck with catkin until they migrate to ROS2. Tough luck.

That does not add up well. If ament is vastly superior yet similar to catkin, then it makes sense to allow building ROS1 packages with it as well.

The main question here is why ament was not build to still support ROS1 builds from the start.

From what I understand, of all improvements made to ament, the only crucial one
was the restructuring to allow ROS2 message generation. Everything else are "nice-to-have's",
that would by themselves never have motivated OSRF to invest time in the buildsystem.

You have brought up this concern multiple times in different threads in the last days.

And I have replied to it repeatedly.

Your misunderstanding is that ament is *not* only a restructuring to allow ROS message generation.

As mentioned before ament also has the following changes (which is not a complete list):

* it does not support the devel space anymore

* it provide ament_cmake_auto to achieve what catkin_simple was never able to deliver

These changes *require* breaking changes.

E.g. the function ament_package() has a different semantic meaning, has a different signature and is called in a different location every packages CMakeLists.txt file.

Since this would require to update every ROS 1 package in order to work and our policy is to *not* disrupt ROS 1 in such a way the feature has *not* been deployed in ROS 1 but exclusively in ROS 2.

Brian said: "this could change, e.g., by adding a 'catkin_cmake' build type to ament". I did not say anything much different from this, only that this could have been done from the start. Packages using new cmake macros introduced with ament may have extra features that a packages using the legacy catkin cmake API have not. Not supporting the devel_space anymore within an ament workspace is also acceptable.

This process would also made it more obvious to keep single message definition packages.

And again you repeat your argument without considering the previous feedback.

As mentioned before:

Message packages in ROS 1 and ROS 2 require different message generators.

Those need to be listed as dependencies in the package.xml file.

Sine they must be different a single package can obviously not satisfy both ROS versions.

I can imagine a ROS1 package called foo1 requiring common_msgs, and a ROS2 package called foo2 also requiring common_msgs. A ROS2 package named common_msgs can depend on both ROS1 and ROS2 message generators, to produce bindings for both.
Both foo1 and foo2 can depend on this common_msgs within an ament workspace. This common_msgs can be a ROS2 fork of ROS1 common_msgs, meaning the ROS1 common_msgs fork remains unchanged causing no disruption to ROS1.

What am I missing?

[Dirk Thomas]

On Thu, Jul 9, 2015 at 2:43 PM, Thibault Kruse <tibo...@googlemail.com> wrote:

This process would also made it more obvious to keep single message definition packages.

And again you repeat your argument without considering the previous feedback.

As mentioned before:

Message packages in ROS 1 and ROS 2 require different message generators.

Those need to be listed as dependencies in the package.xml file.

Sine they must be different a single package can obviously not satisfy both ROS versions.

I can imagine a ROS1 package called foo1 requiring common_msgs, and a ROS2 package called foo2 also requiring common_msgs. A ROS2 package named common_msgs can depend on both ROS1 and ROS2 message generators, to produce bindings for both.
Both foo1 and foo2 can depend on this common_msgs within an ament workspace. This common_msgs can be a ROS2 fork of ROS1 common_msgs, meaning the ROS1 common_msgs fork remains unchanged causing no disruption to ROS1.

What am I missing?

We don't want to make ROS 2 depend on ROS 1.

- Dirk

[Jack O'Quin]

On Thu, Jul 9, 2015 at 3:12 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

We are mixing two different topics in this discussion:

(1) it was requested that a single package with a single code base should be a ROS 1 as well as a ROS 2 package

I don't recall that request, but since you raise the possibility I am against it.

 

(2) two message packages (one for ROS 1 and one for ROS 2) should be able to have the same name

That is what I would like to explore. Is the rationale clear enough already?

 

Regarding (2):

In short: finding two packages in CMake having the same name is not possible.

In long: lets go through a simple example.

We have an example package "foo_msgs" which exists in ROS 1 and ROS 2.

In CMake each package individually can found using the function "find_package()" and signals through whatever mechanism its ROS version.

The function will either use the CMAKE_PREFIX_PATH to search for the package or a custom provided list of locations.

It would be possible to write a Python script which looks for *all* packages which have that name and return a mapping like this:

* ROS 1 foo_msgs was found in: /op/ros/indigo/share/foo_msgs

* ROS 2 foo_msgs was found in: /op/ros2/xxx/share/foo_msgs

Even with that knowledge it is not possible in CMake to "find_package" both.

The first invocation does what it is supposed to do:

* find_package(foo_msgs PATHS known-path-for-ros1)

But the second invocation does not:

* find_package(foo_msgs PATHS known-path-for-ros2)

This wont find the ROS 2 version of package because CMake will reuse the information of the previously defined ROS 1 package "foo_msgs".

The order is arbitrary here - it is always the second find which fails.

Resetting individual variable in the CMake cache before the second find call to retrigger a search is also not a viable option.

It is unknown how many other packages "foo_msgs" has find_package()-ed recursively and what side effect it had.

It's hard to keep track of all the ideas flying back and forth, but I did *not* suggest using find_package() directly on both message packages. My suggestion (and it was only that) was for the ROS 1 version to be referenced obliquely, via an intermediate CMake glue package, with syntax like this:

  find_package(ros1_msgs REQUIRED COMPONENTS foo_msgs std_msgs)

I want to enable creation of a ROS version bridge without forcing the name of every message to change. As a minimum, ``ros1_msgs`` might only handle message and service references, internally setting up the required CMake variables so the bridge package can somehow include old message header files from its listed COMPONENTS. Those targets would have previously been built using ROS 1 tools.

That ``ros1_msgs`` interface is probably messy to write. I don't know CMake well enough to say. But, I do believe the semantics of those references are clear, and that some similar approach could be made to work and would be adequate for our needs. You, Dirk, are far more qualified to invent a solution than I am. Please consider these and other possible approaches. I believe you can make it work.

My assumption is that except for things like the inter-operability bridge, the ROS 1 and ROS 2 worlds should be kept quite separate. I expect it will be useful to mark packages as belonging to one world or the other, and that we will need some utilities that can list the packages available for each ROS major version. 

People migrating to ROS 2 will presumably need to know whether all their package dependencies have already been converted. I suppose that will set off a new iteration of the dry-to-wet migration metaphor, but with a special exception for ROS 1 message and service definitions.

Regards,

--

 joq

[Jack O'Quin]

+1 to this idea, in general. 

There will presumably be a few exceptions for low-level package resolution tools, and where needed for various interstitial parts like the message bridge. But a pure ROS 2 system would need no ROS 1 components.

--

 joq

[Dirk Thomas]

But how should this line work internally?

I can only imagine that find_package() is invoked internally for all components.

That is e.g. exactly how find_package(catkin COMPONENTS a b c) works.

And with that we are back at the problem I described in (2).

 

- Dirk

[Jack O'Quin]

I did say it might be messy. I want to encourage you to look for solutions. Please try to image some other approach. 

Knowing that it only needs to handle messages, perhaps ``ros1_msgs`` can manually add the necessary header paths to various CMake variables and not use find_package() internally. It does not have to be pretty, it just needs to solve that one problem.

Or, maybe we should think entirely outside of this box. Is there some other completely different way to implement the message and service bridge? Mixing ROS 1 and ROS 2 objects in a single compilation seems rather unattractive, after all. 

--

 joq

[Bill Morris]

Is it a design goal of ROS2 to provide a way to play/record rosbags from
ROS1 in ROS2?
I have something like 100+TB of rosbags right now.

[Dirk Thomas]

It will very likely be the case that you play the ROS 1 bag using ROS 1 tools.

But you can use the ROS bridge to receive the data in ROS 2 and record them with the ROS 2 equivalent.

It would also be possible to write a direct conversion tool but we don't have a ROS 2 bag yet so if / when this would be available.

- Dirk

[William Woodall]

The most straight forward thing, it seems to me, would be to use rosbag from ROS 1 to playback and use the bridge to consume it in ROS 2. Using this and the recorder in ROS 2 you should be able to automate the migration, or leave the system stored in ROS 1 bag files and always access them with the rosbag/bridge combo. This could be made to be as efficient as possible by putting them all in one process. Though message will need to be deserialized in ROS 1 and converted to ROS 2 messages before being republished.

We haven't spent much time on recording and playback in ROS 2 just yet. There are some open research questions to be considered. Even if we kept the ROS 1 bag format, the serialized data would be different and presumably some of the meta information would need to be changed to support more QoS settings (more than latched=true/false).

On Thu, Jul 9, 2015 at 5:53 PM, Bill Morris <bi...@neautomation.com> wrote:

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

William Woodall

ROS Development Team

wil...@osrfoundation.org

http://wjwwood.io/

[Bill Morris]

On 07/09/2015 08:35 PM, Jack O'Quin wrote:
> Or, maybe we should think entirely outside of this box. Is there some
> other completely different way to implement the message and service
> bridge? Mixing ROS 1 and ROS 2 objects in a single compilation seems
> rather unattractive, after all.

http://wiki.ros.org/rosbag/migration
Does it make sense to generate something like migration rules for ROS2
packages (if there is an existing ROS1 package)?

This could provide enough information to implement a message/service
bridge and allow ROS1 bags to be converted to ROS1.

[Dirk Thomas]

The ROS bridge will provide that functionality.

Currently it is a hard coded ruleset but in the future it should allow custom mappings.

- Dirk

[Thibault Kruse]

 
So to begin with, you now agree that this would be feasible?
I did not mean for the message package to depend on ROS1, only for it to depend on message generators generating ROS1 bindings. This might involve forking / migrating ROS1 message generators.I understand this is a bit ugly, because it would be nice for ROS2 to remain completely agnostic about anything ROS1. Just as the bridging solutions are ugly.

But I believe the decision should consider cost and benefits, and it is not clear here whether OSRF considered both here, or makes a decision costing much more effort just for the sake of a principle.

In any case the reply "It was not possible" should be changed to "We could have, but did not want to because ...", if that is the case.

[Jack O'Quin]

My miscellaneous initial impressions of ament:

 * Getting rid of devel space is good.

 * Building each package in its own CMake context is good.
​

 * I question the necessity for symlinking source files into install space.

   - It's another difference between ROS and what programmers generally expect.

   - It is merely a convenience: people soon remember to install after changing things. 

 * Keeping current ROS 1 build tools stable is good.

 * "ament_auto" looks wonderful.

   - what are its limitations?

   - will it work for almost everything?

 * Changing the name from "catkin" to "ament" will frighten a lot of people who have bad memories of catkin migration. They'll get over it. It may help to describe it as "catkin 2.0 with some incompatibilities".

Things I would like to understand:

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

 * From Dirk's document: "catkin_pkg as well as many other ROS 1 Python packages cannot be side-by-side installed for Python 2 and Python 3 at the same time."

 * Can a ROS 1 package be built using ament, if desired? (I anticipate a future of maintaining various drivers simultaneously for both ROS versions, probably for years.)

--

 joq

[Dirk Thomas]

On Sat, Jul 11, 2015 at 2:46 PM, Jack O'Quin <jack....@gmail.com> wrote:

​

 * I question the necessity for symlinking source files into install space.

   - It's another difference between ROS and what programmers generally expect.

   - It is merely a convenience: people soon remember to install after changing things. 

It is an optional feature which is not enabled by default.

It aims to provide the same benefit as the removed devel space had.

During the development on ROS 2 in the the past I have found this feature to be very valuable as it improves my development cycle significantly.

 

 * "ament_auto" looks wonderful.

   - what are its limitations?

   - will it work for almost everything?

ament_auto uses the extension point system of ament which makes it very flexible.

I don't know what the limitations are - until know it was always possible to extend it to new use cases.

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

A package can provide a package-specific setup file to change the environment in any way necessary.

Usually that involves setting environment variable like PATH or PYTHONPATH.

That is necessary to find the executables or Python packages in the install space (if it is not /usr).

But basically a package can set whatever it needs to in order to make it's install space fully usable.

A package can provide whatever setup scripts it wants to - currently these are shell scripts and Windows batch files.

The approach works currently well on Linux, OS X and Windows.

 * From Dirk's document: "catkin_pkg as well as many other ROS 1 Python packages cannot be side-by-side installed for Python 2 and Python 3 at the same time."

It is not possible to install e.g. python-catkin-pkg as well as python3-catkin-pkg at the same time since both packages provide the same executables, e.g. /usr/bin/catkin_create_pkg.

 * Can a ROS 1 package be built using ament, if desired? (I anticipate a future of maintaining various drivers simultaneously for both ROS versions, probably for years.)

Are you referring to build an existing ROS 1 package using the ament command line tool?

ament_tools uses different environment and CMake variables.

It also only supports the format 2 of the package.xml spec and many ROS 1 packages use format 1.

Therefore I don't think the ament command line tool it is able to build ROS 1 packages but I have never tried it.

Or are you asking about using ament_cmake instead of catkin in the CMakeLists.txt files in ROS 1 packages?

I haven't tried that because our goal is to not backport ROS 2 features to ROS 1.

- Dirk

[Jack O'Quin]

On Sat, Jul 11, 2015 at 11:27 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On Sat, Jul 11, 2015 at 2:46 PM, Jack O'Quin <jack....@gmail.com> wrote:

​

 * "ament_auto" looks wonderful.

   - what are its limitations?

   - will it work for almost everything?

ament_auto uses the extension point system of ament which makes it very flexible.

I don't know what the limitations are - until know it was always possible to extend it to new use cases.

What is the extension point system? Where can I find out more about it?

 

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

A package can provide a package-specific setup file to change the environment in any way necessary.

Usually that involves setting environment variable like PATH or PYTHONPATH.

That is necessary to find the executables or Python packages in the install space (if it is not /usr).

But basically a package can set whatever it needs to in order to make it's install space fully usable.

A package can provide whatever setup scripts it wants to - currently these are shell scripts and Windows batch files.

The approach works currently well on Linux, OS X and Windows.

Does every package need to include shell scripts and Windows batch files?

Is there a link to some typical example? 

 * Can a ROS 1 package be built using ament, if desired? (I anticipate a future of maintaining various drivers simultaneously for both ROS versions, probably for years.)

Are you referring to build an existing ROS 1 package using the ament command line tool?

ament_tools uses different environment and CMake variables.

It also only supports the format 2 of the package.xml spec and many ROS 1 packages use format 1.

Therefore I don't think the ament command line tool it is able to build ROS 1 packages but I have never tried it.

I am wondering how far apart the ROS 1 and ROS 2 versions of my packages need to be, and how much trouble that is going to be for people who maintain drivers and other commonly-used packages.

Migrating to package format 2 is easy, and works fine with ROS 1. The rest of it may not have been considered much, yet. 

 

Or are you asking about using ament_cmake instead of catkin in the CMakeLists.txt files in ROS 1 packages?

I haven't tried that because our goal is to not backport ROS 2 features to ROS 1.

That is really a goal of the project? 

How did that come to be a requirement?

--

 joq

[Dirk Thomas]

On Sun, Jul 12, 2015 at 11:20 AM, Jack O'Quin <jack....@gmail.com> wrote:

On Sat, Jul 11, 2015 at 11:27 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On Sat, Jul 11, 2015 at 2:46 PM, Jack O'Quin <jack....@gmail.com> wrote:

​

 * "ament_auto" looks wonderful.

   - what are its limitations?

   - will it work for almost everything?

ament_auto uses the extension point system of ament which makes it very flexible.

I don't know what the limitations are - until know it was always possible to extend it to new use cases.

What is the extension point system? Where can I find out more about it?

Extension points are a design pattern e.g. used heavily in Eclipse.

In short:

One code part declares an extension point and "invokes" it:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_execute_extensions.cmake

Other code parts register a callback for an extension points which gets called when the extension point is being invoked:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_register_extension.cmake

You will find numerous uses of these two functions throughout the CMake code of ROS 2.

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

A package can provide a package-specific setup file to change the environment in any way necessary.

Usually that involves setting environment variable like PATH or PYTHONPATH.

That is necessary to find the executables or Python packages in the install space (if it is not /usr).

But basically a package can set whatever it needs to in order to make it's install space fully usable.

A package can provide whatever setup scripts it wants to - currently these are shell scripts and Windows batch files.

The approach works currently well on Linux, OS X and Windows.

Does every package need to include shell scripts and Windows batch files?

If its installed files require any kind of environment variable: yes.

Is there a link to some typical example?

If a package is not a plain Python packages but uses CMake to install a Python package it uses the following function:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/cmake/ament_python_install_package.cmake#L24

This will then ensure to create an environment hook which extends the PYTHONPATH:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/ament_cmake_python-extras.cmake#L18

 

Or are you asking about using ament_cmake instead of catkin in the CMakeLists.txt files in ROS 1 packages?

I haven't tried that because our goal is to not backport ROS 2 features to ROS 1.

That is really a goal of the project? 

How did that come to be a requirement?

Please see the second message from Brian in this thread for a rational behind that decision:

https://groups.google.com/d/msg/ros-sig-ng-ros/suTQfcddeh8/kjaS7BKFDukJ

- Dirk

[Jack O'Quin]

On Mon, Jul 13, 2015 at 12:41 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On Sun, Jul 12, 2015 at 11:20 AM, Jack O'Quin <jack....@gmail.com> wrote:

What is the extension point system? Where can I find out more about it?

Extension points are a design pattern e.g. used heavily in Eclipse.

In short:

One code part declares an extension point and "invokes" it:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_execute_extensions.cmake

Other code parts register a callback for an extension points which gets called when the extension point is being invoked:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_register_extension.cmake

You will find numerous uses of these two functions throughout the CMake code of ROS 2.

I am looking for examples of how and for what it's used, not just the implementation.

 

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

A package can provide a package-specific setup file to change the environment in any way necessary.

Usually that involves setting environment variable like PATH or PYTHONPATH.

That is necessary to find the executables or Python packages in the install space (if it is not /usr).

But basically a package can set whatever it needs to in order to make it's install space fully usable.

A package can provide whatever setup scripts it wants to - currently these are shell scripts and Windows batch files.

The approach works currently well on Linux, OS X and Windows.

Does every package need to include shell scripts and Windows batch files?

If its installed files require any kind of environment variable: yes.

Trying to understand the macros and functions you linked, it appears that *maybe* things like ament_python_install_package() provide all the necessary shell scripts or batch files for pure Python packages. 

Is that right? If so, then most packages would not need to create their own scripts. That seems highly desirable. 

Is there a link to some typical example?

If a package is not a plain Python packages but uses CMake to install a Python package it uses the following function:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/cmake/ament_python_install_package.cmake#L24

This will then ensure to create an environment hook which extends the PYTHONPATH:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/ament_cmake_python-extras.cmake#L18

Again, I am looking for a user perspective, not the bottom-up implementation details. I hope the Python case is similar to using catkin_install_python(). That would be fine. Is there a similar interface for C++?

I haven't tried that because our goal is to not backport ROS 2 features to ROS 1.

That is really a goal of the project? 

How did that come to be a requirement?

Please see the second message from Brian in this thread for a rational behind that decision:

https://groups.google.com/d/msg/ros-sig-ng-ros/suTQfcddeh8/kjaS7BKFDukJ

I read Brian's message pretty carefully when he first posted it, and again today. I suppose you are referring to this part:

1. I'm worried that ROS 2 development will break my ROS 1 system.

To maintain the stability of existing ROS 1 code and avoid the risk of

regressions in ROS 1, we don't intend to backport anything significant

from ROS 2.  That's not to say that it can't happen, but our default

position is that ROS 1, including catkin, should remain as is, with

incremental improvements and bug fixes.

I hate to be pedantic, but we do need to speak clearly about our goals and requirements. The most important thing for this SIG to accomplish is a strong consensus on what we collectively want. 

Brian says the goal is to maintain ROS 1 release and development stability. I think we all agree with that. In pursuit of that goal, he mentions a lack of intention to do significant backports to ROS 1, while admitting that it could happen. 

So, not doing backports is a means to preserving stability, not a goal in itself.

I concur with that, but worry slightly that you seem to consider not backporting as a another separate objective. 

Why does that matter? Because there are other goals, like ease of migration, that are important, but not yet clearly understood. Allowing ROS 1 packages to someday be built using ament tools may or may not help with that. I doubt anyone knows. But, we should not be making design or implementation decisions based on the notion that we can never do it. Surely, some backports of new tools can be done without affecting any existing ROS 1 code. So, exceptions to the rule are possible, subject to good engineering judgement.

I am not asking for anything different here and now, but I do want to permit some flexibility as the design evolves.

--

 joq

[Dirk Thomas]

On Mon, Jul 13, 2015 at 10:34 AM, Jack O'Quin <jack....@gmail.com> wrote:

On Mon, Jul 13, 2015 at 12:41 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On Sun, Jul 12, 2015 at 11:20 AM, Jack O'Quin <jack....@gmail.com> wrote:

What is the extension point system? Where can I find out more about it?

Extension points are a design pattern e.g. used heavily in Eclipse.

In short:

One code part declares an extension point and "invokes" it:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_execute_extensions.cmake

Other code parts register a callback for an extension points which gets called when the extension point is being invoked:

https://github.com/ament/ament_cmake/blob/master/ament_cmake_core/cmake/core/ament_register_extension.cmake

You will find numerous uses of these two functions throughout the CMake code of ROS 2.

I am looking for examples of how and for what it's used, not just the implementation.

Please search the source code for these two function names to find where and how extensions are being used.

The system currently has 6 extension points with 24 extensions which cover the following functionality:

* to register generators for the code generation step (C, C++, DDS-IDL, various type support packages)

* in ament_package to hook into the CMake config generation process (exporting include dirs, libraries, dependencies, definitions)

* in ament_lint_auto to register linters

* in ament_cmake_gtest/gmock to allow alternative ways to find gtest

 * What does environment creation involve? 

   - When and why would a package need to provide its own shell script for this? 

   - How portable is that going to be?

A package can provide a package-specific setup file to change the environment in any way necessary.

Usually that involves setting environment variable like PATH or PYTHONPATH.

That is necessary to find the executables or Python packages in the install space (if it is not /usr).

But basically a package can set whatever it needs to in order to make it's install space fully usable.

A package can provide whatever setup scripts it wants to - currently these are shell scripts and Windows batch files.

The approach works currently well on Linux, OS X and Windows.

Does every package need to include shell scripts and Windows batch files?

If its installed files require any kind of environment variable: yes.

Trying to understand the macros and functions you linked, it appears that *maybe* things like ament_python_install_package() provide all the necessary shell scripts or batch files for pure Python packages. 

Is that right? If so, then most packages would not need to create their own scripts. That seems highly desirable. 

Yes, that is currently the case.

If the userland CMake code calls a function which will require an environment variable the function will take care of it for the user.

Is there a link to some typical example?

If a package is not a plain Python packages but uses CMake to install a Python package it uses the following function:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/cmake/ament_python_install_package.cmake#L24

This will then ensure to create an environment hook which extends the PYTHONPATH:

https://github.com/ament/ament_cmake/blob/625340b083bbdafc8b904d220bb0b12499b221fe/ament_cmake_python/ament_cmake_python-extras.cmake#L18

Again, I am looking for a user perspective, not the bottom-up implementation details. I hope the Python case is similar to using catkin_install_python(). That would be fine. Is there a similar interface for C++?

What environment variables do you have in mind for C++?

Python requires the PYTHONPATH - C++ does not have an equivalent.

If you export libraries it can set the LD_LIBRARY_PATH accordingiy.

I haven't tried that because our goal is to not backport ROS 2 features to ROS 1.

That is really a goal of the project? 

How did that come to be a requirement?

Please see the second message from Brian in this thread for a rational behind that decision:

https://groups.google.com/d/msg/ros-sig-ng-ros/suTQfcddeh8/kjaS7BKFDukJ

I read Brian's message pretty carefully when he first posted it, and again today. I suppose you are referring to this part:

1. I'm worried that ROS 2 development will break my ROS 1 system.

To maintain the stability of existing ROS 1 code and avoid the risk of

regressions in ROS 1, we don't intend to backport anything significant

from ROS 2.  That's not to say that it can't happen, but our default

position is that ROS 1, including catkin, should remain as is, with

incremental improvements and bug fixes.

I hate to be pedantic, but we do need to speak clearly about our goals and requirements. The most important thing for this SIG to accomplish is a strong consensus on what we collectively want. 

Brian says the goal is to maintain ROS 1 release and development stability. I think we all agree with that. In pursuit of that goal, he mentions a lack of intention to do significant backports to ROS 1, while admitting that it could happen. 

So, not doing backports is a means to preserving stability, not a goal in itself.

I concur with that, but worry slightly that you seem to consider not backporting as a another separate objective. 

Why does that matter? Because there are other goals, like ease of migration, that are important, but not yet clearly understood. Allowing ROS 1 packages to someday be built using ament tools may or may not help with that. I doubt anyone knows. But, we should not be making design or implementation decisions based on the notion that we can never do it. Surely, some backports of new tools can be done without affecting any existing ROS 1 code. So, exceptions to the rule are possible, subject to good engineering judgement.

I am not asking for anything different here and now, but I do want to permit some flexibility as the design evolves.

Back porting features will always have the inherent risk of regressions.

No matter how careful the developers are.

And since we want to minimize that risk we don't intend to back port features.

And if a ROS 2 feature would make sense in ROS 1 (since it is a nice-to-have) we will likely still not spend any time on it simply due to resource constraints.

That being said it is always feasible to implement non-breaking changes in ROS 1 which makes the transition path to ROS 1 easier.

If e.g. additional information in message packages would make the bridge easier we would certainly consider adding them to ROS 1.

- Dirk