preliminary review of task-oriented catkin user docs

[Jack O'Quin]

I've been complaining about the lack of user-oriented catkin
documentation. In an effort to do something constructive, I made a
fork of the ros/catkin repo, and began writing the documentation I
wished I had before I started converting a bunch of rosbuild packages.

There are advantages and disadvantages to that approach. At least it
is something that would have been useful to someone, namely me. On the
other hand, I am hardly a catkin expert and have probably gotten some
things wrong.

So, I really need feedback from you to find the errors. There are
specific questions marked "todo::" within the text. Suggestions for
additions and improvements are always welcome.

I posted a formatted copy here for your review:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/index.html

Please respond by reply, or add comments to catkin issue #410:

https://github.com/ros/catkin/issues/410

Once this looks reasonably solid, I plan to open a formal doc review
with the full ROS community.
--
joq

[William Woodall]

First of all, thanks a lot for taking the time to formalize this information into an accessible format. We work hard to stay on top the maintenance tasks and balance our other obligations here at OSRF, but we don't get to spend the time we would like making things easier to use. I appreciate your efforts here.

I really think this is some awesome documentation, I have read through it all and I found it quite accurate. There are a few things I noticed, and hopefully this can help you when finishing up the docs:

For setting the version of shared libraries, this is purely a CMake thing, see:

From http://www.cmake.org/cmake/help/v2.8.10/cmake.html#command:set_target_properties:

For shared libraries VERSION and SOVERSION can be used to specify the build version and api version respectively. When building or installing appropriate symlinks are created if the platform supports symlinks and the linker supports so-names. If only one of both is specified the missing is assumed to have the same version number. For executables VERSION can be used to specify the build version. When building or installing appropriate symlinks are created if the platform supports symlinks. For shared libraries and executables on Windows the VERSION attribute is parsed to extract a "major.minor" version number. These numbers are used as the image version of the binary.

That is referenced from:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_libraries.html#building-libraries

For the question about run_depend on msg packages:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html

(related) http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html

It is true that for C++ programs, you technically only need the message packages at build time, because for the ROS serialization system C++ generated code is header only.

I would consider not mentioning this, or recommending against it though, because for Python you need the message packages at runtime, and if the user trys to use ROS tools like rosmsg to find and introspect the message types which compose the messages used in your program they need to be installed. So for these reasons I would say that we should recommend that you always build and run depend on those message packages (At least until we have msg_pkg and msg_pkg-dev).

Additionally from that page, you should mention that when building a target which uses the generated headers from message packages you should add_dependencies like this:

add_dependencies(your_program ${your_package_EXPORTED_TARGETS})

Which you actually point out here: http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html

On this page:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html

Generation of services is done by `generate_messages(...)`, there is no `generate_services(...)` command.

I am still reviewing the docs, and I encourage others who are less experienced with catkin to review it to help cover the less obvious parts.

Again; Awesome work, thanks a lot Jack!

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

[Jonathan Bohren]

On Wed, May 1, 2013 at 3:00 PM, William Woodall <wil...@osrfoundation.org> wrote:

Again; Awesome work, thanks a lot Jack!

This is great!

Also, I realized at one point that it was really hard to find the full CMake macro API documentation, so I added a link to it here: http://www.ros.org/wiki/catkin/CMakeLists.txt?action=diff&rev1=51&rev2=52 

-j

--

Jonathan Bohren

Laboratory for Computational Sensing and Robotics

http://dscl.lcsr.jhu.edu/People/JonathanBohren

[Dirk Thomas]

Since recently this is correctly linked from within the API doc:

http://www.ros.org/wiki/catkin -> Code API -> Extracted CMake API reference

- Dirk


On 01.05.2013 13:33, Jonathan Bohren wrote:

[Jonathan Bohren]

On Wed, May 1, 2013 at 4:42 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

Since recently this is correctly linked from within the API doc:

http://www.ros.org/wiki/catkin -> Code API -> Extracted CMake API reference

Ah cool. 

There's also this other simplified information here: http://www.ros.org/wiki/catkin/CMakeLists.txt#catkin_package.28.29

Maybe per-function links should be dispersed throughout that wiki page to make it easier to find the full API for some of those macros? 

[Jack O'Quin]

On Wed, May 1, 2013 at 3:48 PM, Jonathan Bohren
<jonatha...@gmail.com> wrote:
> On Wed, May 1, 2013 at 4:42 PM, Dirk Thomas <dth...@osrfoundation.org>
> wrote:
>>
>> Since recently this is correctly linked from within the API doc:
>>
>> http://www.ros.org/wiki/catkin -> Code API -> Extracted CMake API
>> reference

The page Dirk linked is extracted from the source files. It is mainly
written for catkin developers, and includes internal commands and
macros.

> There's also this other simplified information here:
> http://www.ros.org/wiki/catkin/CMakeLists.txt#catkin_package.28.29
>
> Maybe per-function links should be dispersed throughout that wiki page to
> make it easier to find the full API for some of those macros?

I would like to have a hand-written, formal API definition for all the
stable, supported interfaces. Since they would only change with major
catkin version updates, they could be linked-to from everywhere else.
--
joq

[Jonathan Bohren]

Yeah, that's reasonable, I modified it like so: http://www.ros.org/wiki/catkin/CMakeLists.txt?action=diff&rev1=52&rev2=53

[Dirk Thomas]

We can easily split the generated API page into public and internal but still generating them both from the CMake code.
The major point for doing it that way instead of maintaining a separate file is that this one will unavoidably run out of sync.

- Dirk

[Jack O'Quin]

The implementation is not allowed to get out of sync with the public
API. That would break user packages.

An implementation is not a specification. The API is.
--
joq

[Dirk Thomas]

We do not provide or write a "specification for the build system".
We are documenting an API which is of course intended to be stable.

But just one example where having separate documents will be an issue:
We add a new option to one of the catkin macros - perfectly possible in order to add new features even with a stable API.
That makes the both documents be out-of-sync immediately.

- Dirk

[Jack O'Quin]

On Wed, May 1, 2013 at 5:01 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:
> On 01.05.2013 14:49, Jack O'Quin wrote:
>>
>> On Wed, May 1, 2013 at 4:39 PM, Dirk Thomas <dth...@osrfoundation.org>
>> wrote:
>>
>>> On 01.05.2013 14:26, Jack O'Quin wrote:
>> An implementation is not a specification. The API is.
>
> We do not provide or write a "specification for the build system".

I know. That's a problem I would like to fix.

> We are documenting an API which is of course intended to be stable.
>
> But just one example where having separate documents will be an issue:
> We add a new option to one of the catkin macros - perfectly possible in
> order to add new features even with a stable API.
> That makes the both documents be out-of-sync immediately.

You can't just change this stuff any time you feel like it.

Suppose we decide to make a backwards-compatible extension. The spec
needs to be updated for the new feature, and also for when it became
available. Then it can be reviewed.

People working with older versions need to know that they can't use it yet.
--
joq

[Dirk Thomas]

Exactly that kind of information is what is supposed to be in an API documentation.
That is common practice in other project: e.g. http://javadoc.jenkins-ci.org/hudson/model/AbstractBuild.html#setBuiltOnStr(java.lang.String)
The API doc states when new features/members/functions/arguments have been introduces.

So I totally agree with you to include all these information - but within in the code comment.
Also that changes should be reviewed before they are applied - that's what we use pull requests for.

- Dirk

[Jack O'Quin]

On Wed, May 1, 2013 at 2:00 PM, William Woodall
<wil...@osrfoundation.org> wrote:
> For setting the version of shared libraries, this is purely a CMake thing,
> see:
>
> From
> http://www.cmake.org/cmake/help/v2.8.10/cmake.html#command:set_target_properties:
>
> For shared libraries VERSION and SOVERSION can be used to specify the build
> version and api version respectively. When building or installing
> appropriate symlinks are created if the platform supports symlinks and the
> linker supports so-names. If only one of both is specified the missing is
> assumed to have the same version number. For executables VERSION can be used
> to specify the build version. When building or installing appropriate
> symlinks are created if the platform supports symlinks. For shared libraries
> and executables on Windows the VERSION attribute is parsed to extract a
> "major.minor" version number. These numbers are used as the image version of
> the binary.
>
> That is referenced from:
>
> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_libraries.html#building-libraries

Thanks. That is what I was looking to find.

I doubt anyone is using SOVERSION numbers right now. The issues of
properly assigning numbers for API and ABI version changes are
somewhat complex.

Should we ignore that for the moment? The library dependencies
sections do not mention how to specify version dependencies, either. I
am considering writing a more advanced how-to later on, to avoid
complicating the basic information. This stuff is already complex
enough.

> For the question about run_depend on msg packages:
>
> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html
> (related)
> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html
>
> It is true that for C++ programs, you technically only need the message
> packages at build time, because for the ROS serialization system C++
> generated code is header only.
>
> I would consider not mentioning this, or recommending against it though,
> because for Python you need the message packages at runtime, and if the user
> trys to use ROS tools like rosmsg to find and introspect the message types
> which compose the messages used in your program they need to be installed.
> So for these reasons I would say that we should recommend that you always
> build and run depend on those message packages (At least until we have
> msg_pkg and msg_pkg-dev).

The rosmsg example is a good reason. I'll recommend both types of dependencies.

We still lack a clean way to build separate binary and -dev packages.

> Additionally from that page, you should mention that when building a target
> which uses the generated headers from message packages you should
> add_dependencies like this:
>
> add_dependencies(your_program ${your_package_EXPORTED_TARGETS})
>
> Which you actually point out here:
> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html

As I understand it, that command is only necessary for packages that
both build messages and use them. That's why I mention it under
building messages, rather than in the general C++ library section.

Does that make sense?

I don't mind repeating information when it applies to several tasks.
The intent is to provide enough information in each page for
performing that specific task.

> On this page:
>
> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html
>
> Generation of services is done by `generate_messages(...)`, there is no
> `generate_services(...)` command.

Oops! Good catch, William.

> I am still reviewing the docs, and I encourage others who are less
> experienced with catkin to review it to help cover the less obvious parts.

Yes. In addition to finding my many errors, I need people to tell me
whether the information is pitched at the right level. Does it make
sense? Could you use it with your own packages?

Thanks to all who have responded already. I am making updates, will
republish in a day or two. I may have follow-up questions on some
points.
--
joq

[Dirk Thomas]

On 02.05.2013 05:32, Jack O'Quin wrote:
> On Wed, May 1, 2013 at 2:00 PM, William Woodall
> <wil...@osrfoundation.org> wrote:
>>
>
>> For the question about run_depend on msg packages:
>>
>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html
>> (related)
>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html
>>
>> It is true that for C++ programs, you technically only need the message
>> packages at build time, because for the ROS serialization system C++
>> generated code is header only.

That is only correct if a package will never have other packages depending on it.
As soon as another package includes headers transitively include generated headers from the message package you also need a run_depend.
Therefore it should always be stated as a run dependency to not prevent that other packages can depend on a package (without explicitly pull in the package dependencies).

>> Additionally from that page, you should mention that when building a target
>> which uses the generated headers from message packages you should
>> add_dependencies like this:
>>
>> add_dependencies(your_program ${your_package_EXPORTED_TARGETS})
>>
>> Which you actually point out here:
>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html
>
> As I understand it, that command is only necessary for packages that
> both build messages and use them. That's why I mention it under
> building messages, rather than in the general C++ library section.
>
> Does that make sense?

It does not only apply to packages having both.
It is also valid if a package A depends on the generated code for messages/service/actions/dynamic reconfigure from package B.
As long as you build each package separately (like standard CMake) it works without.
But when building in one workspace this explicit dependency is required to prevent building a lib/exe in A which includes generated code which has not been generated yet.
So for any target in A which depends on generated code that explicit add_dependencies() is requires.

- Dirk

[Jack O'Quin]

On Thu, May 2, 2013 at 12:07 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:
> On 02.05.2013 05:32, Jack O'Quin wrote:
>>
>> On Wed, May 1, 2013 at 2:00 PM, William Woodall
>> <wil...@osrfoundation.org> wrote:
>>>
>>>
>>
>>> For the question about run_depend on msg packages:
>>>
>>>
>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html
>>> (related)
>>>
>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html
>>>
>>> It is true that for C++ programs, you technically only need the message
>>> packages at build time, because for the ROS serialization system C++
>>> generated code is header only.
>
>
> That is only correct if a package will never have other packages depending
> on it.
> As soon as another package includes headers transitively include generated
> headers from the message package you also need a run_depend.
> Therefore it should always be stated as a run dependency to not prevent that
> other packages can depend on a package (without explicitly pull in the
> package dependencies).

I am still trying to understand how nested C++ headers can cause a
run-time dependency.

Is that *only* true for nested headers generated from messages?

>>> Additionally from that page, you should mention that when building a
>>> target
>>> which uses the generated headers from message packages you should
>>> add_dependencies like this:
>>>
>>> add_dependencies(your_program ${your_package_EXPORTED_TARGETS})
>>>
>>> Which you actually point out here:
>>>
>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html
>>
>>
>> As I understand it, that command is only necessary for packages that
>> both build messages and use them. That's why I mention it under
>> building messages, rather than in the general C++ library section.
>>
>> Does that make sense?
>
> It does not only apply to packages having both.
> It is also valid if a package A depends on the generated code for
> messages/service/actions/dynamic reconfigure from package B.
> As long as you build each package separately (like standard CMake) it works
> without.
> But when building in one workspace this explicit dependency is required to
> prevent building a lib/exe in A which includes generated code which has not
> been generated yet.
> So for any target in A which depends on generated code that explicit
> add_dependencies() is requires.

Ouch!

I had not seen that documented anywhere. Will definitely add it.
--
joq

[Dirk Thomas]

On 02.05.2013 15:05, Jack O'Quin wrote:
> On Thu, May 2, 2013 at 12:07 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:
>> On 02.05.2013 05:32, Jack O'Quin wrote:
>>>
>>> On Wed, May 1, 2013 at 2:00 PM, William Woodall
>>> <wil...@osrfoundation.org> wrote:
>>>>
>>>>
>>>
>>>> For the question about run_depend on msg packages:
>>>>
>>>>
>>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html
>>>> (related)
>>>>
>>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html
>>>>
>>>> It is true that for C++ programs, you technically only need the message
>>>> packages at build time, because for the ROS serialization system C++
>>>> generated code is header only.
>>
>>
>> That is only correct if a package will never have other packages depending
>> on it.
>> As soon as another package includes headers transitively include generated
>> headers from the message package you also need a run_depend.
>> Therefore it should always be stated as a run dependency to not prevent that
>> other packages can depend on a package (without explicitly pull in the
>> package dependencies).
>
> I am still trying to understand how nested C++ headers can cause a
> run-time dependency.
>
> Is that *only* true for nested headers generated from messages?

No, this affects all packages with C++ headers.

Consider the follow example:
A is a package with a.h.
B is a package with b.h which includes a.h
C is a package with c.cpp which includes b.h

C build_depends on B.
Since B exposes the headers of A (in contrast to only including them in cpp files) they must be available for C.

Build time of C does not mean that it is also build time of B.
In order to build C all direct build dependencies are installed - only B.
When B is installed e.g. from a Debian package it naturally pulls in all its (recursive) run dependencies.

Therefore B has a run_dependency on A and has to declare A as CATKIN_DEPENDS in catkin_package() to expose it to packages like C.

- Dirk

[Jack O'Quin]

Thanks! I finally get it.

We've really got a lot of work ahead of us to ever split ROS packages
into separate run-time and build-time Debian packages. I had thought
we were closer than that.
--
joq

[Jonathan Bohren]

On Thu, May 2, 2013 at 6:05 PM, Jack O'Quin <jack....@gmail.com> wrote:

>>> For the question about run_depend on msg packages:
>>>
>>>
>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/cpp_msg_dependencies.html
>>> (related)
>>>
>>> http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/python_msg_dependencies.html
>>>
>>> It is true that for C++ programs, you technically only need the message
>>> packages at build time, because for the ROS serialization system C++
>>> generated code is header only.
>
>
> That is only correct if a package will never have other packages depending
> on it.
> As soon as another package includes headers transitively include generated
> headers from the message package you also need a run_depend.
> Therefore it should always be stated as a run dependency to not prevent that
> other packages can depend on a package (without explicitly pull in the
> package dependencies).

I am still trying to understand how nested C++ headers can cause a
run-time dependency.

Is that *only* true for nested headers generated from messages?

This is one instance of catkin nomenclature that I find really counter-intuitive. "run" dependencies are actually "use" dependencies. So a package is in "run" dependency mode, for example, when another package is using any sort of resources in that package, including header files at build time.

[Jack O'Quin]

Good point.

For me, it had the unfortunate effect of fooling me into thinking I
knew what it meant, when in fact I did not. Had it been named
something meaningless, like <fubar_depend>, I would have recognized my
ignorance much sooner.

As so often happens: it's not what I don't know that gets me in
trouble, it's what I think I know that is wrong.
--
joq

[Jonathan Bohren]

That's exactly what happened to me when I was first trying to figure out catkin dependencies. It was also something that Daniel Stonier brought up 6 months ago, see the discussion here: https://github.com/ros/catkin/issues/260

-j

[Dirk Thomas]

On 02.05.2013 16:32, Jonathan Bohren wrote:

> On Thu, May 2, 2013 at 7:20 PM, Jack O'Quin <jack....@gmail.com <mailto:jack....@gmail.com>> wrote:
>
> On Thu, May 2, 2013 at 6:08 PM, Jonathan Bohren

> <jonatha...@gmail.com <mailto:jonatha...@gmail.com>> wrote:

> >
> > On Thu, May 2, 2013 at 6:05 PM, Jack O'Quin <jack....@gmail.com <mailto:jack....@gmail.com>> wrote:
>
> >> I am still trying to understand how nested C++ headers can cause a
> >> run-time dependency.
> >>
> >> Is that *only* true for nested headers generated from messages?
> >
> > This is one instance of catkin nomenclature that I find really
> > counter-intuitive. "run" dependencies are actually "use" dependencies. So a
> > package is in "run" dependency mode, for example, when another package is
> > using any sort of resources in that package, including header files at build
> > time.
>
> Good point.
>
> For me, it had the unfortunate effect of fooling me into thinking I
> knew what it meant, when in fact I did not. Had it been named
> something meaningless, like <fubar_depend>, I would have recognized my
> ignorance much sooner.
>
> As so often happens: it's not what I don't know that gets me in
> trouble, it's what I think I know that is wrong.
>
>
> That's exactly what happened to me when I was first trying to figure out catkin dependencies. It was also something that Daniel Stonier brought up 6 months ago, see the discussion here:
> https://github.com/ros/catkin/issues/260

And the conclusion on that issue is likely still valid:

"As much as the naming is not exactly right, I don't think that any single word is going to exactly capture the intent."

So if the issue here is only that the semantic of the existing tags and terms is not clear enough than the documentation must provide a better explanation.

- Dirk

[Jonathan Bohren]

On Thu, May 2, 2013 at 7:36 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 02.05.2013 16:32, Jonathan Bohren wrote:

On Thu, May 2, 2013 at 7:20 PM, Jack O'Quin <jack....@gmail.com <mailto:jack....@gmail.com>> wrote:

    On Thu, May 2, 2013 at 6:08 PM, Jonathan Bohren

    <jonatha...@gmail.com <mailto:jonathan.bohren@gmail.com>> wrote:

     >
     > On Thu, May 2, 2013 at 6:05 PM, Jack O'Quin <jack....@gmail.com <mailto:jack....@gmail.com>> wrote:

     >> I am still trying to understand how nested C++ headers can cause a
     >> run-time dependency.
     >>
     >> Is that *only* true for nested headers generated from messages?
     >
     > This is one instance of catkin nomenclature that I find really
     > counter-intuitive. "run" dependencies are actually "use" dependencies. So a
     > package is in "run" dependency mode, for example, when another package is
     > using any sort of resources in that package, including header files at build
     > time.

    Good point.

    For me, it had the unfortunate effect of fooling me into thinking I
    knew what it meant, when in fact I did not. Had it been named
    something meaningless, like <fubar_depend>, I would have recognized my
    ignorance much sooner.

    As so often happens: it's not what I don't know that gets me in
    trouble, it's what I think I know that is wrong.


That's exactly what happened to me when I was first trying to figure out catkin dependencies. It was also something that Daniel Stonier brought up 6 months ago, see the discussion here:
https://github.com/ros/catkin/issues/260

And the conclusion on that issue is likely still valid:

"As much as the naming is not exactly right, I don't think that any single word is going to exactly capture the intent."

So if the issue here is only that the semantic of the existing tags and terms is not clear enough than the documentation must provide a better explanation.

I think Jack makes a good point, though, if it were "<foo_depend>" or "<use_depend>" or "<dirk_depend>" or "<depend>" then people wouldn't make any assumptions about it, and they would at least seek out the documentation. There is a LOT of documentation for catkin, and this is still going to require people go looking for the documentation for "<run_depend>". But when people get a missing header error, they are not immediately going to go look up the meaning "<run_depend>" and how it relates to each of their (possibly many) dependencies. I think you can add this to a "catkin troubleshooting" page, but I don't think "including a header file" should be something that requires troubleshooting.

My opinion is that the catkin naming is misleading and is a design flaw, since I think the majority of people assume that "build" and "run" time are defined as follows:

build time: when I'm compiling my programs

run time: when I'm running my programs

[Austin Hendrix]

If there's a problem here, I think it's not that run_depend is
misnamed, but that the build system isn't generating the dependencies
in the debs correctly. (someone who knows the debbuild process better
than I do should confirm this)

If we're building debs that we expect people to build against (we do),
those debs should depend on all of their build-time and run-time
dependencies.

If we want to build separate binary and development debs later, then
we can figure out which packages should depend on build vs run
dependencies.

-Austin

[Dirk Thomas]

On 02.05.2013 17:44, Austin Hendrix wrote:
> If there's a problem here, I think it's not that run_depend is
> misnamed, but that the build system isn't generating the dependencies
> in the debs correctly. (someone who knows the debbuild process better
> than I do should confirm this)
>
> If we're building debs that we expect people to build against (we do),
> those debs should depend on all of their build-time and run-time
> dependencies.
>
> If we want to build separate binary and development debs later, then
> we can figure out which packages should depend on build vs run
> dependencies.

This simplification is not sufficient.
If a package B build depends on C it does not necessarily imply that A building against B requires C.

* A build dependency can be "internal", e.g. linking a static library or using header files only in .cpp files.
* Or a build dependency can be "exposed", e.g. linking a dynamic library or using header files in .h files.

Just the fact that B build depends on C does not allow you to derive what C is for A.

And exactly that is the reason why packages like e.g. Debian are hand-crafted - the maintainer uses his knowledge to decides what goes into the dev and non-dev package and what dependencies they both
have.

As long as we do not build separate dev and non-dev packages this is the semantic difference between build_depends and run_depends.
When we would want to go towards building separate dev and non-dev packages we will for sure need more metainformation to distinguish the dependency information...

- Dirk

[Jonathan Bohren]

On Thu, May 2, 2013 at 8:55 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

When we would want to go towards building separate dev and non-dev packages we will for sure need more metainformation to distinguish the dependency information...

-1 

If this problem isn't solved by run_depend and build_depend then why do we even make this distinction?

[Dirk Thomas]

On 02.05.2013 18:01, Jonathan Bohren wrote:

>
> On Thu, May 2, 2013 at 8:55 PM, Dirk Thomas <dth...@osrfoundation.org <mailto:dth...@osrfoundation.org>> wrote:
>
> When we would want to go towards building separate dev and non-dev packages we will for sure need more metainformation to distinguish the dependency information...
>
>
> -1
>
> If this problem isn't solved by run_depend and build_depend then why do we even make this distinction?

As described in my previous email:

>> If a package B build depends on C it does not necessarily imply that A building against B requires C.
>>
>> * A build dependency can be "internal", e.g. linking a static library or using header files only in .cpp files.
>> * Or a build dependency can be "exposed", e.g. linking a dynamic library or using header files in .h files.
>>
>> Just the fact that B build depends on C does not allow you to derive what C is for A.

We model this exact differentiation with build and run dependencies and in CMake of B by "exposing" the package A to others (e.g. A) with the catkin_package(CATKIN_DEPENDS b).
So the differentiation is currently necessary to fulfill the use cases where the maintainer wants to decide between the above mentioned two bullet points.

For building ROS packages into dev and non-dev packages we would require more meta-information since with the current information it is not determinable what goes where.
And currently the users already find catkin "too complicated" which would even get worse with more supported features and use cases.

If someone comes up with a proposed solution which can handle all currently supported use cases with less metainformation and complexity that would be awesome.

Also any concept how to support building dev and non-dev packages from a ROS package would be highly appreciated.
Due to the additional complexity we except it to introduce it is currently not planned to be implemented.

- Dirk

[Jonathan Bohren]

On Thu, May 2, 2013 at 9:26 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

>> If a package B build depends on C it does not necessarily imply that A building against B requires C.
>>
>> * A build dependency can be "internal", e.g. linking a static library or using header files only in .cpp files.
>> * Or a build dependency can be "exposed", e.g. linking a dynamic library or using header files in .h files.
>>
>> Just the fact that B build depends on C does not allow you to derive what C is for A.

We model this exact differentiation with build and run dependencies and in CMake of B by "exposing" the package A to others (e.g. A) with the catkin_package(CATKIN_DEPENDS b).
So the differentiation is currently necessary to fulfill the use cases where the maintainer wants to decide between the above mentioned two bullet points.

Would you agree that internal dependencies are the special cases that Catkin adds support for vs rosbuild? If that's so, then what if we add the following tags (and deprecate the current ones)?

<build_depend> and <run_depend> ---> <depend> 

<build_depend> and not <run_depend> ---> <internal_depend>

<run_depend> and not <build_depend> ---> <exec_depend>

Where:

 * <depend> deps are needed at compile-time and need to be exported for other packages (majority of cases),

 * <internal_depend> deps are only needed at compile-time of this package

 * <exec_depend> deps are only needed when executing scripts, or binary executables of this package

This should have the affect where any given dependency should be listed at most once in any of the above tag types in a package.xml file.

-j

[Dirk Thomas]

On 02.05.2013 18:51, Jonathan Bohren wrote:

Adding a combined dependency tag to replace two lines by one is not really a big difference.
Especially with the fact that we do have four different kind of depend tags I find it more confusing if a new tag combines two of them - but that is mostly personal preference.

Regarding the proposed renaming:
despite the effort and confusion during the transition period I highly doubt that these term are any better.

"run" as well as "exec" do not capture the fact that this is also used for e.g. exposing dynamic libraries and headers for building downstream packages.

I don't see how the proposed name change would make it any clearer to the users.

- Dirk

[Stefan Herbrechtsmeier]

Hi,

I haven't follow the whole discussion but some comments from the work to add ROS packages to an embedded Linux cross compile distribution (meta-ros).

The Yocto/OpenEmbedded project use recipes to build software and use build dependencies (DEPENDS) and manual run time dependencies (RDEPENDS). The DEPENDS consists of native executables (binaries and scripts) and compile dependencies (libraries and includes). The native executables have a "-native" postfix in the name and run on the host. The compile dependencies belongs to the target and don't run on the host. The RDEPENDS only lists dependencies which could not detect automatically like executables (and python scripts). Additionally the RDEPENDS are package based because you can split your source into separated install packages per library or executables.

From your point of view the <build_depend> (compile dependencies) and <buildtool_depend> (native executables) are okay but need to be used consequent. This means for example that rospack need to be listed as both in roslib because it needs the library and executable. The <run_depend> should be mostly replaced by a automatic detection and only list missing parts like it is done by debian (shlibs:Depends). It would be nice if they are separated per part of the ros package (library and executable) so that if you need only the library you don't need to install all the dependencies of the executable.

Kind regards,
  Stefan

[Jonathan Bohren]

On Fri, May 3, 2013 at 12:36 AM, Dirk Thomas <dth...@osrfoundation.org> wrote:

Adding a combined dependency tag to replace two lines by one is not really a big difference.
Especially with the fact that we do have four different kind of depend tags I find it more confusing if a new tag combines two of them - but that is mostly personal preference.

Four mutually-exclusive tags, three of which are special cases, means:

a) the system is easier to use because "if in doubt, use <depend>" is a really nice feature since it covers 95% of of the cases

b) the majority of package.xml files have HALF as many dependency lines

c) a dependency is only ever listed once in a package.xml file

Additionally, <test_depend> is already mutually-exclusive with <run_depend> and <build_depend>. This change would mean that all dependency tags were mutually-exclusive, thus adding consistency to a system that is in dire need of it.

 

Regarding the proposed renaming:
despite the effort and confusion during the transition period I highly doubt that these term are any better.

"the transition period" is also going to be "the learning period" for robotics students everywhere if they want to use ROS. 

 

"run" as well as "exec" do not capture the fact that this is also used for e.g. exposing dynamic libraries and headers for building downstream packages.

 

Maybe it's not a perfect mapping from "run" to exec, but maybe that's the abstraction that we should have instead.

 

I don't see how the proposed name change would make it any clearer to the users.

Well I see how such a change would make it A LOT clearer, but apparently my opinion isn't worth anything; I'm just some guy who writes and runs code with ROS on actual robots every day, teaches ROS to students and peers, and is generally an advocate using ROS to foster collaboration in the robotics community. My advisor and I were planning a semester-long course focused on system integration in robotics using ROS, and I'm seriously considering suggesting that we abandon such a plan. 

I'm going to stop suggesting even rough solutions to Catkin's usability problems because it doesn't seem like solving them is actually a priority of the maintainers and it's just been wasting a ton of my time. 

[Dirk Thomas]

On 03.05.2013 02:16, Stefan Herbrechtsmeier wrote:
> Hi,
>
> I haven't follow the whole discussion but some comments from the work to add ROS packages to an embedded Linux cross compile distribution (meta-ros).
>
> Am 03.05.2013 03:51, schrieb Jonathan Bohren:
>>

>> On Thu, May 2, 2013 at 9:26 PM, Dirk Thomas <dth...@osrfoundation.org <mailto:dth...@osrfoundation.org>> wrote:
>>
>> >> If a package B build depends on C it does not necessarily imply that A building against B requires C.
>> >>
>> >> * A build dependency can be "internal", e.g. linking a static library or using header files only in .cpp files.
>> >> * Or a build dependency can be "exposed", e.g. linking a dynamic library or using header files in .h files.
>> >>
>> >> Just the fact that B build depends on C does not allow you to derive what C is for A.
>>
>> We model this exact differentiation with build and run dependencies and in CMake of B by "exposing" the package A to others (e.g. A) with the catkin_package(CATKIN_DEPENDS b).
>> So the differentiation is currently necessary to fulfill the use cases where the maintainer wants to decide between the above mentioned two bullet points.
>>
>>
>> Would you agree that internal dependencies are the special cases that Catkin adds support for vs rosbuild? If that's so, then what if we add the following tags (and deprecate the current ones)?
>>
>> <build_depend> and <run_depend> ---> <depend>
>> <build_depend> and not <run_depend> ---> <internal_depend>
>> <run_depend> and not <build_depend> ---> <exec_depend>
>>
>> Where:
>> * <depend> deps are needed at compile-time and need to be exported for other packages (majority of cases),
>> * <internal_depend> deps are only needed at compile-time of this package
>> * <exec_depend> deps are only needed when executing scripts, or binary executables of this package
>>
>> This should have the affect where any given dependency should be listed at most once in any of the above tag types in a package.xml file.
> The Yocto/OpenEmbedded project use recipes to build software and use build dependencies (DEPENDS) and manual run time dependencies (RDEPENDS). The DEPENDS consists of native executables (binaries and
> scripts) and compile dependencies (libraries and includes). The native executables have a "-native" postfix in the name and run on the host. The compile dependencies belongs to the target and don't
> run on the host. The RDEPENDS only lists dependencies which could not detect automatically like executables (and python scripts). Additionally the RDEPENDS are package based because you can split your
> source into separated install packages per library or executables.
>
> From your point of view the <build_depend> (compile dependencies) and <buildtool_depend> (native executables) are okay but need to be used consequent. This means for example that rospack need to be
> listed as both in roslib because it needs the library and executable.

The build and buildtoold dependency does not distinguish compile dependencies and native executables.
It identifies for what platform the build dependency must be provided: build must be for the target platform, buildtool for the platform you perform the build on.

Since roslib uses rospack as a library (which mean the library must be for the target platform) it is build_depend.
The same goes for the invocation of the rospack binary - that happens on the target platform - not on the platform performing the build - hence again build_depend.
rospack would only be a buildtool_depend is it would be invoked in the build process of roslib.

catkin is currently the most commonly listed buildtool_depend since it is used on the platform building the package.
This would also be true for e.g. a dependency on pkg-config, gcc or qmake.
pkg-config could be a case where both deps make sense.
A package might use it at build time to find other stuff and/or call it at runtime.

In several cases dependencies are stated as build_depends where they are actually buildtool_depends.
If you find issues like this please fill a ticket with the respective package.

> The <run_depend> should be mostly replaced by a automatic detection and only list missing parts like it is done by debian

> (|shlibs:Depends)|. It would be nice if they are separated per part of the ros package (library and executable) so that if you need only the library you don't need to install all the dependencies of
> the executable.

How is that "automatic detection" being performed?
Can you give any details about that - may be we can apply something similar.

If parts of a package require different dependencies than you have to state the superset in the package.xml.
But there is always the possibility to separate parts which are so different into independent packages.

- Dirk

[Dirk Thomas]

On 03.05.2013 07:54, Jonathan Bohren wrote:
> On Fri, May 3, 2013 at 12:36 AM, Dirk Thomas <dth...@osrfoundation.org <mailto:dth...@osrfoundation.org>> wrote:
>
> Well I see how such a change would make it A LOT clearer, but apparently my opinion isn't worth anything; ...

As every discussion which is not based on 100% logically verifiable facts this is based on personal perspective.

You stated your view on how you think things would be better.
The same for me - but what I am writing is not per se the solution to go for - it is just my personal opinion.

Your opinion has the same weight as any body else contributing to the discussion.
If there is a majority leaning to any proposed solution - no matter from whom - that is what commonly gets realized.

- Dirk

[Tully Foote]

Jon,  

I'm sorry you're getting frustrated.  We are listening to your input as well as others.  There are advantages to your proposal such as a more minimal representation.  However there are also advantages to the current implementation such as having a 1-1 mapping to other common dependency systems (ala debian packages [1]).  

We have spent over a year now on developing the build system and it has been reviewed many times.  It's been a lot of effort to make the existing implementation and we're approaching the release of Hydro with the new system.  As such the first time we could deploy an new set of dependencies is in ITurtle.  Making this change will require everyone to change all their package.xml files again in the next cycle, and we cannot hard deprecate it in one cycle so it will drag out for at least another year.  And in the mean time we will need to keep documentation for both the proposed tags, as well as adequate documentation for the current tags. 

With this much inertia we need to have a very strong justification for making a change.  From a preliminary review of the feedback from the ROS distro survey one of the primary takeaways is that people want things to change less/slower.  In the discussion above it is clear we need to make sure that the documentation is clearer. But that will still be required through the proposed renaming, and be doubly problematic during the next year or more of migration.  

There is significant fatigue for both ROS developers and ROS users with the buildsystem changes and making further changes is encountering resistance because of this.  As the current system is functional, and during the reviews we followed closely the debian model which is a well established system, the case needs to be very strong to change anything this core and disrupt all ROS users. 

I understand your goal of usability and fully agree with a goal of making ROS as accessible as possible.  To that end I think that a script which explains to the user the implications of their currently declared dependencies could be much more useful than changing the semantic names.  It could even have an interactive prompt which asks questions of the developer to make sure they haven't made common errors. 

Tully

[1] http://www.debian.org/doc/debian-policy/ch-relationships.html

-j

--

Jonathan Bohren

Laboratory for Computational Sensing and Robotics

http://dscl.lcsr.jhu.edu/People/JonathanBohren

--

[Stefan Herbrechtsmeier]

This is exact what I mean, but your description was more generic. In OE
the term native is used for the compiled code with runs on the host
(platform you perform the build on).

>
> Since roslib uses rospack as a library (which mean the library must be
> for the target platform) it is build_depend.

Then the problem comes from catkin / cmake which requires the binary
even if it isn't used, because OE only install dev packages (libraries
and includes) into the target sysroot.

> The same goes for the invocation of the rospack binary - that happens
> on the target platform - not on the platform performing the build -
> hence again build_depend.
> rospack would only be a buildtool_depend is it would be invoked in the
> build process of roslib.

But message_generation should be a buildtool_depend in std_msgs?

>
> catkin is currently the most commonly listed buildtool_depend since it
> is used on the platform building the package.
> This would also be true for e.g. a dependency on pkg-config, gcc or
> qmake.
> pkg-config could be a case where both deps make sense.
> A package might use it at build time to find other stuff and/or call
> it at runtime.
>
> In several cases dependencies are stated as build_depends where they
> are actually buildtool_depends.
> If you find issues like this please fill a ticket with the respective
> package.
>
>
>> The <run_depend> should be mostly replaced by a automatic detection
>> and only list missing parts like it is done by debian
>> (|shlibs:Depends)|. It would be nice if they are separated per part
>> of the ros package (library and executable) so that if you need only
>> the library you don't need to install all the dependencies of
>> the executable.
>
> How is that "automatic detection" being performed?
> Can you give any details about that - may be we can apply something
> similar.

I don't know how it works, but you can find some code in
http://cgit.openembedded.org/openembedded-core/tree/meta/classes/package.bbclass

>
> If parts of a package require different dependencies than you have to
> state the superset in the package.xml.
> But there is always the possibility to separate parts which are so
> different into independent packages.

It is more a problem, that you have to provide a binary for a package
even if you don't need it during compile time because it is required by
the cmake file.

[Jonathan Bohren]

On Fri, May 3, 2013 at 1:32 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 03.05.2013 07:54, Jonathan Bohren wrote:

On Fri, May 3, 2013 at 12:36 AM, Dirk Thomas <dth...@osrfoundation.org <mailto:dthomas@osrfoundation.org>> wrote:

Well I see how such a change would make it A LOT clearer, but apparently my opinion isn't worth anything; ...

As every discussion which is not based on 100% logically verifiable facts this is based on personal perspective.

No, my opinion is based on the data that numerous people, both ROS experts and novices, have wasted time because Catkin's interface is confusing.

Your opinion has the same weight as any body else contributing to the discussion.
If there is a majority leaning to any proposed solution - no matter from whom - that is what commonly gets realized.

 

So then maybe we should ask everyone on ROS users:

"Write down what you think <build_depend> and <run_depend> mean in a Catkin package.xml file."

I'm pretty sure most people will be very confident in their answers and at the same time be very wrong.

On Fri, May 3, 2013 at 2:33 PM, Tully Foote <tfo...@osrfoundation.org> wrote:

I'm sorry you're getting frustrated.  We are listening to your input as well as others.  There are advantages to your proposal such as a more minimal representation.  However there are also advantages to the current implementation such as having a 1-1 mapping to other common dependency systems (ala debian packages [1]).  

I recognize that Catkin has been geared towards making releases easier, but I think in the process, the developers lost sight of the fact that most people spend the majority of their time writing packages, and not releasing them. 

 

We have spent over a year now on developing the build system and it has been reviewed many times.  It's been a lot of effort to make the existing implementation and we're approaching the release of Hydro with the new system.  As such the first time we could deploy an new set of dependencies is in ITurtle.  Making this change will require everyone to change all their package.xml files again in the next cycle, and we cannot hard deprecate it in one cycle so it will drag out for at least another year.  And in the mean time we will need to keep documentation for both the proposed tags, as well as adequate documentation for the current tags. 

With this much inertia we need to have a very strong justification for making a change.  From a preliminary review of the feedback from the ROS distro survey one of the primary takeaways is that people want things to change less/slower.  In the discussion above it is clear we need to make sure that the documentation is clearer. But that will still be required through the proposed renaming, and be doubly problematic during the next year or more of migration.  

This justification is even more frustrating since back before the groovy release, the discussion of making the package.xml spec clearer was ended because Catkin needed to be pushed out the door [2]:

At this point I suggest that unless someone has a compelling reason that the existing standard is not technically adequate that we not continue discussing the xml format of the dependency tags.  There are lots of other things which can use attention in our push to get groovy (beta) out the door this week.  

Maybe part of the reason why people want it to change more slowly isn't because of the changes themselves, but because the changes were released prematurely and without adequate user feedback? The challenge with supporting an LTS is that you need to make sure that the decisions that you're committing to for an LTS are well-founded. 

I understand the ROS maintainer team has put a lot of work into catkin, but most people I know who were using ROS before Groovy are still using rosbuild, even if they're now using Groovy. I think the only heartache would be for the small percentage of people who have started to use Catkin, and if usability was improved, maybe they wouldn't mind the work. The buildsystem is the FIRST thing that people interact with when they start using ROS, and if the buildsystem is confusing, then it makes the learning curve especially steep.

 

There is significant fatigue for both ROS developers and ROS users with the buildsystem changes and making further changes is encountering resistance because of this.  As the current system is functional, and during the reviews we followed closely the debian model which is a well established system, the case needs to be very strong to change anything this core and disrupt all ROS users. 

I think that the notion that changing the buildsystem causes significant fatigue for everyone should have made usability for the majority of people the #1 design goal, instead of reducing calls to CMake and making it easier to release packages. The Debian model isn't the best model for ROS users who build numerous source packages and never plan to generate binary packages.

 

I understand your goal of usability and fully agree with a goal of making ROS as accessible as possible.  To that end I think that a script which explains to the user the implications of their currently declared dependencies could be much more useful than changing the semantic names.  It could even have an interactive prompt which asks questions of the developer to make sure they haven't made common errors. 

If we can do that, why can't we just write a script that migrates package.xml files?

[2] https://groups.google.com/d/msg/ros-sig-buildsystem/FEBnAH2nmo8/HRyCle5cvT0J

-j

[Dirk Thomas]

On 03.05.2013 13:13, Jonathan Bohren wrote:

> On Fri, May 3, 2013 at 1:32 PM, Dirk Thomas <dth...@osrfoundation.org <mailto:dth...@osrfoundation.org>> wrote:
>
> Your opinion has the same weight as any body else contributing to the discussion.
> If there is a majority leaning to any proposed solution - no matter from whom - that is what commonly gets realized.
>
>
> So then maybe we should ask everyone on ROS users:
> "Write down what you think <build_depend> and <run_depend> mean in a Catkin package.xml file."
>
> I'm pretty sure most people will be very confident in their answers and at the same time be very wrong.

I do share your concern about that.
But imho changing the name of the dependency tags as you proposed will not make any difference on that.
Without further documentation and the user reading it they won't suddenly have a better understanding on how to use these dependency tags - neither if it is called build/run nor if it is internal/exec.

And again: if a majority of people will think that this simple remaining would provide a significant benefit we could definitely go for that.

> I understand the ROS maintainer team has put a lot of work into catkin, but most people I know who were using ROS before Groovy are still using rosbuild, even if they're now using Groovy. I think the
> only heartache would be for the small percentage of people who have started to use Catkin, and if usability was improved, maybe they wouldn't mind the work.

For released packages this ratio is bit different.
Currently 430 packages in Groovy are wet, 157 are dry.
These numbers are not really comparable since wet means packages and dry stacks but it clearly indicates that a significant part of the released code base has adopted catkin.

- Dirk

[William Woodall]

First, let me suggest that we give Jack back his thread and really focus on small, tractable proposals. There are two I see here:

• Make the syntax more convenient by making a tag which is equivalent to "run and build depend"
  
• Model dependencies by how a package uses them rather than when a package needs them (more below)

--------------------

I think that if we ask ros-users:

"Write down what you think <build_depend> and <run_depend> mean in a Catkin package.xml file."

They will get the correct answer, which is build depends are needed at build time of my package and run depends are needed at run time of my package. What they will get wrong is when and why they need to use run depends.

Currently run, build, and test depends define _when_ you need your dependencies, but not _why_ you need them, and I think that fundamentally this is where the confusion comes in. You may run depend on something for many different reasons.

How you use your dependencies, or put another way why you run or build depend on something, is sort of implicitly defined by your use of the dependency in CMake, build and run time scripts, etc... For example, I may come along and see that you run depend on a package. The tag tells me nothing of why you run depend on that package, you might run depend on that package for any number of reasons like: you use one of its libraries directly or you expose some of its headers to your dependents or because you simply need its python API at runtime. There is no one-to-one mapping between declaration of dependance and reason for the dependance.

I think is the essential point of your proposal is to change to defining how you use your dependencies rather than explicitly defining when you need them. Then defining how you use them has some connotation as to when they are needed.

I am interested in this part of the proposal.

That being said, I agree with Dirk that this does not make it magically simpler for users to understand.

Now instead of trying to figure out when you should run or build depend on something, you will need to figure out why you package depends on another package at build time but not run time, because there is no natural mapping between how you use a dependency and when it will be made available for your package unless you understand the underlying concept. Arguably this shifts the cognitive load to the releasers and away from the day to day users which only build from source, and this is why I am interested in it.

Either way, people will not look at the current tags nor your proposed tags and understand everything implicitly. Fundamentally, I think that users have a hard time with this because the underlying concept is complicated, not because catkin's interface is confusing.

On Fri, May 3, 2013 at 1:13 PM, Jonathan Bohren <jonatha...@gmail.com> wrote:

On Fri, May 3, 2013 at 1:32 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 03.05.2013 07:54, Jonathan Bohren wrote:

On Fri, May 3, 2013 at 12:36 AM, Dirk Thomas <dth...@osrfoundation.org <mailto:dthomas@osrfoundation.org>> wrote:

Well I see how such a change would make it A LOT clearer, but apparently my opinion isn't worth anything; ...

As every discussion which is not based on 100% logically verifiable facts this is based on personal perspective.

No, my opinion is based on the data that numerous people, both ROS experts and novices, have wasted time because Catkin's interface is confusing.

Your opinion has the same weight as any body else contributing to the discussion.
If there is a majority leaning to any proposed solution - no matter from whom - that is what commonly gets realized.

 

So then maybe we should ask everyone on ROS users:

"Write down what you think <build_depend> and <run_depend> mean in a Catkin package.xml file."

I'm pretty sure most people will be very confident in their answers and at the same time be very wrong. 

On Fri, May 3, 2013 at 2:33 PM, Tully Foote <tfo...@osrfoundation.org> wrote:

I'm sorry you're getting frustrated.  We are listening to your input as well as others.  There are advantages to your proposal such as a more minimal representation.  However there are also advantages to the current implementation such as having a 1-1 mapping to other common dependency systems (ala debian packages [1]).  

I recognize that Catkin has been geared towards making releases easier, but I think in the process, the developers lost sight of the fact that most people spend the majority of their time writing packages, and not releasing them. 

 

We have spent over a year now on developing the build system and it has been reviewed many times.  It's been a lot of effort to make the existing implementation and we're approaching the release of Hydro with the new system.  As such the first time we could deploy an new set of dependencies is in ITurtle.  Making this change will require everyone to change all their package.xml files again in the next cycle, and we cannot hard deprecate it in one cycle so it will drag out for at least another year.  And in the mean time we will need to keep documentation for both the proposed tags, as well as adequate documentation for the current tags. 

With this much inertia we need to have a very strong justification for making a change.  From a preliminary review of the feedback from the ROS distro survey one of the primary takeaways is that people want things to change less/slower.  In the discussion above it is clear we need to make sure that the documentation is clearer. But that will still be required through the proposed renaming, and be doubly problematic during the next year or more of migration.  

This justification is even more frustrating since back before the groovy release, the discussion of making the package.xml spec clearer was ended because Catkin needed to be pushed out the door [2]:

At this point I suggest that unless someone has a compelling reason that the existing standard is not technically adequate that we not continue discussing the xml format of the dependency tags.  There are lots of other things which can use attention in our push to get groovy (beta) out the door this week.  

In fairness, there was no discussion of changing the way dependencies were modeled, but how they were typed into the package.xml file. Those discussions were Bike Shed discussions and we had run out of time for Groovy. We knew that once it was released into Groovy there would be inertia to any further changes, but in my opinion we made the right decision and Tully is not wrong now that these changes need to be well justified before we act on them.  The proposals we are discussing now are simply not the same as the ones we were discussing then.

If you are proposing, separately, that we should have a <depend> tag which is equivalent to putting a build and run tag then Tully's above comment doesn't apply, we can add that at any time in a backwards compatible way.

I am actually in favor of a short cut like that, but with the knowledge that users will misuse it. I'm ok with it because I think they misuse build and run already (if they don't understand the difference they just put both anyways). Either way it needs more discussion.

 

Maybe part of the reason why people want it to change more slowly isn't because of the changes themselves, but because the changes were released prematurely and without adequate user feedback? The challenge with supporting an LTS is that you need to make sure that the decisions that you're committing to for an LTS are well-founded. 

I understand the ROS maintainer team has put a lot of work into catkin, but most people I know who were using ROS before Groovy are still using rosbuild, even if they're now using Groovy. I think the only heartache would be for the small percentage of people who have started to use Catkin, and if usability was improved, maybe they wouldn't mind the work. The buildsystem is the FIRST thing that people interact with when they start using ROS, and if the buildsystem is confusing, then it makes the learning curve especially steep.

Make no mistake, the cost of changing these tags in a non-backward compatible way would be a HUGE effort, not just for us but for everyone using catkin (which is not a small number of people).  We did several of these distribution wide changes while trying to iterate on Groovy and it was always a multi week long affair.

 

 

There is significant fatigue for both ROS developers and ROS users with the buildsystem changes and making further changes is encountering resistance because of this.  As the current system is functional, and during the reviews we followed closely the debian model which is a well established system, the case needs to be very strong to change anything this core and disrupt all ROS users. 

I think that the notion that changing the buildsystem causes significant fatigue for everyone should have made usability for the majority of people the #1 design goal, instead of reducing calls to CMake and making it easier to release packages. The Debian model isn't the best model for ROS users who build numerous source packages and never plan to generate binary packages.

That's true, our approach has been to make it correct and then move towards easy to use and convenient.

rosbuild is filled with poor assumptions and non-conventional short cuts which make it easy to use, but made it fundamentally FHS non-compliant, inefficient, and inaccurate when it comes to building and linking C++ code. Not to mention other nasty things like having to load the package manifest in python scripts and all the other reasons catkin was started in the first place.

As the catkin developers we are trying to prevent that from happening again, and as such we have to play the devil's advocate and challenge all of the proposed changes, not just because they would be difficult to implement, but because they need to be well thought out. catkin is not perfect and as we move forward it will change as a result of discussions like this, but just because we do not implement every suggestion made immediately doesn't mean we aren't considering everything.

 

 

I understand your goal of usability and fully agree with a goal of making ROS as accessible as possible.  To that end I think that a script which explains to the user the implications of their currently declared dependencies could be much more useful than changing the semantic names.  It could even have an interactive prompt which asks questions of the developer to make sure they haven't made common errors. 

If we can do that, why can't we just write a script that migrates package.xml files?

[2] https://groups.google.com/d/msg/ros-sig-buildsystem/FEBnAH2nmo8/HRyCle5cvT0J

-j

--

Jonathan Bohren

Laboratory for Computational Sensing and Robotics

http://dscl.lcsr.jhu.edu/People/JonathanBohren

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

[William Woodall]

On Fri, May 3, 2013 at 3:26 PM, William Woodall <wil...@osrfoundation.org> wrote:

First, let me suggest that we give Jack back his thread and really focus on small, tractable proposals.

In separate email threads... (forgot that part)

[Daniel Stonier]

On 4 May 2013 07:26, William Woodall <wil...@osrfoundation.org> wrote:

First, let me suggest that we give Jack back his thread and really focus on small, tractable proposals.

+1

 

This other topic has no easy solution but needs discussion.

--
Phone : +82-10-5400-3296 (010-5400-3296)
Home: http://snorriheim.dnsdojo.com/

Yujin R&D: http://rnd.yujinrobot.com/

[Jonathan Bohren]

I'm in the process of heading to ICRA right now so I don't have time to get down all my thoughts, but I really appreciate this feedback, Will. I think this sort of discussion is all I was hoping for (including that other thread you just started). I'll have a fuller response later this week, and I'm sure we'll talk about it more at ROSCon.

-j

[Jack O'Quin]

(Returning to this thread's previously-scheduled discussion:)

Thanks to all who responded to the preliminary review! 

You found an astonishing number of errors, which I have been working to correct. While I was relatively diligent about trying to fix them all, I probably missed or misunderstood a few.

The edited version is now posted here:

 http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/index.html

Please look it over, paying special attention to places where you found errors before.

Another question came to mind while editing these files. Doesn't the dynamic reconfiguration page also require add_dependencies() commands, similar to those needed for packages that define their own custom messages or services?

 http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/dynamic_reconfiguration.html

 http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html

[Daniel Stonier]

Really nice Jack.

I noticed the tests are still on a todo. I think that's a really important fragment that often gets neglected and I've been fighting on and off with them over the last few weeks.

gtests : mostly the same, although it would be worth including that if you don't have gtest on your system, you can just add the gtest sources to your rosinstall file and catkin will simply just run with it.

rostests : ok, not much changed here

nosetests : don't seem to have changed much, but for the life of me I can't figure out how to get nosetests working on a folder rather than specifying them one by one.

I don't have much time right now, but I could probably contribute to this section after RosCon.

Cheers,

Daniel.

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

[Jack O'Quin]

On Tue, May 7, 2013 at 7:59 PM, Daniel Stonier <d.st...@gmail.com> wrote:

Really nice Jack.

I noticed the tests are still on a todo. I think that's a really important fragment that often gets neglected and I've been fighting on and off with them over the last few weeks.

gtests : mostly the same, although it would be worth including that if you don't have gtest on your system, you can just add the gtest sources to your rosinstall file and catkin will simply just run with it.

rostests : ok, not much changed here

nosetests : don't seem to have changed much, but for the life of me I can't figure out how to get nosetests working on a folder rather than specifying them one by one.

I don't have much time right now, but I could probably contribute to this section after RosCon.

That would be great, Daniel. Any help is most appreciated.

I agree that those sections are important. I was getting bogged down because it took me so long to write the first sections. Finally, I decided it would be better to get something useful out where people could find it, even if it's not complete.

I do have some idea how to write the test sections, but they will probably require me to do some more experimentation to get things correct.

--
 joq

[Jack O'Quin]

On Tue, May 7, 2013 at 6:58 PM, Jack O'Quin <jack....@gmail.com> wrote:

Another question came to mind while editing these files. Doesn't the dynamic reconfiguration page also require add_dependencies() commands, similar to those needed for packages that define their own custom messages or services?

 http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/dynamic_reconfiguration.html

 http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_msgs.html

Dirk had already answered this question, but I didn't notice. 

Yes, add_dependencies() is required with dynamic reconfigure.
--
 joq

[William Woodall]

Austin found this error in the documentation:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_executables.html#building

The `Boost_LIBRARY` variable is used, but it should be `Boost_LIBRARIES`.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jack O'Quin]

On Fri, May 24, 2013 at 3:30 PM, William Woodall <wil...@osrfoundation.org> wrote:

Austin found this error in the documentation:

http://farnsworth.csres.utexas.edu/docs/catkin/html/howto/building_executables.html#building

The `Boost_LIBRARY` variable is used, but it should be `Boost_LIBRARIES`.

OK. Are you sure about that?

We've had trouble with using the _LIBRARIES suffix and better luck with the singular _LIBRARY. But, I can't remember the details.

--
 joq

[William Woodall]

I am completely positive that you should be using Boost_LIBRARIES, you can, however, use Boost_<COMPONENT>_LIBRARY. There is no Boost_LIBRARY.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/CAB6SgyWcfUq0FmaYAkAeF6XZbGKhE-z_z07QmZaXj4%2BJtfjaXQ%40mail.gmail.com?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jack O'Quin]

On Fri, May 24, 2013 at 6:18 PM, William Woodall <wil...@osrfoundation.org> wrote:

I am completely positive that you should be using Boost_LIBRARIES, you can, however, use Boost_<COMPONENT>_LIBRARY. There is no Boost_LIBRARY.

OK. Thanks to you and Austin for the correction.

Must have been some other _LIBRARY we had trouble with.

--
 joq