adding <depend> syntax to package.xml

[Jack O'Quin]

In another thread there was a discussion about how the current catkin
dependency names can be confusing. Specifically, <run_depend> does not
mean exactly what it appears to say. There was a long and useful
digression about what it actually signifies.

Jonathan Bohren proposed some changes:

| <build_depend> and <run_depend> ---> <depend>
| <build_depend> and not <run_depend> ---> <internal_depend>
| <run_depend> and not <build_depend> ---> <exec_depend>

Several people pointed out the difficulty of making API changes to the
hundreds of packages already released.

But, I like Jon's idea of introducing <depend> as a
backward-compatible name for a dependency that is *both* a
<build_depend> *and* a <run_depend>. This seems likely to capture 90%
or more of the ROS usage for years to come. The more-detailed tags
should remain available for those rare cases where they can
meaningfully be used:

<build_depend>message_generation</build_depend>
<run_depend>message_runtime</run_depend>

They can sometimes be used for system package dependencies, which
typically *are* divided into separate build and run-time .deb or .rpm
files. For that to be work, we should start adding both names to the
rosdep YAML files. Current practice is only to define the "foo-dev"
name, but "foo" could be added without breakage. Older entries
(without the "-dev" suffix) will be somewhat problematic.
--
joq

[Dirk Thomas]

On 04.05.2013 17:59, Jack O'Quin wrote:
> In another thread there was a discussion about how the current catkin
> dependency names can be confusing. Specifically, <run_depend> does not
> mean exactly what it appears to say. There was a long and useful
> digression about what it actually signifies.
>
> Jonathan Bohren proposed some changes:
>
> | <build_depend> and <run_depend> ---> <depend>
> | <build_depend> and not <run_depend> ---> <internal_depend>
> | <run_depend> and not <build_depend> ---> <exec_depend>
>
> Several people pointed out the difficulty of making API changes to the
> hundreds of packages already released.
>
> But, I like Jon's idea of introducing <depend> as a
> backward-compatible name for a dependency that is *both* a
> <build_depend> *and* a <run_depend>. This seems likely to capture 90%
> or more of the ROS usage for years to come. The more-detailed tags
> should remain available for those rare cases where they can
> meaningfully be used:
>
> <build_depend>message_generation</build_depend>
> <run_depend>message_runtime</run_depend>

Adding a depend tag should be very little implementation effort.
catkin_pkg can encapsulate that in the parsing of the package.xml while maintaining the same API.
It would expose a depend entry as if it would be listed as build_depend and run_depend.

(One side note: having this combined dependency might encourage the behavior of "just using that" without understanding the implications.)

> They can sometimes be used for system package dependencies, which
> typically *are* divided into separate build and run-time .deb or .rpm
> files. For that to be work, we should start adding both names to the
> rosdep YAML files. Current practice is only to define the "foo-dev"
> name, but "foo" could be added without breakage. Older entries
> (without the "-dev" suffix) will be somewhat problematic.

Since the dev package contains the header files it is actually very common to run_depend on a dev package.
This is due to the fact that these headers are required to build downstream packages.

As William mentioned in a previous email I think we need to discuss how we can better express what a dependency is used for instead of when it is used.
I expect the outcome of that discussion to affect what kind of dependency tags we need.
Therefore I think we should wait before introducing <depend> until we have a clear picture where we are heading with that.

- Dirk

[Jack O'Quin]

On Sat, May 4, 2013 at 12:14 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 04.05.2013 17:59, Jack O'Quin wrote:

In another thread there was a discussion about how the current catkin
dependency names can be confusing. Specifically, <run_depend> does not
mean exactly what it appears to say.  There was a long and useful
digression about what it actually signifies.

Jonathan Bohren proposed some changes:

  | <build_depend> and <run_depend> ---> <depend>
  | <build_depend> and not <run_depend> ---> <internal_depend>
  | <run_depend> and not <build_depend> ---> <exec_depend>

Several people pointed out the difficulty of making API changes to the
hundreds of packages already released.

But, I like Jon's idea of introducing <depend> as a
backward-compatible name for a dependency that is *both* a
<build_depend> *and* a <run_depend>. This seems likely to capture 90%
or more of the ROS usage for years to come. The more-detailed tags
should remain available for those rare cases where they can
meaningfully be used:

   <build_depend>message_generation</build_depend>
   <run_depend>message_runtime</run_depend>

Adding a depend tag should be very little implementation effort.
catkin_pkg can encapsulate that in the parsing of the package.xml while maintaining the same API.
It would expose a depend entry as if it would be listed as build_depend and run_depend.

(One side note: having this combined dependency might encourage the behavior of "just using that" without understanding the implications.)

That will almost certainly happen. But, as William said in an earlier thread:

 > I am actually in favor of a short cut like that, but with the knowledge 

 > that users will misuse it. I'm ok with it because I think they misuse 

 > build and run already (if they don't understand the difference they

 > just put both anyways). Either way it needs more discussion.

 

It would be unrealistic to expect all ROS users to understand these complicated details. Let's provide them with a simpler interface that works, even though it sometimes creates unnecessary dependencies.

Expert packagers can still use the more detailed dependency syntax to eliminate those dependencies where they are important enough to justify the effort.

They can sometimes be used for system package dependencies, which
typically *are* divided  into separate build and run-time .deb or .rpm
files. For that to be work, we should start adding both names to the
rosdep YAML files. Current practice is only to define the "foo-dev"
name, but "foo" could be added without breakage. Older entries
(without the "-dev" suffix) will be somewhat problematic.

Since the dev package contains the header files it is actually very common to run_depend on a dev package.
This is due to the fact that these headers are required to build downstream packages.

As William mentioned in a previous email I think we need to discuss how we can better express what a dependency is used for instead of when it is used.
I expect the outcome of that discussion to affect what kind of dependency tags we need.
Therefore I think we should wait before introducing <depend> until we have a clear picture where we are heading with that.

Most of the confusion that has come up recently is due to the dual meaning of <run_depend>. It can signify a traditional run-time dependency like a C++ library or Python module, or it can indicate a transitive build-time dependency, like a C++ header nested within a header exported by the current package.

In Debian terms, the true run-time dependency translates into a dependency of the binary package, while the nested header dependency becomes a dependency of the development package. In some cases, both are needed. They are not mutually exclusive.

That suggests supplementing <run_depend> with something like <devel_depend> for use with nested headers or static libraries. Certainly, we must deal with important questions of backwards compatibility, but I'd like to clarify the semantics, first. 

--
 joq

[Dirk Thomas]

On 05.05.2013 18:18, Jack O'Quin wrote:

>
> On Sat, May 4, 2013 at 12:14 PM, Dirk Thomas <dth...@osrfoundation.org <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=dth...@osrfoundation.org>> wrote:
>
> Most of the confusion that has come up recently is due to the dual meaning of <run_depend>. It can signify a traditional run-time dependency like a C++ library or Python module, or it can indicate a
> transitive build-time dependency, like a C++ header nested within a header exported by the current package.
>
> In Debian terms, the true run-time dependency translates into a dependency of the binary package, while the nested header dependency becomes a dependency of the development package. In some cases,
> both are needed. They are not mutually exclusive.
>
> That suggests supplementing <run_depend> with something like <devel_depend> for use with nested headers or static libraries. Certainly, we must deal with important questions of backwards
> compatibility, but I'd like to clarify the semantics, first.

I do see that two tags would map more cleanly to the two different semantic meanings of these dependency.
How would these two dependencies be used?
Especially how would they be treated differently in CMake?

- Dirk

[Jack O'Quin]

Right now, they would be treated the same. 

If we ever figure out how to split ROS packages into binary and devel Debian packages, they would set the dependencies of the respective Debian packages.

If we are never going to do that, we should just make everything an unqualified <depend>, for simplicity.

--
 joq

[Jonathan Bohren]

So I've talked to a bunch of people here at ICRA, and found that nobody who uses catkin can tell me the counter-intuitive details of run_depend (deps needed when a package builds against your package) and most are surprised when I tell them. 

Most who I've talked to have said that they always just make every dependency a build and run dependency. Those who do this say that they find it really annoying that they have to specify it twice, especially when developing packages which contain numerous dependencies.

I think we should add a run+build <depend> tag for hydro, if possible. Those who need to make the distinction can do so, but I think in 95% of the cases, a single <depend> tag which means <run_depend> AND <build_depend> is suitable.

Furthermore, I think if we can add this for hydro, it should be mentioned in the Catkin ROSCon tutorial talk alongside the other dependency options.

-j

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Jonathan Bohren

Laboratory for Computational Sensing and Robotics

http://dscl.lcsr.jhu.edu/People/JonathanBohren

[Jack O'Quin]

On Thu, May 9, 2013 at 11:19 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

So I've talked to a bunch of people here at ICRA, and found that nobody who uses catkin can tell me the counter-intuitive details of run_depend (deps needed when a package builds against your package) and most are surprised when I tell them. 

Most who I've talked to have said that they always just make every dependency a build and run dependency. Those who do this say that they find it really annoying that they have to specify it twice, especially when developing packages which contain numerous dependencies.

I think we should add a run+build <depend> tag for hydro, if possible. Those who need to make the distinction can do so, but I think in 95% of the cases, a single <depend> tag which means <run_depend> AND <build_depend> is suitable.

+1  It is a useful and non-breaking extension to the package.xml.

In addition to the convenience, there really are a number of legitimate use cases for it, where we *want* people to provide both run_depend and build_depend. Message packages are one good example.

Furthermore, I think if we can add this for hydro, it should be mentioned in the Catkin ROSCon tutorial talk alongside the other dependency options.

--
 joq

[Dirk Thomas]

On 09.05.2013 18:19, Jonathan Bohren wrote:
> So I've talked to a bunch of people here at ICRA, and found that nobody who uses catkin can tell me the counter-intuitive details of run_depend (deps needed when a package builds against your package)
> and most are surprised when I tell them.
>
> Most who I've talked to have said that they always just make every dependency a build and run dependency. Those who do this say that they find it really annoying that they have to specify it twice,
> especially when developing packages which contain numerous dependencies.
>
> I think we should add a run+build <depend> tag for hydro, if possible. Those who need to make the distinction can do so, but I think in 95% of the cases, a single <depend> tag which means <run_depend>
> AND <build_depend> is suitable.
>
> Furthermore, I think if we can add this for hydro, it should be mentioned in the Catkin ROSCon tutorial talk alongside the other dependency options.

Adding a depend tag seems to be a good idea based on the feedback.

But making this decision two days before the conference is definitely not.
We have not yet made any progress on the discussion on dependencies in general and what we all expect and envision for the future.
Without having a consensus where we want to go / what kind of dependencies we want to address I am highly against adding another tag which we will have to deal with if the discussion goes in a
different direction.

Hopefully we can find some time at ROSCon to discuss this with several people in person in order to achieve some progress on these topics.

- Dirk

[Jack O'Quin]

On Thu, May 9, 2013 at 5:20 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

On 09.05.2013 18:19, Jonathan Bohren wrote:

So I've talked to a bunch of people here at ICRA, and found that nobody who uses catkin can tell me the counter-intuitive details of run_depend (deps needed when a package builds against your package)
and most are surprised when I tell them.

Most who I've talked to have said that they always just make every dependency a build and run dependency. Those who do this say that they find it really annoying that they have to specify it twice,
especially when developing packages which contain numerous dependencies.

I think we should add a run+build <depend> tag for hydro, if possible. Those who need to make the distinction can do so, but I think in 95% of the cases, a single <depend> tag which means <run_depend>
AND <build_depend> is suitable.

Furthermore, I think if we can add this for hydro, it should be mentioned in the Catkin ROSCon tutorial talk alongside the other dependency options.

Adding a depend tag seems to be a good idea based on the feedback.

But making this decision two days before the conference is definitely not.

I agree with Dirk. We should not rush this decision.

I do think it can and should be done in time for Hydro.

 

We have not yet made any progress on the discussion on dependencies in general and what we all expect and envision for the future.
Without having a consensus where we want to go / what kind of dependencies we want to address I am highly against adding another tag which we will have to deal with if the discussion goes in a different direction.

It really does not matter what we decide about the other tags. We are going to want <depend> even if we decide never to support separate run and devel Debians for ROS packages. Maybe even especially in that unlikely eventuality.

 

Hopefully we can find some time at ROSCon to discuss this with several people in person in order to achieve some progress on these topics.

I won't be at ROSCon this year, so I'll wish you all good luck and a productive discussion.

--
 joq

[Jonathan Bohren]

I'd like to resume this discussion and re-assert that we should add a "<depend>" tag which translates literally to "<build_depend> and <run_depend>"

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jack O'Quin]

On Mon, May 27, 2013 at 12:10 PM, Jonathan Bohren <jonatha...@gmail.com> wrote:

I'd like to resume this discussion and re-assert that we should add a "<depend>" tag which translates literally to "<build_depend> and <run_depend>"

+1, this still seems worthwhile to me.

I don't think we have any compelling near-term requirement to separate ROS packages into their build and run-time components. Furthermore, we can't find time to focus on those issues right now.

If anything, that lack of focus on separate build and run-time packaging, just makes the <depend> tag more important.

--
 joq

[William Woodall]

+1

I would be in favor of adding this tag.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/CAB6SgyUgG8U0EpEOM93z1OE6P_fqaG20m-2tPV2zLhWoShb7bA%40mail.gmail.com?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

[Dirk Thomas]

I added support for the <depend> tag in catkin_pkg in https://github.com/ros-infrastructure/catkin_pkg/pull/43.
Please feel free to review / provide feedback.

It is not changing any API since the depend tag is not exposed.
After parsing it is considered to be a build_depend and ros_depend.
(I hope this will not create problems in the future if we decide that we need to add more dependency types e.g. for dev/non-dev dependencies).

- Dirk

On 29.05.2013 07:54, William Woodall wrote:
> +1
>
> I would be in favor of adding this tag.
>
>

> On Mon, May 27, 2013 at 6:07 PM, Jack O'Quin <jack....@gmail.com <mailto:jack....@gmail.com>> wrote:

>
>
> On Mon, May 27, 2013 at 12:10 PM, Jonathan Bohren <jonatha...@gmail.com <mailto:jonatha...@gmail.com>> wrote:
>
> I'd like to resume this discussion and re-assert that we should add a "<depend>" tag which translates literally to "<build_depend> and <run_depend>"
>
>
> +1, this still seems worthwhile to me.
>
> I don't think we have any compelling near-term requirement to separate ROS packages into their build and run-time components. Furthermore, we can't find time to focus on those issues right now.
>
> If anything, that lack of focus on separate build and run-time packaging, just makes the <depend> tag more important.
> --
> joq
>
> --
> You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com <mailto:ros-sig-buildsystem%2Bunsu...@googlegroups.com>.
> To post to this group, send email to ros-sig-b...@googlegroups.com <mailto:ros-sig-b...@googlegroups.com>.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/CAB6SgyUgG8U0EpEOM93z1OE6P_fqaG20m-2tPV2zLhWoShb7bA%40mail.gmail.com?hl=en.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>
> --
> William Woodall
> ROS Development Team

> wil...@osrfoundation.org <mailto:wil...@osrfoundation.org>
> http://williamjwoodall.com/

>
> --
> You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
> To post to this group, send email to ros-sig-b...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/CAFe2DWjrw3z4Ua%2BWao3jQBe_nThYAt-QdOrQDM0uf22Y0BVB1A%40mail.gmail.com?hl=en.

[Jonathan Bohren]

On Wed, May 29, 2013 at 4:39 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

I added support for the <depend> tag in catkin_pkg in https://github.com/ros-infrastructure/catkin_pkg/pull/43.
Please feel free to review / provide feedback.

It is not changing any API since the depend tag is not exposed.
After parsing it is considered to be a build_depend and ros_depend.
(I hope this will not create problems in the future if we decide that we need to add more dependency types e.g. for dev/non-dev dependencies).

Awesome! Thank you. This looks good.

(also I assume you meant "run_depend" and not "ros_depend" :)

-j

[Jack O'Quin]

On Wed, May 29, 2013 at 3:39 PM, Dirk Thomas <dth...@osrfoundation.org> wrote:

I added support for the <depend> tag in catkin_pkg in https://github.com/ros-infrastructure/catkin_pkg/pull/43.
Please feel free to review / provide feedback.

It is not changing any API since the depend tag is not exposed.
After parsing it is considered to be a build_depend and ros_depend.
(I hope this will not create problems in the future if we decide that we need to add more dependency types e.g. for dev/non-dev dependencies).

Perhaps we should define the semantics of ``<depend>`` to imply all current or future ``<*_depend>`` tags, except that it should never cause errors for redundant dependency tags.

--
 joq

[Jonathan Bohren]

On Thu, May 30, 2013 at 10:58 AM, Jack O'Quin <jack....@gmail.com> wrote:

Perhaps we should define the semantics of ``<depend>`` to imply all current or future ``<*_depend>`` tags, except that it should never cause errors for redundant dependency tags.

This sounds reasonable, I do wish we had a depend syntax that wasn't so verbose for when you wanted more granular control than <depend>.

-j

[Damon Kohler]

It may be worth trying to emulate a dependency scoping model like
Maven uses. It's well vetted.

http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope

I think it also makes for a nice illustration of just how complex this
sort of thing really is. Describing dependencies exactly requires a
lot of conventions, a verbose syntax, or both.

It's probably necessary to decide if Catkin will try to define
dependencies exactly or just good enough.

> --
> You received this message because you are subscribed to the Google Groups
> "ROS Buildsystem Special Interest Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ros-sig-buildsy...@googlegroups.com.
> To post to this group, send email to ros-sig-b...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ros-sig-buildsystem/CADy-PFiFwuy7SOCH%2B%3D1rksbZWhBMPd9qmtOndXmvHUEZKy%2BzcA%40mail.gmail.com?hl=en.

>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--

Damon Kohler

Google Germany GmbH
Dienerstr. 12, 80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[William Woodall]

There seems to be some evolution of even the Maven dependency scopes, specifically the system and import scopes seem to be very corner case like and very Java specific. Many of our discussions about corner cases are pretty heavily driven by the way C++ works and may not even translate to Java or other languages. So, deciding exactly vs good enough might be tied to the supported languages.

To view this discussion on the web visit https://groups.google.com/d/msgid/ros-sig-buildsystem/CALU21Xykf-siHu6HChODuBJxdMkPmoeUGZ_SHkeRZJ8dPfv5pQ%40mail.gmail.com?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.

--

William Woodall

ROS Development Team

wil...@osrfoundation.org

http://williamjwoodall.com/

[Damon Kohler]

On Fri, Jun 7, 2013 at 8:55 PM, William Woodall
<wil...@osrfoundation.org> wrote:
> There seems to be some evolution of even the Maven dependency scopes,
> specifically the system and import scopes seem to be very corner case like
> and very Java specific.

I'd say the "provided" scope is the only one that's not used by C++.
The "system" scope maps to installed system libraries and "import" is
like depending on a meta package.

> https://groups.google.com/d/msgid/ros-sig-buildsystem/CAFe2DWhdv-1BGkt4k1C%3DexZ11oQR3xotHcoeZe%3DfmZJgDz%2BmtQ%40mail.gmail.com?hl=en.