meta-package installation

[tkruse]

Hi all,

I just realized that for meta-packages, we do not install the given package.xml. I understand that meta-packages have no other content than a package.xml in source, but even that is content that maybe should be installed.
As soon as anyone wants to use the information from a meta-package for any purpose, currently he has to download the source.
Simple purposes could be for users to look at other meta-packages, to visualize dependency trees and package groups, and maybe also to depend on meta-packages.

I understand there would be quite some effort involved now, since we currently do not allow a CMakeLists in which to define an install target. So i am a little ashamed to suggest this so late.

Thoughts?

[Jonathan Bohren]

Maybe if a package is defined as a metapackage, it's package.xml is automatically put into the binary deb? I think that sounds like the easiest way to do it for now,

-j

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/-9FoB15ixE0J.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[William Woodall]

Metapackages should not be depended on by other catkin packages as they will not pass along their dependencies' information as they have no find_package infrastructure.

The technical reason for the package.xml's not being installed is that they have no CMakeLists.txt. The only purpose of meta packages for the moment is for backwards compatibility with rosbuild and for convenience when installing debians.

Installing the package.xml's might not be necessary, but if we can come up with a good use case for having them, they could specially packaged in the debian process. Either way I think at this point it is something we'll have to consider for Hydro.

--

William Woodall

Willow Garage - Software Engineer

wwoo...@willowgarage.com

[Geoffrey Biggs]

On Dec 11, 2012, at 2:45 AM, William Woodall wrote:

> Metapackages should not be depended on by other catkin packages as they will not pass along their dependencies' information as they have no find_package infrastructure.

That sounds a bit crippling. I think it would be useful for a package to say "I need the navigation system to be installed," where said navigation system is a metapackage.

Geoff

[William Woodall]

Saying you need it installed is fine, but your packages should depend directly on the packages in navigation not on navigation itself.  This maintains the stack/package relationship that existed before.

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[tkruse]

On Monday, December 10, 2012 6:45:01 PM UTC+1, William Woodall wrote:

Metapackages should not be depended on by other catkin packages as they will not pass along their dependencies' information as they have no find_package infrastructure.

If that were to be true, an attempt to depend on a metapackage should raise an error. However, I guess metapackages may depends on metapackages?

I was mostly considering the use-case of using the meta-packages to visualize dependencies, the way stacks did.
As in rospack depends and rospack depends-on.

[Geoffrey Biggs]

On Dec 11, 2012, at 3:39 AM, William Woodall wrote:
> Saying you need it installed is fine, but your packages should depend directly on the packages in navigation not on navigation itself. This maintains the stack/package relationship that existed before.

This would mean that a package X that requires navigation capability would need to depend on all the individual packages that collectively provide that function. If the metapackage for navigation changes from using Package A to using Package B for providing some functionality, a change that is not visible to users of the metapackage because the same features are still being provided (e.g. provided topics), then package X needs to update its dependencies. At this point, an important part of the usefulness of the metapackage becomes questionable.

Geoff

[William Woodall]

There is no efficient way for catkin_pkg to know if a <*_depend> is a metapackage, but if you try to find_package(catkin REQUIRED COMPONENTS <metapackage>) in the CMakeLists.txt it will fail because there is no find_package infrastructure for a metapackage.

You have to think of metapackages as stacks, rosbuild packages should never have depended on stacks, stacks always depended on stacks.  The stacks dependency graph was a umbrella of the package dependencies, but we used to only do dependency resolution on the stack level and build dependencies at the package level.  With catkin the dependency resolution and build dependencies are done at the package level.  This is possible because each package is released separately.  Therefore metapackages are not necessary at all for catkin, they only serve to placeholder for stacks for purposes of rosbuild compatability, wiki documentation consistency, and because there used to be debians for them.

--

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/Z5b5h-8fD-MJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[William Woodall]

If you want a package to pass along all of the components in navigation so that your package only has to depend on one thing, then you want an actual package, not a metapackage.  This package would find_package all of the navigation packages and then pass them along using the catkin_package macro.  Then anyone find_packaging that package would get everything it specified in the catkin_package macro.

This is often not the case though.  Sometimes you really just want to depend on navigation_msgs or some part of a "stack" of related packages.  This was previously not possible.  For example, If I needed just the arm_navigation_msgs I had to pull in the entire arm_navigation stack to use it.

--

Geoff

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dirk Thomas]

Before discussing if and how package.xml files of metapackages should/can be installed consider the reasons for metapackages in the first place.

Metapackages have only two reasons for existence:
- provide metapackage for dry packages to depend on (therefore metapackages are also allowed to depend on metapackages for bc)
- provide grouping for the Wiki to replace the former stacks

They are not used to be dependent on from normal packages because we do not want this duplicated hierarchy as in the dry world again.
Based on this set there is no need to install the package.xml file of a metapackage.

If you need a package which groups a lot of packages for ease of dependency you should just use a normal package for that.
This package must provide a CMakeLists.txt beside the package.xml file because it is responsible to provide the correct information to downstream packages (i.e. include directories, libraries etc. for
find_package() and pkg-config). One example of such a package which just groups several upstream packages is langs (https://github.com/ros/langs) which is therefore not metapackage.

Indeed we should add error messages where appropriate to inform the user if is using it in a way which not intended/supported.
But this is not trivial or even possible in certain situations.
Anyway when building i.e. a local workspace these checks should be added for the checked out code.

- Dirk

On 10.12.2012 10:52, William Woodall wrote:
> If you want a package to pass along all of the components in navigation so that your package only has to depend on one thing, then you want an actual package, not a metapackage. This package would
> find_package all of the navigation packages and then pass them along using the catkin_package macro. Then anyone find_packaging that package would get everything it specified in the catkin_package macro.
>
> This is often not the case though. Sometimes you really just want to depend on navigation_msgs or some part of a "stack" of related packages. This was previously not possible. For example, If I
> needed just the arm_navigation_msgs I had to pull in the entire arm_navigation stack to use it.
>
> --
>

> On Mon, Dec 10, 2012 at 10:46 AM, Geoffrey Biggs <gbi...@killbots.net <mailto:gbi...@killbots.net>> wrote:
>
> On Dec 11, 2012, at 3:39 AM, William Woodall wrote:
> > Saying you need it installed is fine, but your packages should depend directly on the packages in navigation not on navigation itself. This maintains the stack/package relationship that
> existed before.
>
> This would mean that a package X that requires navigation capability would need to depend on all the individual packages that collectively provide that function. If the metapackage for navigation
> changes from using Package A to using Package B for providing some functionality, a change that is not visible to users of the metapackage because the same features are still being provided (e.g.
> provided topics), then package X needs to update its dependencies. At this point, an important part of the usefulness of the metapackage becomes questionable.
>
> Geoff
>
> --
> You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

> To post to this group, send email to ros-sig-b...@googlegroups.com <mailto:ros-sig-b...@googlegroups.com>.
> To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com <mailto:ros-sig-buildsystem%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>
> --
> William Woodall
> Willow Garage - Software Engineer

> wwoo...@willowgarage.com <mailto:wwoo...@willowgarage.com>

[tkruse]

On Monday, December 10, 2012 8:02:57 PM UTC+1, Dirk Thomas wrote:

Before discussing if and how package.xml files of metapackages should/can be installed consider the reasons for metapackages in the first place.

Metapackages have only two reasons for existence:
- provide metapackage for dry packages to depend on (therefore metapackages are also allowed to depend on metapackages for bc)
- provide grouping for the Wiki to replace the former stacks

I would argue that they should provide grouping, and that the wiki is just one example of a client for this grouping. A CLI tool is another possible client or rqt gui.

In any case, if a dry package (or stack?) can depend on a catkin metapackage, how is that going to work if nothing about the meta-package is installed? Would that go through rosdep?

Also I believe that metapackages were introduced as a kind of catkin package, hence the same package.xml syntax end so on. Yet now it seems that rospack does not consider metapackages to be packages at all. This could be somewhat clearer.

[Daniel Stonier]

On 11 December 2012 04:02, Dirk Thomas <dth...@willowgarage.com> wrote:

Before discussing if and how package.xml files of metapackages should/can be installed consider the reasons for metapackages in the first place.

Metapackages have only two reasons for existence:
- provide metapackage for dry packages to depend on (therefore metapackages are also allowed to depend on metapackages for bc)
- provide grouping for the Wiki to replace the former stacks

They are not used to be dependent on from normal packages because we do not want this duplicated hierarchy as in the dry world again.
Based on this set there is no need to install the package.xml file of a metapackage.

If you need a package which groups a lot of packages for ease of dependency you should just use a normal package for that.
This package must provide a CMakeLists.txt beside the package.xml file because it is responsible to provide the correct information to downstream packages (i.e. include directories, libraries etc. for find_package() and pkg-config). One example of such a package which just groups several upstream packages is langs (https://github.com/ros/langs) which is therefore not metapackage.

If you do create a normal package like that, would it fail to provide grouping for the wiki stacks, i.e. do you need to drop the metapackage tag?

I'm inclined to agree with geoffrey about the unintuitiveness of it. A metapackage...

• behaves like a metapackage for apt-get
  
• behaves like a metapackage for the wiki
• does not behave like a metapackage for catkin-cmake

It wouldn't be too hard to fill in most of the cmake magic to do so with a catkin_create_stack wizard template and technical difficulties aside, I can't see any really good reason why it break design/philosophies/etc

Having said all that, I don't think it's all doom and gloom if that behaviour is not there...just unintuitive.

Daniel.

 

Indeed we should add error messages where appropriate to inform the user if is using it in a way which not intended/supported.
But this is not trivial or even possible in certain situations.
Anyway when building i.e. a local workspace these checks should be added for the checked out code.

- Dirk

On 10.12.2012 10:52, William Woodall wrote:

If you want a package to pass along all of the components in navigation so that your package only has to depend on one thing, then you want an actual package, not a metapackage.  This package would
find_package all of the navigation packages and then pass them along using the catkin_package macro.  Then anyone find_packaging that package would get everything it specified in the catkin_package macro.

This is often not the case though.  Sometimes you really just want to depend on navigation_msgs or some part of a "stack" of related packages.  This was previously not possible.  For example, If I
needed just the arm_navigation_msgs I had to pull in the entire arm_navigation stack to use it.

--

On Mon, Dec 10, 2012 at 10:46 AM, Geoffrey Biggs <gbi...@killbots.net <mailto:gbi...@killbots.net>> wrote:

    On Dec 11, 2012, at 3:39 AM, William Woodall wrote:
     > Saying you need it installed is fine, but your packages should depend directly on the packages in navigation not on navigation itself.  This maintains the stack/package relationship that
    existed before.

    This would mean that a package X that requires navigation capability would need to depend on all the individual packages that collectively provide that function. If the metapackage for navigation
    changes from using Package A to using Package B for providing some functionality, a change that is not visible to users of the metapackage because the same features are still being provided (e.g.
    provided topics), then package X needs to update its dependencies. At this point, an important part of the usefulness of the metapackage becomes questionable.

    Geoff

    --
    You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

    To post to this group, send email to ros-sig-buildsystem@googlegroups.com <mailto:ros-sig-buildsystem@googlegroups.com>.
    To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com <mailto:ros-sig-buildsystem%2Bunsu...@googlegroups.com>.

    For more options, visit https://groups.google.com/groups/opt_out.





--
William Woodall
Willow Garage - Software Engineer

wwoo...@willowgarage.com <mailto:wwoodall@willowgarage.com>

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-buildsystem@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-buildsystem@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Phone : +82-10-5400-3296 (010-5400-3296)
Home: http://snorriheim.dnsdojo.com/

Yujin R&D: http://rnd.yujinrobot.com/

[Jack O'Quin]

On Tue, Dec 11, 2012 at 8:20 AM, Daniel Stonier <d.st...@gmail.com> wrote:

On 11 December 2012 04:02, Dirk Thomas <dth...@willowgarage.com> wrote:

Before discussing if and how package.xml files of metapackages should/can be installed consider the reasons for metapackages in the first place.

Metapackages have only two reasons for existence:
- provide metapackage for dry packages to depend on (therefore metapackages are also allowed to depend on metapackages for bc)
- provide grouping for the Wiki to replace the former stacks

They are not used to be dependent on from normal packages because we do not want this duplicated hierarchy as in the dry world again.
Based on this set there is no need to install the package.xml file of a metapackage.

If you need a package which groups a lot of packages for ease of dependency you should just use a normal package for that.
This package must provide a CMakeLists.txt beside the package.xml file because it is responsible to provide the correct information to downstream packages (i.e. include directories, libraries etc. for find_package() and pkg-config). One example of such a package which just groups several upstream packages is langs (https://github.com/ros/langs) which is therefore not metapackage.

If you do create a normal package like that, would it fail to provide grouping for the wiki stacks, i.e. do you need to drop the metapackage tag?

I'm inclined to agree with geoffrey about the unintuitiveness of it. A metapackage...

• behaves like a metapackage for apt-get
  
• behaves like a metapackage for the wiki
• does not behave like a metapackage for catkin-cmake

It wouldn't be too hard to fill in most of the cmake magic to do so with a catkin_create_stack wizard template and technical difficulties aside, I can't see any really good reason why it break design/philosophies/etc

Having said all that, I don't think it's all doom and gloom if that behaviour is not there...just unintuitive.

Daniel sums up the problem well. Metapackages are not well-defined. In Groovy, they were an after-thought, derived from what catkin needed and could not otherwise supply.  

There has not been enough discussion of what actual users need. Much of that has nothing to do with catkin. It seems that the requirement for metapackages (a.k.a. "stacks") morphed into two incompatible forms: metapackages and real packages defining only dependencies. 

That seems wrong and should be fixed in Hydro, if possible. This is probably the right time to begin any such discussions.

--
 joq

[tkruse]

I'd like to continue this discussion with another argument. We have the export section in packages (and metapackages) which is a free-for-all section where anything can be put into. I believe already there is no webpage that lists all the acknowledged values as of today. This allows users to put some additional meta-information into metapackages, but this information currently would not make it into the install space, so it's lost.

On Monday, December 10, 2012 8:02:57 PM UTC+1, Dirk Thomas wrote:

Before discussing if and how package.xml files of metapackages should/can be installed consider the reasons for metapackages in the first place.

Metapackages have only two reasons for existence:
- provide metapackage for dry packages to depend on (therefore metapackages are also allowed to depend on metapackages for bc)
- provide grouping for the Wiki to replace the former stacks

They are not used to be dependent on from normal packages because we do not want this duplicated hierarchy as in the dry world again.
Based on this set there is no need to install the package.xml file of a metapackage.

If you need a package which groups a lot of packages for ease of dependency you should just use a normal package for that.

 
This argument confuses me as well.On the one hand you want to avoid packages to depend on metapackages, because you want to avoid a duplicated hierarchy.
Then you suggest using a non-metapackage like langs (now message_generation and message runtime) to do exactly the same thing.

So when catkinizing rosbuild stacks, users are currently free to create a metapackage named like their stack, or create a normal package containing
nothing but a CMakeLists.txt. So what is the benefit for the user in using a metapackage in this case?

Also, metapackages were provided for backwards-compatibility with rosbuild, so that rosbuild stacks having a dependency on a stack foo would not complain if stack foo went away with catkinization. Now however, since the package.xml does not make it into installation, rosbuild has been made to accept the absence of a stack anyway, leading to a largely unintuitive behavior of a rosbuild stack depending on another stack, but basically still building if there is no trace of that stack.

[Geoff Biggs]

Hi Dirk,

On 21/01/13 17:43, tkruse wrote:
> On Monday, December 10, 2012 8:02:57 PM UTC+1, Dirk Thomas wrote:
> Before discussing if and how package.xml files of metapackages
> should/can be installed consider the reasons for metapackages in the
> first place.
>
> Metapackages have only two reasons for existence:
> - provide metapackage for dry packages to depend on (therefore
> metapackages are also allowed to depend on metapackages for bc)
> - provide grouping for the Wiki to replace the former stacks
>
> They are not used to be dependent on from normal packages because we
> do not want this duplicated hierarchy as in the dry world again.
> Based on this set there is no need to install the package.xml file
> of a metapackage.
>
> If you need a package which groups a lot of packages for ease of
> dependency you should just use a normal package for that.

I don't understand why metapackages would create a duplicated hierarchy.
I think I understand why stacks did (stacks could only depend on stacks,
packages could only depend on packages). Could you explain why
metapackages have the same problem?

Geoff

[Dirk Thomas]

Currently metapackages are basically variants with additional meta information.

* they provide a convenient way for the user to install a set of packages by generating Debian metapackages
* they provide the information to perform grouping in the ROS wiki
Nothing more.

I am really interested to revive the discussion on this and make whatever changes we find are necessary for the next release.

But if we want to continue the discussion we should not start with "why is it not installed" or "why can other packages not depend on them".
These are all secondary implementation details.
We should clarify first what functionality we want metapackages (or however we call what we want) to fulfill.

Based on that we can go on and figure out how that can be implemented and how it relates to the current metapackages.
But this should be the second step after clarifying and having a consensus on the desired functionality.

- Dirk

> --
> You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
> To post to this group, send email to ros-sig-b...@googlegroups.com.
> To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/kyURwXYsyR0J.

[tkruse]

Here are the functionalities I care about:

- provide rosbuild compatibility where stacks are resolved instead to meta-packages (rather than accepting absence of a stack dependency)
- provide the same meta-information of metapackages to tools in installspace as in develspace, instead of losing metapackages halfway.
 - rqt_package_graph/rospack/catkin_pkg/catkin_find functions should return the same results
 - provide any <export> flag of a metapackage that is available
 - same for future tools like reporting errors with a "stack", when user cannot pinpoint exact package, bugtracker/maintainer information

- possibly offer the same dependency wrapping capabilites as used by message_generation and message_runtime packages

I am neither too sure nor passionate about the last one. I guess the question is whether all metapackages should also have a find_package infrastructure so that find_package could be used on a metapackage to get build flags. I just regard message_generation and message_runtime as a special kind of packages that could as well be called metapackages if we changed our view of metapackages slightly (and auto-generated their CMakeLists.txt). The point here is that if users can do that dependency wrapping anyway with normal packages, we might as well provide it with metapackages.

> To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

[Jack O'Quin]

On Tue, Jan 22, 2013 at 7:43 AM, tkruse <tibo...@googlemail.com> wrote:

Here are the functionalities I care about:

- provide rosbuild compatibility where stacks are resolved instead to meta-packages (rather than accepting absence of a stack dependency)

+1; this is needed for tick-tock, providing an interim compatibility period while dividing a package or stack into separate components, often needed to remove unwanted GUI dependencies.

 

- provide the same meta-information of metapackages to tools in installspace as in develspace, instead of losing metapackages halfway.

+1; the convenience advantage of develspace is less to the extent it behaves differently from installspace.

 

 - rqt_package_graph/rospack/catkin_pkg/catkin_find functions should return the same results

+1

 

 - provide any <export> flag of a metapackage that is available

+1

 

 - same for future tools like reporting errors with a "stack", when user cannot pinpoint exact package, bugtracker/maintainer information

+1

- possibly offer the same dependency wrapping capabilites as used by message_generation and message_runtime packages

I am neither too sure nor passionate about the last one. I guess the question is whether all metapackages should also have a find_package infrastructure so that find_package could be used on a metapackage to get build flags. I just regard message_generation and message_runtime as a special kind of packages that could as well be called metapackages if we changed our view of metapackages slightly (and auto-generated their CMakeLists.txt). The point here is that if users can do that dependency wrapping anyway with normal packages, we might as well provide it with metapackages.

Why not merge metapackages and dependency-only packages into a single concept? Right now that distinction is hard to explain, especially why a user should choose one or the other.

--
 joq

[Dirk Thomas]

We have talked about metapackages/variants/grouping in out meeting today and came up with the following points:

* design goal is to provide grouping of packages

* for each �grouping entity� a Debian package exists to ease installation of a set of packages

* but we do not want plain packages to depend on �grouping entities� (like metapackages and variant)
* why? because this would bloat the dependencies significantly
* therefore �grouping entities� must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

* �grouping entities� should have meta information and they should be available in install space

* �grouping entities� are add-ons and not mandatory to operate


Based on this list the implementation could look like:

* metapackage provide this grouping for wet, variants provide this grouping for dry (variants do not provide meta information)
* in the future all variants will become metapackages when all underlaying packages have been catkinized

* the package.xml of metapackages gets installed

* it is checked and prevented that plain packages depend on metapackages

* try to let rosmake explicitly check for metapackages (as a replacement of former stack dependencies) instead of just ignoring them


Wiki grouping is explicitly not part ob the above consideration.
The need for Wiki grouping is not a one-to-one mapping to metapackages.
It should be realized individually to use it not only for these group but flexible for grouping packages of a repo, grouping packages of a release/packaging unit.
This could be achieve with any of the following approaches:

* wiki grouping could be triggered by an export tag (i.e. wiki_group) which will either use all declared run deps

* the export tag could even whitelist the to-be-grouped packages (which would allow more control but be more verbose)

* could be achieved in-the-wiki only (instead if being extracted from meta information)

I hope I catched the essence of our meeting and am looking forward for feedback.

- Dirk

[Jonathan Bohren]

On Tue, Jan 22, 2013 at 7:54 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

* design goal is to provide grouping of packages

* for each “grouping entity” a Debian package exists to ease installation of a set of packages

* but we do not want plain packages to depend on “grouping entities” (like metapackages and variant)

  * why? because this would bloat the dependencies significantly

  * therefore “grouping entities” must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

* “grouping entities” should have meta information and they should be available in install space

* “grouping entities” are add-ons and not mandatory to operate

Based on these points, I have to questions:

1. If a given developer has no intention to generate debian (or other binary) packages for his or her ROS package, is there any use for metapackages? 

2. If metapackages can't be dependencies of other packages, and don't have any content, should they be described by package.xml files?

-j

[Dirk Thomas]

On 22.01.2013 17:43, Jonathan Bohren wrote:

>
> On Tue, Jan 22, 2013 at 7:54 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dth...@willowgarage.com>> wrote:
>
> * design goal is to provide grouping of packages
>

> * for each �grouping entity� a Debian package exists to ease installation of a set of packages
>
> * but we do not want plain packages to depend on �grouping entities� (like metapackages and variant)

> * why? because this would bloat the dependencies significantly

> * therefore �grouping entities� must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)
>
> * �grouping entities� should have meta information and they should be available in install space
>
> * �grouping entities� are add-ons and not mandatory to operate

>
>
> Based on these points, I have to questions:
>
> 1. If a given developer has no intention to generate debian (or other binary) packages for his or her ROS package, is there any use for metapackages?

I guess no.

> 2. If metapackages can't be dependencies of other packages, and don't have any content, should they be described by package.xml files?

The consens on the SIG in fall was that we still want meta information attached to such entities, meaning a maintainer name and email, a url for website / bug tracker, a description, license info.
Since package.xml pretty much covers exactly those we would reuse the same spec in order to not have another spec with another API to parse and access the data.

- Dirk

[tkruse]

On Wednesday, January 23, 2013 1:54:50 AM UTC+1, Dirk Thomas wrote:

* metapackage provide this grouping for wet, variants provide this grouping for dry (variants do not provide meta information)
   * in the future all variants will become metapackages when all underlaying packages have been catkinized

So will there be meta-packages for desktop-full and such?
 

* it is checked and prevented that plain packages depend on metapackages

Do we also check that metapackages have no build, buildtool and test dependencies?
 

* try to let rosmake explicitly check for metapackages (as a replacement of former stack dependencies) instead of just ignoring them

 

Wiki grouping is explicitly not part ob the above consideration. The need for Wiki grouping is not a one-to-one mapping to metapackages.

I believe this one needs more explaining. The current wiki grouping is based on the original stacks. Stacks grouped packages, and the following group properties applied:
- packages shared the same vcs repo-location, e.g. one repo per stack, or one svn canonical root per stack
- packages were semantically strongly coupled (foo, foo_tests, foo_tutorials)
- packages were released as stacks

So now that stacks are gone, we are free to group in the wiki by any of the above, or by metapackage, or any other criteria.
Part of the convenience we got was that users could go to the wiki to find out about the source location of packages. What you checked out is the group as a whole.

A thing that was also discussed was where to put xyz_tutorials packages, and I believe the same goes for xyz_tests packages, in particular packages testing functionalitites spanning several packages, like 2d-navigation. (I call them integration tests). It makes some sense to put such packages into the same vcs repo as the sources, as the tests and tutorials should be changed with the source. On the other hand, it may be good to have "variants" that install all packages of such a group or just the ones providing core functionality.
So e.g. there could be debians like: navigation_core, navigation_experimental, navigation_tutorials, navigation_tests, navigation(_full), all spanning packages of the navigation "group". And then the question would be how to display that in the wiki.

So I see different kinds of groups for different purposes. I still believe a group for documentation that should be mandatory should be "packages that live in the same repository root", meaning there should be a grouping which is safe from being "invaded" by packages from a different repo. And this level of grouping is the one we talked about in fall I believe, the one that we want further meta-information like licensing and maintainer and bugtracker and such. I believe this is what metapackages currently are, unless I am mistaken we currently do not have any metapackage that defines a different level of grouping. I would also like very much that we enforce in the indexer that there is a one-to-one relationship between packages and this level of grouping. (Currently we e.g. have the situation that python_orocos_kdl exists twice in the index, as a dry package part of a dry stack, and as a wet package. I believe that if we allow that, we do not gain anything, but lose some consistency. See https://github.com/ros/rosdistro/issues/356)

Then "variants" as in REP131 provide grouping that usually span packages from several sources, and above I have shown possible groupings that are a subset of the packages from one a repository. Whether those need meta-information is still open to debate, I believe, just as how they should be realized technically.

While grouping is currently optional, I consider it very valuable. E.g. if one currently looks at a graph of package dependencies with rqt package graph, then the dry packages still provide some structure via stacks, whereas wet packages create a furball in the graph. And it is the same for any other view on a large set of packages, whether you use a commandline list or a spreadsheet or a wiki.

It should be realized individually to use it not only for these group but flexible for grouping packages of a repo, grouping packages of a release/packaging unit.

This could be achieved with any of the following approaches:

* wiki grouping could be triggered by an export tag (i.e. wiki_group) which will either use all declared run deps [or a subset]

* the export tag could even whitelist the to-be-grouped packages (which would allow more control but be more verbose)

* could be achieved in-the-wiki only (instead if being extracted from meta information)

So here is my personal opinion on these suggestions.
I am against all non-generated groupings in the wiki, as those also need to be maintained by hand, and quickly go out-of-date.
I remember and idea in the fall discussions on other semantic tagging the same way pypi allows classifying keyword hierarchies (http://pypi.python.org/pypi?%3Aaction=list_classifiers). This can help in the wiki to find packages on subjects (it is a kind of grouping I did not mention before). This is also a nice idea for grouping that can also be displayed in the wiki, but we need no meta-information like maintainer or license for those, and this kind of grouping does not help much with the current purpose of meta-packages.

To create smaller-than-vcs groups like navigation_core, navigation_tutorials (if others see a benefit in that), I currently would prefer this to be very closely tied to current metapackages. So e.g. what was a dry stack would still stay a wet metapackage (as agreed for Groovy), but we allow to define smaller groups than the full "stack" to be part of a packaging set. Such subgroups could IMO be defined inside the package.xml of the metapackage, for easy consistency checks by humans. For large variants, like "desktop-full", I also see no need to maintain the meta-information we have for metapackages, license, maintainer and such. So I'd rather not maintain those using the meta-package infrastructure. But I would like a technical solution for those.

Currently we have
ROS Diamondback Variants http://www.ros.org/reps/rep-0108.html
ROS Electric Variants http://www.ros.org/reps/rep-0113.html
ROS Fuerte --
ROS Groovy Variants http://www.ros.org/reps/rep-0131.html

So there is a break in the variants in the REPs, and it also shows in the rosinstall_generators. AFAIK there is no way to get a "ros-base" or "ros-full" variant rosinstall for fuerte (nor is there a debian). This makes it also hard to move documentation from one distro to the next, and so on. I find it clumsy that we'd need a REP to define variants per distro, I'd rather only have a REP if a variant of major importance was dropped, added or changed, and else keep the variants stable over distros.

[Dirk Thomas]

On 23.01.2013 03:18, tkruse wrote:
>
>
> On Wednesday, January 23, 2013 1:54:50 AM UTC+1, Dirk Thomas wrote:
>
>
> * metapackage provide this grouping for wet, variants provide this grouping for dry (variants do not provide meta information)
> * in the future all variants will become metapackages when all underlaying packages have been catkinized
>
>
> So will there be meta-packages for desktop-full and such?

Since metapackages only exist in the wet world, no - or at least not yet.
If the grouping entity depends on dry stuff it stays a variant.
As soon as it only depends on wet stuff (which will be nearly everything for Hydro) the variant will be changed into a metapackage.

> * it is checked and prevented that plain packages depend on metapackages
>
>
> Do we also check that metapackages have no build, buildtool and test dependencies?

+1 absolutely.

- Dirk

[Geoffrey Biggs]

On Jan 23, 2013, at 9:54 AM, Dirk Thomas wrote:
> * but we do not want plain packages to depend on “grouping entities” (like metapackages and variant)

> * why? because this would bloat the dependencies significantly

> * therefore “grouping entities” must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

With this requirement, you eliminate the ability of a package to easily require a sub-system. For example, if I have a package that wants the robot to be able to navigate, I would ideally just provide a dependency on the metapackage for the navigation package of my choice (or, in the Gentoo world, I say "something providing navigation must be installed" and portage pulls in the system default one if another is not already available, which may or may not be a metapackage). Instead, I must either make my package depend on every package involved in navigation that it interacts with directly (I'm assuming here that those will pull in the remaining packages based on their own dependencies), which is a problem when that metapackage's contents change without changing the external interface, or I need to make a metapackage for my package so I can set a dependency.

Which raises a related question: can “grouping entities” depend on “grouping entities”?

> Wiki grouping is explicitly not part ob the above consideration.
> The need for Wiki grouping is not a one-to-one mapping to metapackages.
> It should be realized individually to use it not only for these group but flexible for grouping packages of a repo, grouping packages of a release/packaging unit.
> This could be achieve with any of the following approaches:
>
> * wiki grouping could be triggered by an export tag (i.e. wiki_group) which will either use all declared run deps

I like this approach. It gives the developer much more control. I think that a package should be able to belong to any and as many groups as the developer thinks are suitable.

>
> * the export tag could even whitelist the to-be-grouped packages (which would allow more control but be more verbose)
>
> * could be achieved in-the-wiki only (instead if being extracted from meta information)

I agree with others that not doing it automatically is a dangerous idea.

Geoff

[Jonathan Bohren]

On Thu, Jan 24, 2013 at 12:57 AM, Geoffrey Biggs <gbi...@killbots.net> wrote:

Which raises a related question: can “grouping entities” depend on “grouping entities”?

If packages CAN'T depend on "grouping entities" but "grouping entities" CAN depend on other "grouping entities" then we're back at rosbuild-type stacks, aren't we?

When they were introduced, I thought the point of meta-packages was to have the grouping capabilities of stacks but without creating a parallel dependency tree to the package dependencies (i.e. packages would be able to depend on meta-packages).

-j

[tkruse]

On Thursday, January 24, 2013 6:57:34 AM UTC+1, Geoffrey Biggs wrote:

On Jan 23, 2013, at 9:54 AM, Dirk Thomas wrote:
> * but we do not want plain packages to depend on “grouping entities” (like metapackages and variant)
>  * why? because this would bloat the dependencies significantly
>  * therefore “grouping entities” must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

With this requirement, you eliminate the ability of a package to easily require a sub-system. For example, if I have a package that wants the robot to be able to navigate, I would ideally just provide a dependency on the metapackage for the navigation package of my choice (or, in the Gentoo world, I say "something providing navigation must be installed" and portage pulls in the system default one if another is not already available, which may or may not be a metapackage). Instead, I must either make my package depend on every package involved in navigation that it interacts with directly (I'm assuming here that those will pull in the remaining packages based on their own dependencies), which is a problem when that metapackage's contents change without changing the external interface, or I need to make a metapackage for my package so I can set a dependency.

 
You can still use a non-metapackage to achieve this, where it makes sense. We cannot prevent users from creating package the same way as message_generation, without content, just wrapping several others for convenience. So the idea here is only to discourage this practice as much as possible by not allowing to use metapackages for that purpose.

Also usually you do not have to explicitly depend on every package involved in navigation, as those have inter-dependencies. You only depend on those that you reference in your code, and those packages pull in whatever other navigation package they require.

Which raises a related question: can “grouping entities” depend on “grouping entities”?

Grouping entities in the current design can only have runtime dependencies, I believe, so this only becomes relevant for packaging, not building. And so far nobody raised concerns over this being a problem for packaging, the concerns are only using such dependencies for building.

[tkruse]

On Thursday, January 24, 2013 7:04:11 AM UTC+1, Jonathan Bohren wrote:

On Thu, Jan 24, 2013 at 12:57 AM, Geoffrey Biggs <gbi...@killbots.net> wrote:

Which raises a related question: can “grouping entities” depend on “grouping entities”?

If packages CAN'T depend on "grouping entities" but "grouping entities" CAN depend on other "grouping entities" then we're back at rosbuild-type stacks, aren't we?

When they were introduced, I thought the point of meta-packages was to have the grouping capabilities of stacks but without creating a parallel dependency tree to the package dependencies (i.e. packages would be able to depend on meta-packages).

This was never a goal in itself, but meant to avoid a problem. Within a stack, when the dependencies of a package withint the stack changed, the stack manifest had to also be changed for consistency, which caused the maintainer to have to look up all stacks of all packages that were depended upon, and compare that list to the list of stack dependencies in the stack.xml. This is what we want to avoid.

With grouping entities the situation is somewhat different, as the grouping dependencies do not have to match package dependencies. You can group for a variety of reasons. You could make a group consisting of a navigation group and an object recognition group, without navigation depending on recognition or vice-versa. They'd be in a group not because they have to be for building (as was the case with stacks) but because you want them to.

Since this only affects packaging, you can never "miss" a dependency that way, as packaging will take care that all transitive dependencies have also to be met. So just because the dependencies of a package within a group changes e.g. to depend on a new package, this does not man you have to change the group. So that problem we had with stacks is avoided.

I am not 100% sure whether that was the only reason we did not want a parallel dependency tree, though.

[Jack O'Quin]

Let's pay attention to the solutions provided by various Linux distributions. They have decades of experience packaging tens of thousands of packages.

I am speaking below in Debian terms, because it is well-documented and of immediate interest to ROS developers. But, other packaging systems, such as Red Hat and Gentoo, provide similar solutions to meet similar needs.

On Wed, Jan 23, 2013 at 11:57 PM, Geoffrey Biggs <gbi...@killbots.net> wrote:

On Jan 23, 2013, at 9:54 AM, Dirk Thomas wrote:
> * but we do not want plain packages to depend on “grouping entities” (like metapackages and variant)
>  * why? because this would bloat the dependencies significantly
>  * therefore “grouping entities” must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

With this requirement, you eliminate the ability of a package to easily require a sub-system. For example, if I have a package that wants the robot to be able to navigate, I would ideally just provide a dependency on the metapackage for the navigation package of my choice (or, in the Gentoo world, I say "something providing navigation must be installed" and portage pulls in the system default one if another is not already available, which may or may not be a metapackage). Instead, I must either make my package depend on every package involved in navigation that it interacts with directly (I'm assuming here that those will pull in the remaining packages based on their own dependencies), which is a problem when that metapackage's contents change without changing the external interface, or I need to make a metapackage for my package so I can set a dependency.

The Debian solution for generic dependencies is "virtual packages", which are more powerful than our current metapackage discussion. ROS probably needs something similar, so people can provide various alternatives to sub-systems like 2D navigation, where one solution clearly does not meet every need.

 http://www.debian.org/doc/debian-policy/ch-binary.html#s-virtual_pkg

A virtual package is not physically installed. Instead, some actual package that "Provides" that interface is needed.

 http://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual

The ROS community would define the expected attributes of various virtual packages via REPs. Here is the official Debian list:

 http://www.debian.org/doc/packaging-manuals/virtual-package-names-list.txt

Maybe Groovy metapackages could evolve in this direction. 

Otherwise, I don't see much use for metapackages, other than as surrogate dependencies for dry stacks. For most other purposes, an actual package with its own catkin CMakeLists.txt works better. 

 * I can still serve as a documentation node.

 * It can conveniently be used to compile an entire collection of "component" packages, just as rosmake did for stacks. 

 * It can be also used for collective external dependencies, where appropriate. 

 * It can provide a tick-tock target for compatibility when migrating a package or stack to a different internal organization.

 * It could "Provide" any appropriate virtual interface, if we choose to follow that example.

--
 joq

[Geoff]

I agree with Jack's approach.

 

On Fri, Jan 25, 2013, at 3:03, Jack O'Quin wrote:

Let's pay attention to the solutions provided by various Linux distributions. They have decades of experience packaging tens of thousands of packages.

 

I am speaking below in Debian terms, because it is well-documented and of immediate interest to ROS developers. But, other packaging systems, such as Red Hat and Gentoo, provide similar solutions to meet similar needs.

 

On Wed, Jan 23, 2013 at 11:57 PM, Geoffrey Biggs <gbi...@killbots.net> wrote:

On Jan 23, 2013, at 9:54 AM, Dirk Thomas wrote:

> * but we do not want plain packages to depend on “grouping entities” (like metapackages and variant)

>  * why? because this would bloat the dependencies significantly

>  * therefore “grouping entities” must not provide any further functionality (no header, libraries, scripts, cmake, pkgconfig, exports)

With this requirement, you eliminate the ability of a package to easily require a sub-system. For example, if I have a package that wants the robot to be able to navigate, I would ideally just provide a dependency on the metapackage for the navigation package of my choice (or, in the Gentoo world, I say "something providing navigation must be installed" and portage pulls in the system default one if another is not already available, which may or may not be a metapackage). Instead, I must either make my package depend on every package involved in navigation that it interacts with directly (I'm assuming here that those will pull in the remaining packages based on their own dependencies), which is a problem when that metapackage's contents change without changing the external interface, or I need to make a metapackage for my package so I can set a dependency.

 

The Debian solution for generic dependencies is "virtual packages", which are more powerful than our current metapackage discussion. ROS probably needs something similar, so people can provide various alternatives to sub-systems like 2D navigation, where one solution clearly does not meet every need.

 

http://www.debian.org/doc/debian-policy/ch-binary.html#s-virtual_pkg

 

A virtual package is not physically installed. Instead, some actual package that "Provides" that interface is needed.

 

http://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual

 

The ROS community would define the expected attributes of various virtual packages via REPs. Here is the official Debian list:

 

http://www.debian.org/doc/packaging-manuals/virtual-package-names-list.txt

 

Maybe Groovy metapackages could evolve in this direction. 

 

As I hinted at earlier, Gentoo uses a very similar system. Here are some details:

 

A "virtual" package provides a bottleneck for depending on a piece of *functionality* rather than a specific package that provides that functionality.

 

Others in this thread have mentioned that we should just depend on the parts of a collection of packages that we directly use and let those pull in their own dependencies, but this goes against current software engineering best practice regarding inter-operability of software.

 

Gentoo uses virtuals to achieve replaceable packages: if you require that a Java virtual machine be installed, but multiple implementations are available, you need some way to satisfy the dependency of packages that want Java without them all needing to specify all the options available.

 

Here's Gentoo's virtual package for the Java Development Kit, which is named virtuals/jdk:

 

 

# Copyright 1999-2012 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: /var/cvsroot/gentoo-x86/virtual/jdk/jdk-1.7.0.ebuild,v 1.9 2012/12/08 04:09:48 zerochaos Exp $

 

DESCRIPTION="Virtual for JDK"

HOMEPAGE=""

SRC_URI=""

 

LICENSE=""

SLOT="1.7"

KEYWORDS="~amd64 ~ia64 ~x86 ~amd64-linux ~x86-linux ~ppc-macos ~sparc-solaris ~sparc64-solaris ~x64-solaris ~x86-solaris"

IUSE=""

 

RDEPEND="|| (

                =dev-java/icedtea-bin-7*

                =dev-java/icedtea-7*

                =dev-java/oracle-jdk-bin-1.7.0*

                =dev-java/soylatte-jdk-bin-7*

        )"

DEPEND=""

 

It's a very simple system: a virtual package is simply a package with no content that specifies a set of OR'd dependencies. If none is currently installed, Portage will install the first package in the list that satisfies the dependency. You can see version management going on, too.

 

 

Moving on to other functionality from the wishlist: grouping of packages. In Portage 2.1, the current release version, Gentoo does this in a similar way. It calls these packages "meta-ebuilds," (Gentoo's packages are called ebuilds) but the name doesn't really mean anything special. Here's the meta-ebuild for KDE:

 

# Copyright 1999-2013 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: /var/cvsroot/gentoo-x86/kde-base/kde-meta/kde-meta-4.9.5.ebuild,v 1.1 2013/01/05 20:19:03 creffett Exp $

 

EAPI=4

inherit kde4-meta-pkg

 

DESCRIPTION="KDE - merge this to pull in all split kde-base/* packages"

KEYWORDS="~amd64 ~arm ~ppc ~x86 ~amd64-linux ~x86-linux"

IUSE="accessibility nls sdk semantic-desktop"

 

RDEPEND="

        $(add_kdebase_dep kate)

        $(add_kdebase_dep kdeadmin-meta)

        $(add_kdebase_dep kdeartwork-meta)

        $(add_kdebase_dep kdebase-meta)

        $(add_kdebase_dep kdeedu-meta)

        $(add_kdebase_dep kdegames-meta)

        $(add_kdebase_dep kdegraphics-meta)

        $(add_kdebase_dep kdemultimedia-meta)

        $(add_kdebase_dep kdenetwork-meta)

        $(add_kdebase_dep kdeplasma-addons)

        $(add_kdebase_dep kdetoys-meta)

        $(add_kdebase_dep kdeutils-meta)

        accessibility? ( $(add_kdebase_dep kdeaccessibility-meta) )

        nls? ( $(add_kdebase_dep kde-l10n) )

        sdk? (

                $(add_kdebase_dep kdebindings-meta)

                $(add_kdebase_dep kdesdk-meta)

                $(add_kdebase_dep kdewebdev-meta)

        )

        semantic-desktop? (

                $(add_kdebase_dep kdepim-meta "" 4.4.11.1)

        )"

 

Apart from some bash functions to simplify the ebuild, it's doing nearly the same thing as a virtual. The only difference is that it is an AND combination rather than OR. Because meta-ebuilds are themselves ebuilds, they can also depend on each other: the KDE meta-ebuild above does its work by mostly pulling in meta-ebuilds for the major components of KDE.

 

(As an aside, this meta-package is also showing off Gentoo's use flags facility: some parts of KDE will only be installed if the user has specified that they want that functionality.)

 

In Portage 2.2, which is currently under development, meta-packages have been replaced with sets. A set is essentially a list of packages and sets. It seems uncertain whether they will be adopted in the final version 2.2 release or if everyone will go back to meta-packages. An advantage of the set system is that Portage can make dynamic sets. For example, the list of all packages currently installed is maintained as the "world" set.

 

 

I also remembered while writing this email that I promised to discuss the eselect tool and how it sets a different Java VM for each user and for the system. I looked it up and it's actually quite simple:

 

- A configuration directory in the user's home (stored under ~/.gentoo) contains a symlink to the directory of the currently-selected JVM for that user. For example, on mine it is:

$ ls -l .gentoo/java-config-2/current-user-vm

lrwxrwxrwx 1 geoff geoff 27 Oct 24 18:33 .gentoo/java-config-2/current-user-vm -> /usr/lib/jvm//icedtea-bin-6

 

- A system-wide setting contains a symlink to the directory of the JVM selected for the system. Mine currently is:

$ ls -l /etc/java-config-2/current-system-vm

lrwxrwxrwx 1 root root 26 Feb 20  2012 /etc/java-config-2/current-system-vm -> /usr/lib/jvm/icedtea-bin-6

 

- /usr/bin/java is symlinked to a bash script called run-java-tool. The first thing this does is check which JVM should be called. It tries three options, in order, until it finds one that is set. The first is the value of ${GENTOO_VM}, the second is the user's JVM, and the third is the system JVM (which is always set). After some sanity checks to make sure the chosen value is valid, it executes the JVM.

 

- The eselect module for java-vm sets the symlink in your home directory or (if you have permission) the system directory according to which JVM you have chosen.

 

This has turned out to be a long email, but I hope the information will be of some use in designing the ROS packaging system.

 

I won't comment on Arch because it's been too long now since I used it, but I know that it does a lot of similar things to Gentoo in its package management.

 

 

 

Otherwise, I don't see much use for metapackages, other than as surrogate dependencies for dry stacks. For most other purposes, an actual package with its own catkin CMakeLists.txt works better. 

 

 * I can still serve as a documentation node.

 

 * It can conveniently be used to compile an entire collection of "component" packages, just as rosmake did for stacks. 

 

 * It can be also used for collective external dependencies, where appropriate. 

 

 * It can provide a tick-tock target for compatibility when migrating a package or stack to a different internal organization.

 

 * It could "Provide" any appropriate virtual interface, if we choose to follow that example.

 

Based on what has been said by others, I agree with Jack's approach. I don't see much benefit in having meta-packages as a separate entity in the catkin-only world, unless there is no other way to produce the Debian meta-packages.

 

Developer guidelines ("don't put installed resources in your meta-packages") and an XML tag for wiki grouping could achieve the same functionality that is currently desired with less complexity.

 

Geoff

[tkruse]

So, here are some of the things we discussed in a meeting (Dirk, William and me), with respect to how metapackages shall evolve in the future. This does not yet answer all questions and comments, I realize,so a second round might be necessary next week, but feel free to already comment.


Regarding Virtual Packages, this seems like a valid new feature, but it is orthogonal to the other issues currently mentioned around metapackages. Also the need for this feature seems not that urgent, so unless some volunteer wants to write up a REP, prototype and such, it is not going to happen anytime soon. Anyone might open a ticket so that others may upvote it.


The package.xml for metapackages shall be installed by catkin soon, Dirk volunteered to implement this change.


Some recommendations for how metapackages shall be used in the future (Comments welcome):

- metapackages shall remain the mechanism to generate debian packages that group smaller debian packages

- Variants like ros-base, ros-full, robot, desktop, etc. shall be implemented as metapackages as soon as the packages they point to are catkinized. Those metapackages are to live in some github repo(s). Those would then also get respective wiki pages. The difference between a variant and a metapackage is then technically none, maybe variant could be reserved for groups listed in a REP.

- Other groupings via metapackages are completely up to users,for now no recomendation (except to keep metapackages for rosbuild backward compatibility)

- metapackages may live in any repo, though when they only point to packages in one repo, it makes sense to let the metapackage live there as well.

- wiki pages should continue to show metapackage grouping (as opposed to have a different mechanism for grouping in the wiki)

- the metapackage relationship thus is different from normal package dependencies in that it appears differently in the wiki

- "Related packages" can already be manually entered in the wiki via mere wiki links, currently no further support seem urgent

- wiki pages for packages currently already support the case when a package is pointed to by several metapackages (e.g. http://www.ros.org/wiki/ROS). This can become ugly if many meta-packages exist and overlap, so once we have that, we may need to rethink the wiki layout for this (ideas welcome)

- users may create several metapackages for a single "bunch" of packages, like xyz_core, xyz_full, xyz_experimental,... we currently give no recommendation on bundling and naming

- metapackage manifest information shall only be for the metapackage itself (e.g. maintainer, license)



Open questions / comments (That I intent to raise in the next meeting unless they are resolved here):

- We did not discuss package tags for package lookup, such as used by pypi.

- Why forbid packages to run_depend (not build_depend) on metapackages for convenience, even if we dislike that approach?

- Why not allow declaring multipe groups within the same package.xml of a metapackage? In particular if they share meta-information.

- Should rosmake strictly check stack dependencies against metapackages in the future (one package.xml gets installed)?

- The allowed properties of metapackage's package.xml should be documented.


This has been a long thread, and reading it twice is not much fun, so if you feel I missed something that still should be discussed, it's safer if you comment again.

[tkruse]

Okay, so here are answers to the open questions I listed:

• We did not discuss package tags for package lookup, such as used by pypi

•  rosbrowse now supports searching in description, good enough to search ona topic, so this feature has insufficient benefit to spend core maintainers time on it. Volunteers are welcome to provide REP + patches (though we don't expect that to happen)
  

• Why forbid packages to run_depend (not build_depend) on metapackages for convenience, even if we dislike that approach?

• metapackages cannot be find_packaged, so consistency of package.xml and CMakeLists.txt cannot easily be reviewed (it is not well-checked by catkin anyway)
  

• Why not allow declaring multipe groups within the same package.xml of a  metapackage? In particular if they share meta-information.

• to not break one-to-one relationship package-debian

• Should rosmake strictly check stack dependencies against metapackages in the future (one package.xml gets installed)?

• Yes, to be changed as soon as metapackage package.xml is being installed

• The allowed properties of metapackage's package.xml should be documented.

• Will be added to REP127

[Tully Foote]

On Wed, Jan 30, 2013 at 7:59 AM, tkruse <tibo...@googlemail.com> wrote:

So, here are some of the things we discussed in a meeting (Dirk, William and me), with respect to how metapackages shall evolve in the future. This does not yet answer all questions and comments, I realize,so a second round might be necessary next week, but feel free to already comment.


Regarding Virtual Packages, this seems like a valid new feature, but it is orthogonal to the other issues currently mentioned around metapackages. Also the need for this feature seems not that urgent, so unless some volunteer wants to write up a REP, prototype and such, it is not going to happen anytime soon. Anyone might open a ticket so that others may upvote it.

To keep track of this I opened a ticket for it, but put it into the Untargeted milestone.   https://github.com/ros/rosdistro/issues/438

The package.xml for metapackages shall be installed by catkin soon, Dirk volunteered to implement this change.


Some recommendations for how metapackages shall be used in the future (Comments welcome):

- metapackages shall remain the mechanism to generate debian packages that group smaller debian packages

- Variants like ros-base, ros-full, robot, desktop, etc. shall be implemented as metapackages as soon as the packages they point to are catkinized. Those metapackages are to live in some github repo(s). Those would then also get respective wiki pages. The difference between a variant and a metapackage is then technically none, maybe variant could be reserved for groups listed in a REP.

- Other groupings via metapackages are completely up to users,for now no recomendation (except to keep metapackages for rosbuild backward compatibility)

- metapackages may live in any repo, though when they only point to packages in one repo, it makes sense to let the metapackage live there as well.

- wiki pages should continue to show metapackage grouping (as opposed to have a different mechanism for grouping in the wiki)

- the metapackage relationship thus is different from normal package dependencies in that it appears differently in the wiki

- "Related packages" can already be manually entered in the wiki via mere wiki links, currently no further support seem urgent

- wiki pages for packages currently already support the case when a package is pointed to by several metapackages (e.g. http://www.ros.org/wiki/ROS). This can become ugly if many meta-packages exist and overlap, so once we have that, we may need to rethink the wiki layout for this (ideas welcome)

- users may create several metapackages for a single "bunch" of packages, like xyz_core, xyz_full, xyz_experimental,... we currently give no recommendation on bundling and naming

- metapackage manifest information shall only be for the metapackage itself (e.g. maintainer, license)



Open questions / comments (That I intent to raise in the next meeting unless they are resolved here):

- We did not discuss package tags for package lookup, such as used by pypi.

- Why forbid packages to run_depend (not build_depend) on metapackages for convenience, even if we dislike that approach?

- Why not allow declaring multipe groups within the same package.xml of a metapackage? In particular if they share meta-information.

- Should rosmake strictly check stack dependencies against metapackages in the future (one package.xml gets installed)?

- The allowed properties of metapackage's package.xml should be documented.


This has been a long thread, and reading it twice is not much fun, so if you feel I missed something that still should be discussed, it's safer if you comment again.

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.

To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/9n11pCdEON8J.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[tkruse]

An update from todays meeting on this.
There seems to be no clean way to support installation of metapackage package.xml manifests under the constraints set by REP127:
http://ros.org/reps/rep-0127.html#metapackage
"<metapackage/>
The empty tag is used to indicate that a package is a metapackage which implies that it does not have a CMakeLists.txt and therefore no content of any kind."

The problem is that a any catkin package should be buildable without a catkin workspace, just using cmake, and for metapackages if we want anything else than no-op, this requires a CMakeLists.txt.
We discussed the alternative of declaring and installing metapackages as part of packages, but this breaks more design, and also would only work for stack-like metapackages, not variant-like ones.
We discussed not deploying normal package manifests neither to make devel and installspace consistent in that respect. But that would break rospack and related tools.

So the currently favoured solution will be to allow CMakeLists.txt with metapackages. In order to encourage users to not fill their metapackages with anything else than just the installation of the package manifest itself, catkin will scan the CmakeLists.txt and warn if users put in more than the minimally required lines. This may still change if someone has a better idea, or other blockers appear.