Updated Packaging Proposal

[Tully Foote]

Hi everyone,

This may be long but please stick with it.  It came up earlier in this forum to consider returning to the design decision we made a while ago about having two levels of packaging, currently stacks and packages.  I wanted to explore this a little bit more.  

When the idea came up initially I started listing off the reasons for having the stacks and packages separate which we have learned over time.  These advantages are primarily that the overhead for maintainership is lower as the maintainer can release several packages at once.  By keeping strong isolation of the dependency trees, stacks depend on stacks and packages depend on packages, it also has the benefit that packages can be moved to different release units transparently to the user and developer, except that the stack dependencies need to be updated which is relatively simple.  One of the other driving factors about the stack system was that it significantly decreases the cognitive load when initially released, taking a few hundred packages and dividing them up into dozens of stacks.  

However as the ROS ecosystem is growing two things are going on.  First is the hundreds of packages have grown to thousands and the number of stacks has grown into the hundreds.  At the same time the ecosystem is maturing and the core components are stabilizing and there is more demand for fine grain control of dependencies and smaller release units.  As we’ve done this we’ve seen more and more stacks be converted to unary stacks with the package as the release unit.

Looking at the system as a whole we end up with two dependency trees, where the stack level one is entirely a derivative of the package level dependency tree.  As the entire ecosystem has grown, maintaining these parallel trees has become more of an effort for maintainers.  One of the obvious symptoms is that we created a script that automatically sets stack dependencies based on package dependencies.  Also I have observed a significant amount of time being spent discussing which stack a package should be released into to not muddy up the stack level dependency tree.  Especially when there are two closely related stacks, that can’t have a cyclic dependency, they often end up being broken up into 3.  And reallocating the contents of various stacks became a common occurrence.  Thus causing more effort and cognitive load on the maintainers where it originally saved.

There is also a major issue with discoverability in the dual layered system.  A case where this becomes an obvious problem is if you download an arbitrary stack to try out from an unknown source, and they have setup their package dependencies, but since they have not thought about releasing formally they have not setup their stack dependencies at all (this is not uncommon because everything they need is in their workspace and works fine ).  But as the potential new user, you are left with a list of package dependencies.  To be able to resolve this issue your only hope is that the packages you need are documented somewhere publicly and the documentation includes what stack they are inside of, so that you can find download and install that stack into your workspace.  There is the potential to possibly automatically compute the stack dependencies, but that requires an authoritative lookup table of what stack every package is contained within.  For publicly available packages the ros indexer generally can work, but considering you are working in a non public environment, such as a classroom with a private project repository, there’s no index available for you to use.  

Another discussion point is that so far in ROS, we have not had versioned dependencies or versioned .so files.  Although we’re not planning to enable this in the current round, we need to make sure we don’t prevent doing so in the future.  The standard solution is to version at the release unit which would suggest that we have versions of stacks.  However if we consider the capability of being able to move packages between stacks freely, it could easily happen that a package moves from a 1.5.x stack to a 1.3.x stack, and suddenly the package has its version number decreased.  And this breaks many assumptions of tools such as CMake’s find_package logic, which can detect versions of packages.  And if we are considering adding support for versioned .so files, suddenly you have the possibility of having two versions of the .so file with the same version number because the package changed stacks and the version number dropped, and then reincremented back up.

Another feature which has been mentioned in the discussions is to start providing the capabilities used in packaging such as replaces, provides, and conflicts flags for packaging.  And this could be woven into the dual layer system, but semantically we’d either have to have two namespaces like we do now, or else choose only one level to provide these capabilities in the future.  

I know there’s been several downsides mentioned to unbundling packages from stacks, the strongest I’ve heard is that there are some things that should be released together, (such as an SDK and drivers).  And the other issue mentioned has been that it will be more overhead in the actual release process, however as we scale we will need to be able to support releasing many units we will have to streamline the release process anyway (which is in process).

Proposed Solution:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.
• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.
• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  

Let us list the different pros/cons for such a solution compared to recently proposed solutions:
PROS:

• maintainers no longer need to maintain the dual redundant dependency trees
• things the previous proposed solution did not offer:
• backward compatibility with the wiki
• backward compatibility with the package dependencies
• multiple packages per repo
• faster build as we can parallelize the individual units (there was already a step in that direction but we now go further in granularity)
• finer granularity for dependencies
• packages can easily be moved to what was called a “unitary stack” or a different repo (just copy the folder, that’s it)
• packages will have their own version number:
• there is a direct upgrade path for providing versioned .so files (maybe that will be done in time for Groovy)
• finer granularity for releases
• version locking of packages (not through an artificial meta-unit) which enables us to release packages at the same time
• there is no more worrying about where a package is released and what the neighboring packages bring in as dependencies.
• no more stack.yaml/main CMakeLists.txt to maintain
• the concept of stack/variant are merged and are still here for the user for a simple install
• ROS packages are conceptually the same as Linux packages.  
• Indexing is easier (only one type of unit to index)
• Discovery is easier (all things in the same namespace are the same type)

CONS:

• increase of the number of Linux (and soon Mac/Windows) packages to build and offer the user by an order of magnitude
• upgrade requires moving some info from the stack.xml to all the manifest.xml (version number, authors/maintainers)
• we still need to adapt quite a few scripts: bloom, rosrelease, roscreate-pkg, ... (but most of those were not even adapted for the previously proposed solution). E.g. people should be able to release several packages at once just like they can do now with a stack (maybe with an automatic version bump). Those will be discussed individually.

Tully

[Geoffrey Biggs]

On Aug 7, 2012, at 8:11 AM, Tully Foote wrote:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.
• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.
• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  

+1

You just described the average Linux package manager. It certainly fits Portage perfectly. I fully support moving in this direction.

Geoff

[Daniel Stonier]

On 7 August 2012 08:11, Tully Foote <tfo...@willowgarage.com> wrote:

Proposed Solution:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.
• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.
• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  

Ready!

+1 here as well. 

[William Woodall]

I think ultimately this is a promising proposal, but honestly I am not sure about some of the consequences.

I agree that making the release unit for ROS as close to a Linux pkg as possible is a good idea, because it then translates well into other package systems like Homebrew.  I also agree that the overhead of maintaing two levels of dependencies is growing.

I am concerned, however, about the increased number of release units.  The meta-pkgs are an obvious solution that Linux distribution systems have converged on, but I am not sure how those translate to Windows or even Homebrew (looking into that now).  Additionally, I am concerned about how simple it will be to separately specify the version and authors for each ROS package, but I think it can be solved by properly designed tools.

+1, as long as we can properly mitigate those cons.

--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
William Woodall
Graduate Software Engineering
Auburn University
w...@auburn.edu
wjw...@gmail.com

williamjwoodall.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Bohren]

On Tue, Aug 7, 2012 at 1:11 AM, Tully Foote <tfo...@willowgarage.com> wrote:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.

+1

 

• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.

+1

 

• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  

+1

I think these properties will really simplify things even as catkin gives us more build flexibility.

-j

[tkruse]

On Tuesday, August 7, 2012 1:11:55 AM UTC+2, Tully Foote wrote:

Proposed Solution:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.
• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.
• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  
  

Will the meta-packages be managed like variants, centralized?
Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?
Does that mean that all bugtrackers for current stacks will have to be split up to defer tickets to the respective package?
Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?
Currently on can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. IN your scenario, I assume we can get a rosinstall for the meta-package "navigation" into out workspace. How about removing that meta-package from the workspace?

Can we have an API review page for that proposal to collect more Pros and Cons (unless we think you nailed all of them and we need no review)?

[Jonathan Bohren]

Tully, I hope you're alright with me creating a new review in your name:

http://ros.org/wiki/catkin/Reviews/2012-08-07_API_Review

-j

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/LvLwBoLt2-wJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[Andy Somerville]

Proposed Solution:

Based on these thoughts we propose the following solution (are you ready for it ?):

• we will release packages individually.
• we will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency.
• there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.  

I fully support this solution.

This would be best possible solution from the perspective of someone releasing software to deliverable systems.

+1 

[Dirk Thomas]

On 07.08.2012 02:06, tkruse wrote:
>
> On Tuesday, August 7, 2012 1:11:55 AM UTC+2, Tully Foote wrote:
>
> Proposed Solution:
>
> Based on these thoughts we propose the following solution (are you ready for it ?):
>

> * we will release packages individually.
> * we will have �meta-packages� which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled

> content that must agree with the package dependency.

> * there will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the

> future.
>
>
> Will the meta-packages be managed like variants, centralized?

Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.

> Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?

The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.

> Does that mean that all bugtrackers for current stacks will have to be split up to defer tickets to the respective package?

There is no fixed mapping between a package (or stack) and a bugtracker. Multiple packages from a former stack can still use the same bugtracker. It would even be possible that a single package uses
multiple bugtrackers (even if that looks pretty inconvenient). That is completely up the developer/maintainer. The feature request to store a bugtracker URL in the manifest looks like a good idea to
guide users to the right place.

> Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?

Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that package
(without the other packages of that repository). But that should not be an issue.

> Currently on can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. IN your scenario, I assume we can get a rosinstall for the meta-package
> "navigation" into out workspace. How about removing that meta-package from the workspace?

rosinstall currently gets its information form a .rosinstall file and does not rely on the DISTRO.yaml file. So there is no notion of a meta-package for rosinstall.

In a catkin workspace all packages overlay packages with the same name in parent workspaces (as it is in rosbuild).
New packages can be added by putting them in the workspace folder and removed by deleting the folder again.
With a global build folder per workspace a `make clean` would be necessary to remove all artifacts (this was already discussed in another topic).


> "there will be one xml file per package (the manifest.xml) which will have all the relevant meta data,": How does that relate to dependencies within the CMakeLists.txt of a package? I assume that packages will require a CMakelists to act like catkin-fuerte-stacks?

Each package will contain a CMakeLists.txt in the root folder which act the same as in a catkin-fuerte-stack. It used find_package() to find other packages. Some of the catkin macro are likely to
change and probably some are going away (like catkin_stack()).

- Dirk

[tkruse]

> Will the meta-packages be managed like variants, centralized?

Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.

 

To me that would be a CON:
The concept of centralized meta packages depends on Willow Garage / OSRF infrastructure, which is less flexible than decentralized models, such as meta-package.xml defining a meta-package without the restrictions of a stack.xml.

I will also guess that meta-packages will not declare dependencies to other meta-packages, but instead only depend on something like the union of the package dependencies (not sure how union works with versioned dependencies). So where before a stack might depend on 5 other stacks, now a meta-package will depend on 50 packages. That's more difficult to proof-read.
 

> Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?

The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.

 
To me that means we lose meta-information.
Consider the overview over stacks that a tool like the rosgui plugin Ros Package graph. The structural information is lost with the lack of stacks. Should this lack of structure be listed as a CON to the proposal? If meta-packages had an enforced 1-to-n restriction (each package must name one parent meta-package), the structure could be established. Also, again, by letting packages maintain their parent meta-package in the manifest, meta-packages can be established in a decentralized and automated fashion (with the caveat that typos may cause undesired meta-packages).

> Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?

Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that package
(without the other packages of that repository). But that should not be an issue.

Does that mean if multiple packages live in the same SCM subtree (e.g. git repo), that they will still need a common CMakelists.txt in the subtree root to be catkin compliant? 

> Currently on can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. IN your scenario, I assume we can get a rosinstall for the meta-package
> "navigation" into out workspace. How about removing that meta-package from the workspace?

rosinstall currently gets its information form a .rosinstall file and does not rely on the DISTRO.yaml file. So there is no notion of a meta-package for rosinstall.

AFAIK, the script "http://packages.ros.org/cgi-bin/gen_rosinstall.py" generates rosinstall data from the distro.yaml, and has a "variant" parameter
 

In a catkin workspace all packages overlay packages with the same name in parent workspaces (as it is in rosbuild).
New packages can be added by putting them in the workspace folder and removed by deleting the folder again.
With a global build folder per workspace a `make clean` would be necessary to remove all artifacts (this was already discussed in another topic).

I am talking about removing 10 out of 100 packages in a workspace, where the developer has no overview of which of the 100 packages are the 10 that belong to the meta-package he wants to remove. Currently to get rid of the "navigation" stack, a user removes the folder named "navigation". The new proposal means the user needs to find out what packages are part of navigation, and remove them one by one. This unless meta-packages are checked out as a single folder, or some other tool to help exists.

[Tully Foote]

I've added some responses below:

On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com> wrote:

> Will the meta-packages be managed like variants, centralized?

Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.

 

To me that would be a CON:
The concept of centralized meta packages depends on Willow Garage / OSRF infrastructure, which is less flexible than decentralized models, such as meta-package.xml defining a meta-package without the restrictions of a stack.xml.

To update the rosdistro for your own meta-package it's just a pull request away to get it into the mainline one now.  We've had great responses to updating the rosdeps that way.  Also all the toolchain is now parameterized to allow you to fork the rosdistro and use your own version if you'd like, or extend it and add it to your sources.list listing.  Another viewpoint is that part of the value of packaging is that you are making things common and facilitating a shared ontology.  

 

I will also guess that meta-packages will not declare dependencies to other meta-packages, but instead only depend on something like the union of the package dependencies (not sure how union works with versioned dependencies). So where before a stack might depend on 5 other stacks, now a meta-package will depend on 50 packages. That's more difficult to proof-read.

There's no reason that a variant can't depend on another one.  That's how things work at the moment.  See the bottom of fuerte.rosdistro: https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro

 

 

> Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?

The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.

 
To me that means we lose meta-information.
Consider the overview over stacks that a tool like the rosgui plugin Ros Package graph. The structural information is lost with the lack of stacks. Should this lack of structure be listed as a CON to the proposal? If meta-packages had an enforced 1-to-n restriction (each package must name one parent meta-package), the structure could be established. Also, again, by letting packages maintain their parent meta-package in the manifest, meta-packages can be established in a decentralized and automated fashion (with the caveat that typos may cause undesired meta-packages).

Requiring one "primary" meta-package is restrictive.  The  packageA may be a primary component of the  meta-packageA.  But it may also be a primary component of the meta-packageB.  If you require that packageA declare one parent meta-package, suddenly it can't be part of both? Or the other one is different/inferior somehow?  Also I would argue that requiring a developer of a library to rerelease their package to change it's meta-packaging is broken as the meta-packaging is truely a packaging discussion, and should not bother the developer for a patch release. Often this person is the same, but even if it's the same person the overhead of releasing is unnecessary/detrimental.  As changing the grouping of meta-packages does not effect the functionality in any way, as the meta-packaging does not override the dependencies.

 

> Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?

Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that package
(without the other packages of that repository). But that should not be an issue.

Does that mean if multiple packages live in the same SCM subtree (e.g. git repo), that they will still need a common CMakelists.txt in the subtree root to be catkin compliant? 

No they will not require anything connecting them.  We will need to upgrade rosinstall/vcstools to be able to specify subdirectories to checkout.  It's a less efficient way to use DVCSs from the storage aspect, but we expect to be more efficient from a speed and effort of development viewpoint.  

 

> Currently on can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. IN your scenario, I assume we can get a rosinstall for the meta-package
> "navigation" into out workspace. How about removing that meta-package from the workspace?

rosinstall currently gets its information form a .rosinstall file and does not rely on the DISTRO.yaml file. So there is no notion of a meta-package for rosinstall.

AFAIK, the script "http://packages.ros.org/cgi-bin/gen_rosinstall.py" generates rosinstall data from the distro.yaml, and has a "variant" parameter

Yes, that script is aware of rosdistro and variants, thus rosinstall is isolated from knowing about the distro/variants etc.  We could add a feature to rosinstall to do subtractive merging based on subsets of rosinstalls pass as arguments.  However, I expect a cleaner solution might be to 

 

 

In a catkin workspace all packages overlay packages with the same name in parent workspaces (as it is in rosbuild).
New packages can be added by putting them in the workspace folder and removed by deleting the folder again.
With a global build folder per workspace a `make clean` would be necessary to remove all artifacts (this was already discussed in another topic).

I am talking about removing 10 out of 100 packages in a workspace, where the developer has no overview of which of the 100 packages are the 10 that belong to the meta-package he wants to remove. Currently to get rid of the "navigation" stack, a user removes the folder named "navigation". The new proposal means the user needs to find out what packages are part of navigation, and remove them one by one. This unless meta-packages are checked out as a single folder, or some other tool to help exists.

I agree that I think a tool to help with this would be very valuable.  

For the simple case we can add a script which looks like:

for Package in my_variant.packages(); rm -rf Package; done;

However this will fall short because if you look back at the idea that packages can now be part of multiple meta-packages, removing all the packages in meta-packageA may break meta-packageB because there's an overlapping package or there's a package inside meta-packageA that is depended upon by a package in meta-packageB.  There's no way you can safely remove any set of packages without checking dependencies before you do so.  Thus a simple tool which allows you to list packages, select ones for removal(possibly by variant), and that also checks dependencies before you do so, with a prompt would be valuable. 

 Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/IVjBGpM_0xsJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[tkruse]

On Thursday, August 9, 2012 12:21:09 PM UTC+2, Tully Foote wrote:

I've added some responses below:

On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com> wrote:

> Will the meta-packages be managed like variants, centralized?

Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.

 

To me that would be a CON:
The concept of centralized meta packages depends on Willow Garage / OSRF infrastructure, which is less flexible than decentralized models, such as meta-package.xml defining a meta-package without the restrictions of a stack.xml.

To update the rosdistro for your own meta-package it's just a pull request away to get it into the mainline one now.  We've had great responses to updating the rosdeps that way.  Also all the toolchain is now parameterized to allow you to fork the rosdistro and use your own version if you'd like, or extend it and add it to your sources.list listing.  Another viewpoint is that part of the value of packaging is that you are making things common and facilitating a shared ontology.  

Does this also mean each corporate ROS community who develops non-open-source packages will have to create their own meta-packaging infrastructure and toolset, or will ROS provide tools to have a mix of company-internal distro.yaml mixed with the Willow Garage/OSRF one?

I will also guess that meta-packages will not declare dependencies to other meta-packages, but instead only depend on something like the union of the package dependencies (not sure how union works with versioned dependencies). So where before a stack might depend on 5 other stacks, now a meta-package will depend on 50 packages. That's more difficult to proof-read.

There's no reason that a variant can't depend on another one.  That's how things work at the moment.  See the bottom of fuerte.rosdistro: https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro

But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.
 

 

> Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?

The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.

 
To me that means we lose meta-information.
Consider the overview over stacks that a tool like the rosgui plugin Ros Package graph. The structural information is lost with the lack of stacks. Should this lack of structure be listed as a CON to the proposal? If meta-packages had an enforced 1-to-n restriction (each package must name one parent meta-package), the structure could be established. Also, again, by letting packages maintain their parent meta-package in the manifest, meta-packages can be established in a decentralized and automated fashion (with the caveat that typos may cause undesired meta-packages).

Requiring one "primary" meta-package is restrictive. The  packageA may be a primary component of the  meta-packageA.  But it may also be a primary component of the meta-packageB.

You seem to mix up meta-package being primary to a package and a package being primary to a meta-packages.
I am suggesting is that move_base should have a dedicated parent meta-package navigation, that does not exclude the possibility to define some other "uncle" meta-package containing move_base as well. However that relationship between move_base and navigation is stronger than that of other meta-packages.
I am not sure myself how to shape this into a clean concept, I just see that dropping restrictions also drops helpful structure. And I still fail to see how steadily growing number of packages motivates having less structure to keep an overview.

Also I would argue that requiring a developer of a library to rerelease their package to change it's meta-packaging is broken as the meta-packaging is truely a packaging discussion, and should not bother the developer for a patch release.

I'd say the setup.py of a python package also contains packaging information, and changing that also means rereleasing. I think that's also similar to the pom.xml of maven. So it is not generally regarded broken to maintain packaging information in the source tree. The general meta-packaging concept may be maintained anywhere, but the special primary parent meta-package concept, like between move_base and navigation, could be maintained in the package source, in my opinion, without being broken.

Also, even for general meta-packaging maintenance, it can be quite wise to maintain this in a more local VCS repository than the overall distro.yaml, even if that VCS does not include the source of the packages. So instead of having to do a pull request (comfortable as that may be), one coulde define a repository location which contains the meta-package information, just as one defines a repository location for packages. So not having the meta-packaging information in the source tree of a package does not mean we are left with the global distro.yaml as the only alternative.
 

 

> Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?

Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that package
(without the other packages of that repository). But that should not be an issue.

Does that mean if multiple packages live in the same SCM subtree (e.g. git repo), that they will still need a common CMakelists.txt in the subtree root to be catkin compliant? 

No they will not require anything connecting them.  We will need to upgrade rosinstall/vcstools to be able to specify subdirectories to checkout.  It's a less efficient way to use DVCSs from the storage aspect, but we expect to be more efficient from a speed and effort of development viewpoint.  

It's also new maintenance effort for vcstools / rosinstall, as well as burdening developers to understand the concept of checking out subdirectories of git or mercurial repos. What you gain by dropping rosstack you lose somewhere else.

Just to make sure, the only reason we do that (not having additional folders in catkin workspaces which are just parents to catkin projects) is because it was decided for catkin that recursively crawling the ROS_PACKAGE_PATH was unacceptably slower / unacceptably more complex than crawling the ROS_PACKAGE_PATH without recursion?

Meaning there was no actual hard technical constraint, just the opinion that it would be best to drop support for recursive crawls?

 

> Currently on can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. IN your scenario, I assume we can get a rosinstall for the meta-package
> "navigation" into out workspace. How about removing that meta-package from the workspace?

rosinstall currently gets its information form a .rosinstall file and does not rely on the DISTRO.yaml file. So there is no notion of a meta-package for rosinstall.

AFAIK, the script "http://packages.ros.org/cgi-bin/gen_rosinstall.py" generates rosinstall data from the distro.yaml, and has a "variant" parameter

Yes, that script is aware of rosdistro and variants, thus rosinstall is isolated from knowing about the distro/variants etc.  We could add a feature to rosinstall to do subtractive merging based on subsets of rosinstalls pass as arguments.  However, I expect a cleaner solution might be to 

Go on.

[tkruse]

There's no reason that a variant can't depend on another one.  That's how things work at the moment.  See the bottom of fuerte.rosdistro: https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro

But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.

And I hope it is obvious that maintaining the dependencies of ONE stack when package dependencies change was much less effort than maintaining the dependencies of  N variants to other variants.

[Lorenz Mösenlechner]

>
>
> And I hope it is obvious that maintaining the dependencies of ONE stack when
> package dependencies change was much less effort than maintaining the
> dependencies of N variants to other variants.
>

I don't really understand that argument. Variants are just collections
of packages. The dependencies of the variants are implicitly defined
by the dependencies of these packages. The user doesn't have to
specify them.

Lorenz

--
Lorenz Mösenlechner | moes...@in.tum.de
Technische Universität München | Karlstraße 45
80335 München | Germany
http://ias.cs.tum.edu/ | Tel: +49 (89) 289-26910

[tkruse]

As I stated, the problem is merely for human readers. It may be more useful to a reader to read that one variant depends on 5 other variant than for reader to read a variant depends on 50 packages. The information is the same, but the presentation is different. Usability is one of the reasons people switched to rosbuild. I am not saying we should or should not do this, I am just saying whichever way we go needs to go into the review, so that people know exactly what they'll be getting and why.

[Dirk Thomas]

On 09.08.2012 04:30, tkruse wrote:
>
> On Thursday, August 9, 2012 12:21:09 PM UTC+2, Tully Foote wrote:
>
> I've added some responses below:
>

> On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com <javascript:>> wrote:
>
>
> > Will the meta-packages be managed like variants, centralized?
>
>
> Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.
>
> To me that would be a CON:
> The concept of centralized meta packages depends on Willow Garage / OSRF infrastructure, which is less flexible than decentralized models, such as meta-package.xml defining a meta-package
> without the restrictions of a stack.xml.
>
>
> To update the rosdistro for your own meta-package it's just a pull request away to get it into the mainline one now. We've had great responses to updating the rosdeps that way. Also all the
> toolchain is now parameterized to allow you to fork the rosdistro and use your own version if you'd like, or extend it and add it to your sources.list listing. Another viewpoint is that part of
> the value of packaging is that you are making things common and facilitating a shared ontology.
>
>
> Does this also mean each corporate ROS community who develops non-open-source packages will have to create their own meta-packaging infrastructure and toolset, or will ROS provide tools to have a mix
> of company-internal distro.yaml mixed with the Willow Garage/OSRF one?

The distro.yaml file specified what packages and which versions define ROS.
To easy the installation of a set of packages it includes variants like pr2-desktop.

If someone else wants to maintain a project ROSx he has to maintain a distroX.yaml with similar information.

> I will also guess that meta-packages will not declare dependencies to other meta-packages, but instead only depend on something like the union of the package dependencies (not sure how union
> works with versioned dependencies). So where before a stack might depend on 5 other stacks, now a meta-package will depend on 50 packages. That's more difficult to proof-read.
>
>
> There's no reason that a variant can't depend on another one. That's how things work at the moment. See the bottom of fuerte.rosdistro:

> https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro <https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro>

>
>
> But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
> If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.

Maintainers only maintain package-to-package dependencies.
The information defining a variant include only a list of packages (or other variants) and are only for user convenience.

> > Will there be an 1-to-n or an n-to-m relationship between meta-packages and packages?
>
> The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.
>
>
> To me that means we lose meta-information.
> Consider the overview over stacks that a tool like the rosgui plugin Ros Package graph. The structural information is lost with the lack of stacks. Should this lack of structure be listed as a
> CON to the proposal? If meta-packages had an enforced 1-to-n restriction (each package must name one parent meta-package), the structure could be established. Also, again, by letting packages
> maintain their parent meta-package in the manifest, meta-packages can be established in a decentralized and automated fashion (with the caveat that typos may cause undesired meta-packages).
>
>
> Requiring one "primary" meta-package is restrictive. The packageA may be a primary component of the meta-packageA. But it may also be a primary component of the meta-packageB.
>
>
> You seem to mix up meta-package being primary to a package and a package being primary to a meta-packages.
> I am suggesting is that move_base should have a dedicated parent meta-package navigation, that does not exclude the possibility to define some other "uncle" meta-package containing move_base as well.
> However that relationship between move_base and navigation is stronger than that of other meta-packages.
> I am not sure myself how to shape this into a clean concept, I just see that dropping restrictions also drops helpful structure. And I still fail to see how steadily growing number of packages
> motivates having less structure to keep an overview.

The crucial point of the proposal is that there is nothing but packages.
Packages do NOT know about anything else than other packages they depend on.

Especially packages should NOT know that something like meta-packages exist since these are only syntactic sugar for the user.

> > Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?
>
> Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that
> package
> (without the other packages of that repository). But that should not be an issue.
>
>
> Does that mean if multiple packages live in the same SCM subtree (e.g. git repo), that they will still need a common CMakelists.txt in the subtree root to be catkin compliant?
>
>
> No they will not require anything connecting them. We will need to upgrade rosinstall/vcstools to be able to specify subdirectories to checkout. It's a less efficient way to use DVCSs from the
> storage aspect, but we expect to be more efficient from a speed and effort of development viewpoint.
>
>
> It's also new maintenance effort for vcstools / rosinstall, as well as burdening developers to understand the concept of checking out subdirectories of git or mercurial repos. What you gain by
> dropping rosstack you lose somewhere else.
>
> Just to make sure, the only reason we do that (not having additional folders in catkin workspaces which are just parents to catkin projects) is because it was decided for catkin that recursively
> crawling the ROS_PACKAGE_PATH was unacceptably slower / unacceptably more complex than crawling the ROS_PACKAGE_PATH without recursion?
>
> Meaning there was no actual hard technical constraint, just the opinion that it would be best to drop support for recursive crawls?

With the proposed direction of checking out packages in a workspace (which might end up in subfolders since multiple packages are stored in one DVCS) catkin nevertheless needs to search the subfolders
of a workspace for manifest.xml files.

- Dirk

[Dirk Thomas]

On 09.08.2012 05:20, tkruse wrote:
> As I stated, the problem is merely for human readers. It may be more useful to a reader to read that one variant depends on 5 other variant than for reader to read a variant depends on 50 packages.
> The information is the same, but the presentation is different. Usability is one of the reasons people switched to rosbuild. I am not saying we should or should not do this, I am just saying whichever
> way we go needs to go into the review, so that people know exactly what they'll be getting and why.

As stated multiple times: variants can depend on other variants (and of course packages).

So it would be possible to have something like that in the distro.yaml file:

gbp-repos:
- name: pckA
url: ...
version: ...
- name: pkgB
url: ...
version: ...
- AND MANY MORE PACKAGES

variants:
group1: [pkgA, pkgB]
group2: [pkgB, pkgC]
navigation: [group1, groupX, pkgY]
ros-base: [navigation, vision, pkgY, ...]
ros-full: [ros-base, ...]

IMHO that looks very readable.

- Dirk

[Jack O'Quin]

On Thu, Aug 9, 2012 at 5:21 AM, Tully Foote <tfo...@willowgarage.com> wrote:

> On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com> wrote:

>>> > Does that mean VCS repositories will have to be split up to have one
>>> > per package (all packages to github)?
>>>
>>> Absolutely not. With the large number of packages that is not feasible.
>>> If multiple packages are stored in the same repository it is a bit more
>>> difficult for a user to checkout only that package
>>> (without the other packages of that repository). But that should not be
>>> an issue.
>>
>>
>> Does that mean if multiple packages live in the same SCM subtree (e.g. git
>> repo), that they will still need a common CMakelists.txt in the subtree root
>> to be catkin compliant?
>
>
> No they will not require anything connecting them. We will need to upgrade
> rosinstall/vcstools to be able to specify subdirectories to checkout. It's
> a less efficient way to use DVCSs from the storage aspect, but we expect to
> be more efficient from a speed and effort of development viewpoint.

I don't understand how this is going to work. Suppose I have some GIT
repositories each containing one ROS stack with an average of ten
packages. Before, I checked out each stack using roslocate and rosws.
As I updated the code, I would push tested changes back to the GIT
origin.

Will I now need to check out 10 copies of each repository, so every
package appears at the top level in the working tree? If I change
several packages, will I need to commit and push separately for each
package affected? Then do I have to pull the collected changes into
all the other packages?

If this is easy to fix, why didn't we do it years ago when it became
apparent that a single GIT repository could not contain multiple ROS
stacks? That was already a problem.

If it is hard (as I suspect), why are we making it a requirement now?
Wouldn't it be better for rosws and roslocate to work with complete
repositories and the build tools to handle subtrees cleanly?

If repositories are the primary units of source checkout, maybe some
of these problems would go away.
--
joq

[Jack O'Quin]

On Thu, Aug 9, 2012 at 3:37 PM, Jack O'Quin <jack....@gmail.com> wrote:
> On Thu, Aug 9, 2012 at 5:21 AM, Tully Foote <tfo...@willowgarage.com> wrote:
>> On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com> wrote:

>> No they will not require anything connecting them. We will need to upgrade
>> rosinstall/vcstools to be able to specify subdirectories to checkout. It's
>> a less efficient way to use DVCSs from the storage aspect, but we expect to
>> be more efficient from a speed and effort of development viewpoint.
>
> I don't understand how this is going to work. Suppose I have some GIT
> repositories each containing one ROS stack with an average of ten
> packages. Before, I checked out each stack using roslocate and rosws.
> As I updated the code, I would push tested changes back to the GIT
> origin.
>
> Will I now need to check out 10 copies of each repository, so every
> package appears at the top level in the working tree? If I change
> several packages, will I need to commit and push separately for each
> package affected? Then do I have to pull the collected changes into
> all the other packages?

Also, if I release changes to several packages (but not all of them)
what kinds of GIT branches and tags will the release scripts create?
How does a user check out the exact set of sources that correspond to
the groovy distribution (for example)?

--
joq

[Dirk Thomas]

Since with DVCS the folder hierarchy is always preserved catkin will need to look for manifest.xml files recursively anyway (not only on the first level).

In the rosinstall file all packages for a workspace are listed.
If a package is in a subfolder of a DVCS the package information must include a "base_url" and a "subfolder".
Since it is not possible for DVCS to checkout each package on the root level, rosinstall should use one sparse checkout with all subfolders per repository.
Since a sparse checkout still clones the complete repository (and only hides some parts) it is only useful to reduce the packages build in the workspace.

> If this is easy to fix, why didn't we do it years ago when it became
> apparent that a single GIT repository could not contain multiple ROS
> stacks? That was already a problem.

GIT supports sparse checkout since version 1.7 - so that is not that long ago.

> If it is hard (as I suspect), why are we making it a requirement now?
> Wouldn't it be better for rosws and roslocate to work with complete
> repositories and the build tools to handle subtrees cleanly?

It is always possible to just checkout the complete repository.
The con would be that it builds all packages in the workspace (which might be more than we actually need).

In the previous discussion we were talking about multiple stacks per repository, which (without sparse checkouts) would mean pulling in more stacks into the workspace (which would have been even more
overhead).

- Dirk

[Jack O'Quin]

On Thu, Aug 9, 2012 at 6:30 AM, tkruse <tibo...@googlemail.com> wrote:

> On Thursday, August 9, 2012 12:21:09 PM UTC+2, Tully Foote wrote:

>> On Thu, Aug 9, 2012 at 2:00 AM, tkruse <tibo...@googlemail.com> wrote:
>>>
>>> > Will the meta-packages be managed like variants, centralized?
>>>>
>>>> Yes, variants are actually meta-packages themselves - both will be
>>>> stored in the DISTRO.yaml on GitHub.
>>>
>>> To me that would be a CON:
>>> The concept of centralized meta packages depends on Willow Garage / OSRF
>>> infrastructure, which is less flexible than decentralized models, such as
>>> meta-package.xml defining a meta-package without the restrictions of a
>>> stack.xml.
>>
>> To update the rosdistro for your own meta-package it's just a pull request
>> away to get it into the mainline one now. We've had great responses to
>> updating the rosdeps that way. Also all the toolchain is now parameterized
>> to allow you to fork the rosdistro and use your own version if you'd like,
>> or extend it and add it to your sources.list listing. Another viewpoint is
>> that part of the value of packaging is that you are making things common and
>> facilitating a shared ontology.

I can accept the github rosdeps solution, but everyone should
understand that this approach makes a time and convenience trade-off
in favor of Willow Garage while taxing the rest of the ROS community.

My first rosdeps update during Fuerte development took a couple of
days, because I had to register on github, learn how to clone the
distros repo, clone that on my development system, update my
/etc/ros/rosdep/sources.list.d, test my changes, push them back to my
clone, then learn enough about the quirks of github "pull" requests to
propagate my changes upstream. Every new ROS developer has some
learning curve with it. Some are already familiar with github, others
don't even know git with its own steep learning curve. This regularly
comes up on answers.ros.org.

Now that I've done it a few times the extra effort is tolerable. But,
I have also released a couple of broken stacks because I had an old
rosdep file lying around on my development machine. (Stupid, but an
easy mistake to make.)

Despite these problems, the github rosdeps scheme is OK. I don't have
any better solution to offer. But, similar trade-offs are not always
appropriate. We should not say: "rosdeps works great, so let's do
everything that way".

> Does this also mean each corporate ROS community who develops
> non-open-source packages will have to create their own meta-packaging
> infrastructure and toolset, or will ROS provide tools to have a mix of
> company-internal distro.yaml mixed with the Willow Garage/OSRF one?

A good question. We need people here who can represent the company and
industrial perspective. They are crucial for the long-term success of
ROS.

>> Requiring one "primary" meta-package is restrictive. The packageA may be
>> a primary component of the meta-packageA. But it may also be a primary
>> component of the meta-packageB.
>
> You seem to mix up meta-package being primary to a package and a package
> being primary to a meta-packages.
> I am suggesting is that move_base should have a dedicated parent
> meta-package navigation, that does not exclude the possibility to define
> some other "uncle" meta-package containing move_base as well. However that
> relationship between move_base and navigation is stronger than that of other
> meta-packages.
> I am not sure myself how to shape this into a clean concept, I just see that
> dropping restrictions also drops helpful structure. And I still fail to see
> how steadily growing number of packages motivates having less structure to
> keep an overview.

From a package maintainer's perspective "more flexibility" is not
always a good thing. It blows up the combinatorics of the testing
matrix, a fundamental problem of software development.

These "variant" names are significant. Since one can refer to another,
they must be maintained across releases with tick-tock to smooth out
any changes. There needs to be a mechanism for issuing deprecation
warnings.

>> Also I would argue that requiring a developer of a library to rerelease
>> their package to change it's meta-packaging is broken as the meta-packaging
>> is truely a packaging discussion, and should not bother the developer for a
>> patch release.
>
> I'd say the setup.py of a python package also contains packaging
> information, and changing that also means rereleasing. I think that's also
> similar to the pom.xml of maven. So it is not generally regarded broken to
> maintain packaging information in the source tree. The general
> meta-packaging concept may be maintained anywhere, but the special primary
> parent meta-package concept, like between move_base and navigation, could be
> maintained in the package source, in my opinion, without being broken.
>
> Also, even for general meta-packaging maintenance, it can be quite wise to
> maintain this in a more local VCS repository than the overall distro.yaml,
> even if that VCS does not include the source of the packages. So instead of
> having to do a pull request (comfortable as that may be), one coulde define
> a repository location which contains the meta-package information, just as
> one defines a repository location for packages. So not having the
> meta-packaging information in the source tree of a package does not mean we
> are left with the global distro.yaml as the only alternative.

+1; we already find it useful to release and maintain empty
meta-stacks, for tick-tock compatibility and to serve as persistent
documentation nodes (like camera_drivers).

Without stacks to organize the great mass of ROS packages, maintaining
meta-packages will become even more necessary.
--
joq

[tkruse]

On Thursday, August 9, 2012 6:48:29 PM UTC+2, Dirk Thomas wrote:

>     There's no reason that a variant can't depend on another one.  That's how things work at the moment.  See the bottom of fuerte.rosdistro:
>     https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro <https://code.ros.org/svn/release/trunk/distros/fuerte.rosdistro>
>
>
> But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
> If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.

Maintainers only maintain package-to-package dependencies.
The information defining a variant include only a list of packages (or other variants) and are only for user convenience.

And who maintains variant to variant dependencies, if not the maintainers? When a maintainer changes the dependencies of a package, who makes sure the dependencies of all variants that include this package stop including variants that were only included because of the previous dependencies of that package? Before, stack maintainer updated the stack dependencies. Now, there will be N variants in M different distro.yamls (if there are some in other corporations), which have to be updated. In what way would that reduce the effort of maintaining dual dependency trees?
 

>     Requiring one "primary" meta-package is restrictive. The  packageA may be a primary component of the  meta-packageA.  But it may also be a primary component of the meta-packageB.
>
> You seem to mix up meta-package being primary to a package and a package being primary to a meta-packages.
> I am suggesting is that move_base should have a dedicated parent meta-package navigation, that does not exclude the possibility to define some other "uncle" meta-package containing move_base as well.
> However that relationship between move_base and navigation is stronger than that of other meta-packages.
> I am not sure myself how to shape this into a clean concept, I just see that dropping restrictions also drops helpful structure. And I still fail to see how steadily growing number of packages
> motivates having less structure to keep an overview.

The crucial point of the proposal is that there is nothing but packages.
Packages do NOT know about anything else than other packages they depend on.

I know that, and I did not dispute that the proposal suggests that.
I am asking whether the lack of a special relationship between e.g. move_base and navigation should not be listed as a CON, a downside, given that the relationship between e.g. navigation and move_base has been helpful in the past. There might be plenty of advantages, I just like to also have the list of disadvantages on the records, and the justifications.

On 09.08.2012 05:20, tkruse wrote:

> As I stated, the problem is merely for human readers. It may be more useful to a reader to read that one variant depends on 5 other variant than for reader to read a variant depends on 50 packages.

> The information is the same, but the presentation is different. Usability is one of the reasons people switched to rosbuild. I am not saying we should or should not do this, I am just saying whichever

> way we go needs to go into the review, so that people know exactly what they'll be getting and why.

As stated multiple times: variants can depend on other variants (and of course packages).

So it would be possible to have something like that in the distro.yaml file:

[...]

IMHO that looks very readable. 

 
Sure it does.

If variants do NOT depend on variants (What Lorenz said):
- not readable
- no duplicate dependency trees

If variants DID depend on variants (What Tully said):
- readable
- duplicate dependency trees (In contrast to what Tully's PRO/CON list says)

So we cannot have the cake and eat it.

On Thursday, August 9, 2012 10:59:11 PM UTC+2, Dirk Thomas wrote:

 

It is always possible to just checkout the complete repository.

The con would be that it builds all packages in the workspace (which might be more than we actually need).

I believe cmake offers ways to exclude projects from make all, professional users can use those. I see no harm if make builds all projects for novice users, and not confuse them with sparse git checkouts and such. Also cmake/make offer build targets that are easy for novice users to understand (even have tab completion), for minimal builds.

Can we update the first catkin review to state that ctk_DD8 is not viable anymore which would also get rid of the limitation "Catkin requires flat workspace folder layout (some developers may not like the lack of structure)." ?

[Dirk Thomas]

On 09.08.2012 14:48, Jack O'Quin wrote:
> I can accept the github rosdeps solution, but everyone should
> understand that this approach makes a time and convenience trade-off
> in favor of Willow Garage while taxing the rest of the ROS community.
>
> My first rosdeps update during Fuerte development took a couple of
> days, because I had to register on github, learn how to clone the
> distros repo, clone that on my development system, update my
> /etc/ros/rosdep/sources.list.d, test my changes, push them back to my
> clone, then learn enough about the quirks of github "pull" requests to
> propagate my changes upstream. Every new ROS developer has some
> learning curve with it. Some are already familiar with github, others
> don't even know git with its own steep learning curve. This regularly
> comes up on answers.ros.org.
>
> Now that I've done it a few times the extra effort is tolerable. But,
> I have also released a couple of broken stacks because I had an old
> rosdep file lying around on my development machine. (Stupid, but an
> easy mistake to make.)

This overhead can easily go away.
Just imaging bloom offering you a question during the release process "Would you like to send a pull request to change package version to X.Y?".
The task to create the patch and the pull request can be automated with the GitHub API.
Only until now nobody has spent some hours to implement it...

> These "variant" names are significant. Since one can refer to another,
> they must be maintained across releases with tick-tock to smooth out
> any changes. There needs to be a mechanism for issuing deprecation
> warnings.

Meta-packages as described in the proposal are an "add-on" which is only used by the user to install a bunch of packages.

These meta-packages are not referenced by other packages as dependencies.
So meta-packages will not help when package names change.

Meta-packages are not part of the core system - so if they would not be there everything still works the same way.

- Dirk

[Jack O'Quin]

On Thu, Aug 9, 2012 at 5:15 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
> On 09.08.2012 14:48, Jack O'Quin wrote:
>> These "variant" names are significant. Since one can refer to another,
>> they must be maintained across releases with tick-tock to smooth out
>> any changes. There needs to be a mechanism for issuing deprecation
>> warnings.
>
> Meta-packages as described in the proposal are an "add-on" which is only
> used by the user to install a bunch of packages.

In the discussions earlier in this thread, they are much more than that.

> These meta-packages are not referenced by other packages as dependencies.
> So meta-packages will not help when package names change.

No one has been proposing changing package names. Package names are
the key to every ROS interface definition. They *cannot* be changed
without major breakage.

> Meta-packages are not part of the core system - so if they would not be
> there everything still works the same way.

It worries me when you say things like this. I don't know how to
interpret it, and do not want to assume bad faith.

Are you saying "catkin does not care about meta-packages, therefore
meta-packages are unimportant"?

Or, are you making a statement about our whole discussion concerning
the need for groups of packages and meta-packages needing to reference
other meta-packages? Do you think those issues are stupid and
unimportant?
--
joq

[Dirk Thomas]

We do have two opposing goals here (most of that is due to DVCS):

* Have only the content of one versioned Debian package per repository.

+ Natural tagging and branching
+ Straight forward checkout of specific version/branch
+ Straight forward committing

- Very high number of repositories (an order of magnitude more)
(Since we want "small" Debian packages with little dependencies)
- Moving all existing package code (except unary stacks) into new individual repositories is not realistic

* Have the content of multiple versioned Debian packages per repository.

+ Reduced number of repositories (similar as currently with rosbuild)
+ Possibility to commit to multiple packages from one working copy

- Tags and branches must be prefixed with package name
- Must be checked out multiple times when not all packages are on the same branch (i.e. master)
Committing to multiple packages at once is not possible (if not all packages are in a single working copy, aka coming from the same branch)

Both approaches have compromises, but it looks like these are limitations which come with DVCS.
We are currently writing down the next proposal describing how repositories can come into play.
It will have a clear goal which makes it natural to use GIT while still providing a decent upgrade path for existing stuff.
Please stay tuned...

- Dirk

[Dirk Thomas]

On 09.08.2012 16:07, Jack O'Quin wrote:
> On Thu, Aug 9, 2012 at 5:15 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
>> On 09.08.2012 14:48, Jack O'Quin wrote:
>>> These "variant" names are significant. Since one can refer to another,
>>> they must be maintained across releases with tick-tock to smooth out
>>> any changes. There needs to be a mechanism for issuing deprecation
>>> warnings.
>>
>> Meta-packages as described in the proposal are an "add-on" which is only
>> used by the user to install a bunch of packages.
>
> In the discussions earlier in this thread, they are much more than that.

I was referencing what is stated in the proposal posted by Tully:

"we will have �meta-packages� which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that
must agree with the package dependency."

I understand it the following way:
- No additional meta information is associated with them.
- Meta-packages are not referenced by anything else than other meta-packages.
- They are maintained in the distro.yaml file as a simple mapping from a meta-package name to a list of (meta-)packages.

>> Meta-packages are not part of the core system - so if they would not be
>> there everything still works the same way.
>
> It worries me when you say things like this. I don't know how to
> interpret it, and do not want to assume bad faith.
>
> Are you saying "catkin does not care about meta-packages, therefore
> meta-packages are unimportant"?

From my understanding of meta-packages (as summarized above) catkin does not need to know about meta-packages.
Catkin gets a folder of packages feed in, it can build them in correct order, provides infrastructure like find_package() etc., supports an install target and creates an environment which allow the
stuff to be run.
Catkin (as well as rosbuild) is unaware of a distro.yaml and does not require more information to do it's job.

I was referring to my understanding that it is not crucial to setup/build/install/run a bunch of ROS packages.
Everything which is necessary for these step I would name "core".
And based on my understanding what meta-packages will do / are used for, I would consider them "not core".

But "not core" has no negative meaning.
I.e. rosinstall is "not core" (according to my naming) - but it provide valuable additional functionality and eases the workflow.

- Dirk


PS: if I did not formulate my opinion well enough in the first place I am sorry for that.
I hope this more detailed explanation make is clear why I named things the way I did.
But please try to avoid allegation like in the last paragraph - they will very likely not bring the discussion forward (and I will not comment them).

[Jack O'Quin]

On Thu, Aug 9, 2012 at 6:44 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
> On 09.08.2012 16:07, Jack O'Quin wrote:
>> On Thu, Aug 9, 2012 at 5:15 PM, Dirk Thomas <dth...@willowgarage.com>
>> wrote:
>>> On 09.08.2012 14:48, Jack O'Quin wrote:
>>>>
>>>> These "variant" names are significant. Since one can refer to another,
>>>> they must be maintained across releases with tick-tock to smooth out
>>>> any changes. There needs to be a mechanism for issuing deprecation
>>>> warnings.
>>>
>>> Meta-packages as described in the proposal are an "add-on" which is only
>>> used by the user to install a bunch of packages.
>>
>> In the discussions earlier in this thread, they are much more than that.
>
> I was referencing what is stated in the proposal posted by Tully:
>>

> "we will have “meta-packages” which are groups of packages that can be

> installed by named aliases easily, but these will simply be content free
> packaging decisions, not source controlled content that must agree with the
> package dependency."

Thibault and I said there is more to meta-packages than that.

> I understand it the following way:
> - No additional meta information is associated with them.
> - Meta-packages are not referenced by anything else than other
> meta-packages.
> - They are maintained in the distro.yaml file as a simple mapping from a
> meta-package name to a list of (meta-)packages.

I think meta-packages probably should be explicitly represented via
files in various repositories. You may disagree. But, it would be
helpful if you respond to specific requirements, and not dismiss the
whole topic as a non-core discussion.

If you don't care about those issues because you have enough on your
plate dealing with just catkin, that's fine. But, please don't dismiss
the contributions of those of us who do care.

>>> Meta-packages are not part of the core system - so if they would not be
>>> there everything still works the same way.
>>
>> It worries me when you say things like this. I don't know how to
>> interpret it, and do not want to assume bad faith.
>>
>> Are you saying "catkin does not care about meta-packages, therefore
>> meta-packages are unimportant"?
>
> From my understanding of meta-packages (as summarized above) catkin does not
> need to know about meta-packages.
> Catkin gets a folder of packages feed in, it can build them in correct
> order, provides infrastructure like find_package() etc., supports an install
> target and creates an environment which allow the stuff to be run.
> Catkin (as well as rosbuild) is unaware of a distro.yaml and does not
> require more information to do it's job.
>
> I was referring to my understanding that it is not crucial to
> setup/build/install/run a bunch of ROS packages.
> Everything which is necessary for these step I would name "core".
> And based on my understanding what meta-packages will do / are used for, I
> would consider them "not core".

What about developing, releasing and maintaining ROS packages? Aren't
meta-packages important for that? Even if they are not "core"?

> But "not core" has no negative meaning.
> I.e. rosinstall is "not core" (according to my naming) - but it provide
> valuable additional functionality and eases the workflow.

OK. That helps.

Do you agree that many of the far-reaching effects of decisions being
made here are important, despite not being part of catkin?

> But please try to avoid allegation like in the last paragraph - they will
> very likely not bring the discussion forward (and I will not comment them).

I made no allegation; it was a question. I gave you an opportunity to
clarify your original statement, which sounded like a dismissal of the
whole meta-package discussion. I gave you the benefit of the doubt. I
am glad that is not what you meant.

Right now, I am feeling somewhat discouraged about the possibility of
being heard in this forum. If my comments are unwelcome and will be
ignored anyway, I should save everyone's time by working on other
tasks where I can more likely make a useful contribution.
--
joq

[Dirk Thomas]

On 09.08.2012 17:33, Jack O'Quin wrote:
> On Thu, Aug 9, 2012 at 6:44 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
>> On 09.08.2012 16:07, Jack O'Quin wrote:
>>> On Thu, Aug 9, 2012 at 5:15 PM, Dirk Thomas <dth...@willowgarage.com>
>>> wrote:
>>>>
>>>> Meta-packages as described in the proposal are an "add-on" which is only
>>>> used by the user to install a bunch of packages.
>>>
>>> In the discussions earlier in this thread, they are much more than that.
>>

>> I understand it the following way:
>> - No additional meta information is associated with them.
>> - Meta-packages are not referenced by anything else than other
>> meta-packages.
>> - They are maintained in the distro.yaml file as a simple mapping from a
>> meta-package name to a list of (meta-)packages.
>
> I think meta-packages probably should be explicitly represented via
> files in various repositories.
>

> What about developing, releasing and maintaining ROS packages? Aren't
> meta-packages important for that?

Could you outline your perspective of meta-packages?

What I have gather from the discussion so far is:
- store meta-package information as files in the repository
- provide more than just being empty meta-package on-top of the Debian packages

My open questions are:
- What are the information which describe a meta-package?
(Besides a meta-package name and a list of depending package names)

- How do meta-packages relate to packages?

- What should they be used for?
(Beside generating an empty Debian package which depends on the packages listed for the meta-package.)

- How do meta-packages relate to the context of developing, releasing and maintaining?

That will help to understand your perspective.

Dirk

[Tully Foote]

Gosh, I spend a day in meetings and I'm now way behind on this thread.  

Jack, we are definitely listening to all the feedback from you and the others on the mailing list.  We sincerely appreciate your time and input to help with the process.  Based on the feedback from this sig the concept of catkin has been completely revamped over the last few weeks.  We'll work on more visibility into the state of the system.  

With the deadline for the feature freeze coming up fast, we're working hard to implement most of what we're talking about as we talk about it. One thing that would be really helpful would be suggestions of what form of solutions you would like to see as opposed to just objections to the current suggestion.  Also as the freeze come up, we're also focusing on the core buildsystem which will need to be solidly frozen for others to release against.  We can continue to incrementally improve on the release and packaging systems after the primary freeze as it's not a dependency of others code.  

I have a few clarifications to questions raised an not answered below:

Does this also mean each corporate ROS community who develops non-open-source packages will have to create their own meta-packaging infrastructure and toolset, or will ROS provide tools to have a mix of company-internal distro.yaml mixed with the Willow Garage/OSRF one?

You can set up your own custom list of sources for rosdep.  This includes both rosdep.yaml files and rosdistro files.  Thus you can add your own files to the sources which extends the definitions.  This is designed explicitly for this sort of use case such that a 3rdparty can privately extend the released distro, or override it as they please, or even replace it if so desired. 

But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.

The current implementation I pointed to uses extends syntax such that you get the union of the packages.  It's just a grouping, and you can set a group to include the contents of another group.  And as the underlying packages contain all the dependency information, a variant does not need to worry about including all dependencies. So say we make a pr2_foo variant, it can just list pr2_x, pr2_y packages and pr2_bar meta-package. All the dependencies will be pulled in via the package dependencies of pr2_x, pr2_y, and the packages in pr2_bar if pr2_foo is installed.  A clever implementaiton will even be able to handle circular dependencies, as it can just be the union of the packages in codependent variants.  Thus it's not the same as maintaining a parallel dependency tree, you don't have to worry anything except listing the packages which provide the functionality desired. As Dirk just asked if you can clarify what other information you'd like contained in the meta-packages that would help us understand how we might integrate it.    

Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-buildsystem@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Daniel Stonier]

I agree, and it's something I've been waiting for over the last year or so. It will be a big step forward enabling us to maintain and upgrade our robots easily. I had originally started developing our own about a year ago but held off once Troy started working towards something with potential (aka catkin and bloom). It'd be great if we had tooling we could all use and then bring in all the repositories under one banner. Doing that part rosdep style with pull requests I'm quite ok with - moving rosdep to that structure was also a big step forward in centralising something that was creating quite a bit of chaos. And I figure you have to learn something with any new process - whether it's git/github or something else, it doesn't matter so much to me (I too have no better solution to offer).

I looked at the catkin closed source deb building process just as fuerte rolled out, it seemd ok. I've been too busy to look at bloom for building closed source debs though - probably been many changes since and sounds like many more to come. When the time is right for testing/input on closed source debs, let me know when and we'll hit it.

Daniel (Yujin).

 

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Phone : +82-10-5400-3296 (010-5400-3296)
Home: http://snorriheim.dnsdojo.com/

Yujin R&D: http://rnd.yujinrobot.com/

[tkruse]

Hi Tully,

On Friday, August 10, 2012 8:15:39 AM UTC+2, Tully Foote wrote:

With the deadline for the feature freeze coming up fast, we're working hard to implement most of what we're talking about as we talk about it.

I for one am mostly talking about what I would like to see for catkin in Hydro, not Groovy. I am no fan of large last minute changes for central pieces of software. And if catkin in the state of fuerte (minus problems Jack mentioned) was to be delivered for Groovy, I would not mind. During the early evolution of rosbuild, the ROS user community was small and consisted of experts mostly, so many changes in a small time were maybe painful (i wasn't there) but sufferable. Today that is different, and I'd rather see catkin (in the official releases) evolve at a slow pace that normal people can cope with, with new distro versions having been in their state for several months, not weeks, and several experts in several time zones to answer on answers.ros.org.

So for most of what I say about catkin, I would be perfectly happy for you to say: "OK, sounds relevant, we'll look into that for Hydro, for Groovy it is too late to make drastic changes". That's your call, IMO.
Catkin might even already define a hydro branch for implementing and testing such changes. Or a public Roadmap to clarify what will be dealt with when.

But that's just my view.

One thing that would be really helpful would be suggestions of what form of solutions you would like to see as opposed to just objections to the current suggestion.

Obviously a bug tracker issue with a patch is more helpful than one just describing a problem. But both have their place.

But that does away with the PRO argument: "maintainers no longer need to maintain the dual redundant dependency trees"
If variants are the new stacks, then maintaining variant interdependencies is a second dependency tree just as the stack dependency tree is.

The current implementation I pointed to uses extends syntax such that you get the union of the packages.  It's just a grouping, and you can set a group to include the contents of another group.  And as the underlying packages contain all the dependency information, a variant does not need to worry about including all dependencies. So say we make a pr2_foo variant, it can just list pr2_x, pr2_y packages and pr2_bar meta-package. All the dependencies will be pulled in via the package dependencies of pr2_x, pr2_y, and the packages in pr2_bar if pr2_foo is installed.  A clever implementaiton will even be able to handle circular dependencies, as it can just be the union of the packages in codependent variants.  Thus it's not the same as maintaining a parallel dependency tree, you don't have to worry anything except listing the packages which provide the functionality desired.

I am not sure whether you regard variant and meta-package as synonyms, or not. I understand technically they are similar in your proposal, but from what you say it could be that you still distinguish between them somehow.

In any case:
As soon as the pr2_foo variant mentions pr2_bar meta-package, you have a parallel dependency tree. Whatever reason motivated the maintainer of pr2_foo to include pr2_bar can change. Anytime anything changes in the dependencies of pr2_x, pr2_y packages and pr2_bar meta-package, the guy who decided to mention pr2_bar meta-package in pr2_foo variant will have to go back and see whether he still wants to have pr2_bar meta-package as a dependency of pr2_foo, or not.
As another example, maybe the creator of pr2_foo variant wanted to include pr2_barx, pr2_bary packages from meta_package pr2_bar. Now someone (anyone) decides to move pr2_barx out of pr2_bar for whatever reason. Now pr2_foo does no more include pr2_barx, which causes breakage and/or confusion.

You cannot have a any variant (or meta-package) mention anything else than a package without creating a second tree (forest) of variants / meta-packages that needs maintaining.

As Dirk just asked if you can clarify what other information you'd like contained in the meta-packages that would help us understand how we might integrate it.

The second catkin API review page ( http://ros.org/wiki/catkin/Reviews/2012-08-07_API_Review ) is the right place for that. It is still a bit in flux, obviously, as things get clarified.

[tkruse]

On Friday, August 10, 2012 11:26:10 AM UTC+2, tkruse wrote:

You cannot have a any variant (or meta-package) mention anything else than a package without creating a second tree (forest) of variants / meta-packages that needs maintaining.

Note though that I understand that there is a difference in the completely redundant dependency tree of stack systems, and the non-redundant system in a hierarchy where packages are part of the meta-package dependency tree.

[Geoff Biggs]

I don't see how this is a second tree (or forest) of dependencies. A
meta-package provides a set of functionality, interfaces, etc. How it
does that in reality is by causing a bunch of other packages to be
installed. If the maintainer of that meta-package removes one of its
dependencies (i.e. one of the packages that makes it up), then they're
changing the functionality it provides. This is no different to changing
the functionality provided by a "normal" package, and any developer who
does it unannounced is doing things wrong.


Geoff

[Andy Somerville]

I think the misunderstanding might be about what meta-package dependencies consist of. If I'm interpreting the confusion correctly several people might be thinking that there are dependencies between meta-packages, and as far as I understand this is not the case. 

A meta-package can only be the root of a dependency tree, never a branch. That is to say all other dependencies are handled/decided by checking the  dependencies of the constituent packages, and constituent packages never depend on meta-packages.

In this case installing a meta package is exactly equivalent to installing the constituent packages individually, (except that you would then need to uninstall them individually if later if desired)

     Andy

[Jack O'Quin]

On Fri, Aug 10, 2012 at 10:47 AM, Andy Somerville
<andy.so...@gmail.com> wrote:
>
> On Friday, August 10, 2012 11:33:13 AM UTC-4, Geoffrey Biggs wrote:
>>
>> On 10/08/12 18:26, tkruse wrote:
>> > You cannot have a any variant (or meta-package) mention anything else
>> > than a package without creating a second tree (forest) of variants /
>> > meta-packages that needs maintaining.
>>
>> I don't see how this is a second tree (or forest) of dependencies. A
>> meta-package provides a set of functionality, interfaces, etc. How it
>> does that in reality is by causing a bunch of other packages to be
>> installed. If the maintainer of that meta-package removes one of its
>> dependencies (i.e. one of the packages that makes it up), then they're
>> changing the functionality it provides. This is no different to changing
>> the functionality provided by a "normal" package, and any developer who
>> does it unannounced is doing things wrong.

That is why meta-packages need explicitly named maintainers, doc URLs,
bug reporting URLs, and a deprecation mechanism for implementing
changes via tick-tock.

Thibault is right. These things need to be maintained much the same
way as traditional ROS stacks. They will change over time. Human
intelligence (and good tools) will be needed to manage that.

This has advantages as well as the disadvantages. Assuming we beef
them up with the needed tools and documentation, meta-packages could
become a more powerful generalization of the traditional ROS stack:

* They can depend on other meta-packages, which was an annoying
limitation with stacks. I suspect variants would not have been
invented had stacks already been able to handle a forest structure.

* The dependency graph is provided directly by the packages, which
are leaf nodes in the tree (forest). No need to generate that
information by hand, an error-prone operation with traditional stacks.

> I think the misunderstanding might be about what meta-package dependencies
> consist of. If I'm interpreting the confusion correctly several people might
> be thinking that there are dependencies between meta-packages, and as far as
> I understand this is not the case.

Implicit in Thibault and Tully's discussion is the idea that
meta-packages *can* depend on other meta-packages. That is already the
case today with "variants". Tully's proposal was to merge those two
concepts.

That is really necessary for managing dependencies among hundreds or
thousands of packages.
--
joq

[Jack O'Quin]

os

Thanks for keeping an eye on that, Daniel.

I expect that industrial users (and OSRF) will need to replicate the
entire WG build farm, too. There have already been requests for and
discussion of that requirement.
--
joq

[Andy Somerville]

Parsing more closely, I should recant and clarify. 

I should have said that meta-packages don't depend on other meta-packages for the sake of satisfying the dependencies of the packages below them.

That  is to say that if meta-package "Foo" includes a package "foo_node" that then depends on "laser_geometry" which lives in the "laser_pipeline" meta-package, "Foo" would not include the meta-package "laser_pipeline" as a dependency, but rather ignore it, and allow "foo_node"'s dependency tree to be sorted out automatically.

    Andy

[tkruse]

On Friday, August 10, 2012 7:05:20 PM UTC+2, Andy Somerville wrote:

Parsing more closely, I should recant and clarify. 

I should have said that meta-packages don't depend on other meta-packages for the sake of satisfying the dependencies of the packages below them.

That  is to say that if meta-package "Foo" includes a package "foo_node" that then depends on "laser_geometry" which lives in the "laser_pipeline" meta-package, "Foo" would not include the meta-package "laser_pipeline" as a dependency, but rather ignore it, and allow "foo_node"'s dependency tree to be sorted out automatically.

I admit I got that wrong the first time.

On Friday, August 10, 2012 6:22:08 PM UTC+2, Jack O'Quin wrote:

That is why meta-packages need explicitly named maintainers, doc URLs,

bug reporting URLs, and a deprecation mechanism for implementing

changes via tick-tock.

I'd say meta-packages need descriptions beyond descriptive names.
Maintainers, bug trackers and deprecation can be handled on a package
level, whoever deals with a pull requests will need to check with package
maintainers whether a change to a meta-package is okay.

Some current stacks have wiki pages for themselves, the contents of which
would move into non-stack content.
If all packages of a current stack share a bugtracker, that will go into each of
the packages manifest.xml, which is not nice (because its redundant), but bearable.

So other than a description, I do not see that much of a need to store data on
meta-packages, I still think that packages and meta-packages are not enough
to provide a good overview of what lives in the ROS ecosystem.

Maintaining meta-packages instead of stack will still be effort, and some kind of
process / best practice / naming conventions could be defined to keep the
number of meta-packages low, and keep creativity at bay.
Currently variants were defined in a REP for each distro, I'll guess the same
won't work for meta-packages, but still it might be wise not to allow too many
different naming philosphies into the meta-package namespace.

In a way I think it can be good to start out with minimalist concepts for now
and expand later with concepts when we feel the pains that come with just
having packages and meta-packages, since it seems to be up to opinions
or tastes whether the benefit of more structure is worth the effort or not.
That's better than introducing concepts now that have to be changed later.

[tkruse]

Is there any reason speaking against constraining the proposal as follows?

"For each migrated rosbuild stack, there will be one meta-package having exactly the same name as the stack, meta-package names that exists as stacks in rosbuild world are thus exclusively reserved for those stacks."

This as opposed to people thinking they always wanted to have a cooler name for their stack, and now is the time to switch.

It might be implicit in what Tully though of.

[Tully Foote]

Thanks for the clarifications. How about we do the following:

 - We keep the variants in the rosdistro file.  (Noting you can add your own rosdistro file to extend privately.) 

 - But we will add two data fields: description and url (both optional, description recommended)

 - We will update the PackageHeader wiki macro to list all variants in which that the package is contained.

 - We will create a VariantHeader wiki macro which lists all the packages in the variant, and what it extends.  This will allow us to use the wiki page to store all 

 - For migration purposes, we would create a variant for each stack.  (probably a bit of manual work, but a one time cost. )  

Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/r2PRdxErDEwJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Geoff Biggs]

I 100% agree with everything you said. A meta-package should be treated
as a package. It just happens to provide its functionality purely
through dependencies.

Gentoo has been using meta-packages (they're called "sets" in Portage 2,
now) for years quite successfully. All they are is a package with no
source of its own and a bunch of dependencies. The rest, including
things like maintainer, homepage, bug tracker, and so on all flows
naturally from the package manager tools. This includes meta-packages
depending on other meta-packages. The only issue I've ever had with them
is that Portage 1 did not have good support for uninstalling the
dependencies along with the meta-package.

Geoff

[Jack O'Quin]

On Fri, Aug 10, 2012 at 4:05 PM, tkruse <tibo...@googlemail.com> wrote:
> On Friday, August 10, 2012 6:22:08 PM UTC+2, Jack O'Quin wrote:
>>
>> That is why meta-packages need explicitly named maintainers, doc URLs,
>> bug reporting URLs, and a deprecation mechanism for implementing
>> changes via tick-tock.
>
> I'd say meta-packages need descriptions beyond descriptive names.
> Maintainers, bug trackers and deprecation can be handled on a package
> level, whoever deals with a pull requests will need to check with package
> maintainers whether a change to a meta-package is okay.

With an M-to-N relationship between packages and meta-packages, I
don't see how that can solve the problem.

> Some current stacks have wiki pages for themselves, the contents of which
> would move into non-stack content.
> If all packages of a current stack share a bugtracker, that will go into
> each of
> the packages manifest.xml, which is not nice (because its redundant), but
> bearable.

Meta-packages should also have wiki pages (or some other doc URL).

> So other than a description, I do not see that much of a need to store data
> on
> meta-packages, I still think that packages and meta-packages are not enough
> to provide a good overview of what lives in the ROS ecosystem.

I was arriving at the opposite conclusion. With proper tools and
documentation, I think meta-packages might fill the hole left by
eliminating stacks. We need a way to mitigate this whole nightmare of
thousands of packages with no structure and no way for anyone to
understand what is available for code reuse. That was supposed to be
the main purpose of ROS.

I really hate the idea of maintaining meta-packages in a central
location with minimal information associated. I was working on a
proposal for a substantial, maintainable, distributed meta-package
definition in response to Dirk's question from yesterday.

> Maintaining meta-packages instead of stack will still be effort, and some
> kind of
> process / best practice / naming conventions could be defined to keep the
> number of meta-packages low, and keep creativity at bay.
> Currently variants were defined in a REP for each distro, I'll guess the
> same
> won't work for meta-packages, but still it might be wise not to allow too
> many
> different naming philosphies into the meta-package namespace.

You are right about the dangers of namespace pollution. I had
forgotten about that.

But, we have that problem with packages already. A few stacks or
meta-packages don't fundamentally affect it. What users really need is
a way to easily distinguish *supported* packages from one-off lab
experiments that worked one semester on a single machine never to be
touched again.

I would like to come up with a better solution to the namespace issue.
Users should be able to distinguish *released* stacks from "wild
code".

> In a way I think it can be good to start out with minimalist concepts for
> now
> and expand later with concepts when we feel the pains that come with just
> having packages and meta-packages, since it seems to be up to opinions
> or tastes whether the benefit of more structure is worth the effort or not.
> That's better than introducing concepts now that have to be changed later.

I don't think what we have right now is good enough. We have lost
needed function without adequate substitutes.
--
joq

[tkruse]

On Saturday, August 11, 2012 2:25:51 AM UTC+2, Jack O'Quin wrote:

Meta-packages should also have wiki pages (or some other doc URL).

According to Tully suggestion, they'll have wiki pages with names according to their names. Meaning an implicit doc URL. That's less effort to maintain than explicit URLs.
 

> So other than a description, I do not see that much of a need to store data
> on
> meta-packages, I still think that packages and meta-packages are not enough
> to provide a good overview of what lives in the ROS ecosystem.

I was arriving at the opposite conclusion. With proper tools and
documentation, I think meta-packages might fill the hole left by
eliminating stacks. We need a way to mitigate this whole nightmare of
thousands of packages with no structure and no way for anyone to
understand what is available for code reuse. That was supposed to be
the main purpose of ROS.

That's not the current purpose of meta-packages, but something else can still be introduced later.I would like to come up with a better solution to the namespace issue.

Users should be able to distinguish *released* stacks from "wild
code".

That's easy. With ROS everything is still wild.
 

> In a way I think it can be good to start out with minimalist concepts for
> now
> and expand later with concepts when we feel the pains that come with just
> having packages and meta-packages, since it seems to be up to opinions
> or tastes whether the benefit of more structure is worth the effort or not.
> That's better than introducing concepts now that have to be changed later.

I don't think what we have right now is good enough. We have lost
needed function without adequate substitutes.

My thinking is that so far, nobody cares about catkin. So there are not that many
catkin packages. So we can have an iteration of catkin just with meta-packages for now,
and have some more structure in Hydro. So with the current proposal we mostly lose
something conceptually, but since most code remains in rosbuild stacks, practically
there is no loss for now.

[Jack O'Quin]

On Sat, Aug 11, 2012 at 2:31 AM, tkruse <tibo...@googlemail.com> wrote:
>
> On Saturday, August 11, 2012 2:25:51 AM UTC+2, Jack O'Quin wrote:
>>
>> Meta-packages should also have wiki pages (or some other doc URL).
>
> According to Tully suggestion, they'll have wiki pages with names according
> to their names. Meaning an implicit doc URL. That's less effort to maintain
> than explicit URLs.

All that does is create lots of broken links to wiki pages.

Plus, the design should not force all documentation to ros.org. If
people are willing to write and maintain documentation, let them put
it wherever they like. Github may be a reasonable choice for many.

>> > So other than a description, I do not see that much of a need to store
>> > data
>> > on
>> > meta-packages, I still think that packages and meta-packages are not
>> > enough
>> > to provide a good overview of what lives in the ROS ecosystem.
>>
>> I was arriving at the opposite conclusion. With proper tools and
>> documentation, I think meta-packages might fill the hole left by
>> eliminating stacks. We need a way to mitigate this whole nightmare of
>> thousands of packages with no structure and no way for anyone to
>> understand what is available for code reuse. That was supposed to be
>> the main purpose of ROS.
>
> That's not the current purpose of meta-packages, but something else can
> still be introduced later.I would like to come up with a better solution to
> the namespace issue.

I believe we can invent a reasonable solution for the namespace issue.

>> Users should be able to distinguish *released* stacks from "wild
>> code".
>
> That's easy. With ROS everything is still wild.

Ouch!

I share your discouragement, but, there *is* some useful code being
supported outside Willow Garage. Maybe the ratio of wild to reusable
is as low as 90:10, but some of us are making good-faith efforts to
release useful stacks and keep them working as ROS evolves. (I would
love to see some actual statistics on package reuse.)

In my view, we should focus on making it easier for external
developers to release reusable code in ROS. That is important for the
long-term success of ROS and its community, and not just because it
would help me (although it would).

>> > In a way I think it can be good to start out with minimalist concepts
>> > for
>> > now
>> > and expand later with concepts when we feel the pains that come with
>> > just
>> > having packages and meta-packages, since it seems to be up to opinions
>> > or tastes whether the benefit of more structure is worth the effort or
>> > not.
>> > That's better than introducing concepts now that have to be changed
>> > later.
>>
>> I don't think what we have right now is good enough. We have lost
>> needed function without adequate substitutes.
>
> My thinking is that so far, nobody cares about catkin. So there are not that
> many catkin packages. So we can have an iteration of catkin just with
> meta-packages for now,
> and have some more structure in Hydro. So with the current proposal we
> mostly lose
> something conceptually, but since most code remains in rosbuild stacks,
> practically there is no loss for now.

Certainly, the current proposal will slow conversion of packages to
catkin. As you say, it was going to be slow, anyway. Most outside
developers have little need to make the change.

The major exception is for people like Daniel who require good
cross-compile tools. For that reason, he is already an early adopter
of the new build methodology.

But, saying the quality of catkin in Groovy does not matter because
few will use it is self-defeating. We need a better build system.
Catkin provides several significant improvements. If we work together,
we can devise a very good solution that outside groups will want to
adopt. It will still be slow, but at least people will feel better
about spending time on it (time that could otherwise be devoted to
robotics tasks).

I would like to see more thought given to the mechanics of the
migration process. When I imagine converting some of my larger stacks,
I quickly run up against the fact that wet and dry packages cannot be
built in the same workspace. That makes it awkward to convert a large
group of packages one at a time, which is the way I would choose to do
it.

But, the worst problem is that since Fuerte, it has gotten harder to
release and support "dry" stacks. When roscreate-stack went away, the
best method of reconciling package and stack dependencies was lost.
Several times I found myself going back to Electric so I could use the
old version to (approximately) update some stack dependencies. That is
not a good solution and does not work reliably, since some packages
moved to different stacks in Fuerte.

The rosdep and rosrelease changes also made things more error-prone
(for me, at least). Trouble-shooting those errors in the build farm
took far too much time for me and for several kind people who assisted
me.

Because the core developers are naturally focused on "wet" tools, this
kind of deterioration is likely to continue with Groovy. We find
ourselves in the dangerous position of having "one foot in the boat
and one foot on the dock". I already hear anecdotal evidence of people
who tried Fuerte, had some problems, and went back to Electric. (I
would love to see some good statistics on how many people are running
the various distributions.)
--
joq

[tkruse]

On Saturday, August 11, 2012 5:50:10 PM UTC+2, Jack O'Quin wrote:

On Sat, Aug 11, 2012 at 2:31 AM, tkruse <tibo...@googlemail.com> wrote:
>
> On Saturday, August 11, 2012 2:25:51 AM UTC+2, Jack O'Quin wrote:
>>
>> Meta-packages should also have wiki pages (or some other doc URL).
>
> According to Tully suggestion, they'll have wiki pages with names according
> to their names. Meaning an implicit doc URL. That's less effort to maintain
> than explicit URLs.

All that does is create lots of broken links to wiki pages.

Remember I suggested to keep meta-package names exactly as rosbuild stack names.
 

Plus, the design should not force all documentation to ros.org.

That can be solved later by adding non-default URLs. That's not urgent now.

But, saying the quality of catkin in Groovy does not matter because
few will use it is self-defeating.

I just said the quality of structured INDEXING does not matter in Groovy, as the
number of Groovy catkin packages is so low.

To get higher adoption, I would love to see changes to catkin, mostly revert to
rosbuild like separation of builds (maybe with an introduction of dedicated build groups),
but also "moist" workspaces where both rosbuild and catkin packages are build in the
right order with a single "rosmake" command.

To me, that's much more important now than structuring a couple of catkin packages.
All further structuring and meta-indexing can still be added to the toolchain later, I think.

The other issues you mentioned about rosbuild, roscreate, rosdep, rosrelease all
also sound more important and more urgent than meta-information for meta-packages.
So i think the point of necessary meta-information was made, we can let it sink in for now
and focus on the most urgent issues.

[Daniel Stonier]

On 12 August 2012 01:50, Jack O'Quin <jack....@gmail.com> wrote:

On Sat, Aug 11, 2012 at 2:31 AM, tkruse <tibo...@googlemail.com> wrote:
>
> On Saturday, August 11, 2012 2:25:51 AM UTC+2, Jack O'Quin wrote:
>>
>> Meta-packages should also have wiki pages (or some other doc URL).
>
> According to Tully suggestion, they'll have wiki pages with names according
> to their names. Meaning an implicit doc URL. That's less effort to maintain
> than explicit URLs.

All that does is create lots of broken links to wiki pages.

Plus, the design should not force all documentation to ros.org. If
people are willing to write and maintain documentation, let them put
it wherever they like. Github may be a reasonable choice for many.

In the last few months we've seen user documentation for a few core tools become a little varied (I'm thinking especially of the python build tools here - rospkg, rosinstall, catkin) and they're a bit harder to follow when each follows their own style. Not to mention some of the information gets duplicated. 

It's not a big thing, and logically decentralising user documentation is really no different to how the wiki already handles decentralised repos and issue trackers seems fine, but I do think it is sometimes a bit awkward.

I agree. Although I would say that ros has to some extent, handled this already. The variants have always been (for me at least) a good way of quickly assimilating what is informally 'non-wild'. The information could have been better highlighted though - e.g. note this from the REP:

"There are currently no plans to use variants in the documentation (rosdoc, wiki) toolchain. They could be used for categorization, but this is speculative."

So, back to meta-packages. I can see these replacing the notion of stacks, even the organisation of several stacks or robot specific variants. I don't see them effectively being used to highlight the quality, even at the simplest level of 'official' vs 'wild'. Discriminating by name between a wild meta package and an official one isn't conclusive enough. The quality of code should be represented at the package level anyway and easily sortable at levels above. A meta-package could be thought of as 'official' if all its packages are.

And there are mechanisms in ros already that can help highlight how stable a package is - the version number, the review status come to mind. But these have always been very informal and not often used. I think of ubuntu packages with its categories - main, universe, ... but who would officiate this process in the ros community? Up until now it's been somewhat taken care of by Willow via their organisation of variants.

Actually I think it's amazing that they released catkin with Fuerte with as much compatibility with the legacy system as they did. That coupled with the fact that Willow has seen a few of their core team members leave over the last few months and that ros is starting to show signs that its struggling to scale even further than it already has...

My take on it is that fuerte and groovy will inevitably have some disruption and so I'm not planning anything critical on these releases. Unavoidable - the ros wiki is struggling, the manpower isn't there, the build environment and release system has some big constraints, ros comms is struggling to move outside the robot and the core developers are once again new. Some fundamental building blocks need to be revisited so it can grow in the long term. I'm somewhat on the inside, so that's obvious to me, but perhaps that needs to be communicated more clearly.

If the build environment, release system, wiki and to a lesser extent, ros comms can be attacked vigorously over the next release or two (as fast as possible), then I think RoS can really start to be a good foundation for official long term releases, cross platform development and non pr2 style robotics. In particular, it would become really attractive to us as a company. 

Daniel.

 

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

[Jack O'Quin]

On Sat, Aug 11, 2012 at 12:30 PM, tkruse <tibo...@googlemail.com> wrote:
>
> On Saturday, August 11, 2012 5:50:10 PM UTC+2, Jack O'Quin wrote:
>>
>> On Sat, Aug 11, 2012 at 2:31 AM, tkruse <tibo...@googlemail.com> wrote:
>> >
>> > On Saturday, August 11, 2012 2:25:51 AM UTC+2, Jack O'Quin wrote:
>> >>
>> >> Meta-packages should also have wiki pages (or some other doc URL).
>> >
>> > According to Tully suggestion, they'll have wiki pages with names
>> > according
>> > to their names. Meaning an implicit doc URL. That's less effort to
>> > maintain
>> > than explicit URLs.
>>
>> All that does is create lots of broken links to wiki pages.
>
> Remember I suggested to keep meta-package names exactly as rosbuild stack
> names.

Sure.

Will that be done in Groovy for all dry stacks, or just the wet ones?
Will the indexer no longer include dry stacks from then on? If I want
to create a new dry stack, will I have to ask Willow Garage to accept
a distros.yaml pull request?

I really, really dislike using the rosdistro variant mechanism for
this. It was barely adequate before, but only because there were very
few variants and they seldom changed.

>> But, saying the quality of catkin in Groovy does not matter because
>> few will use it is self-defeating.
>
> I just said the quality of structured INDEXING does not matter in Groovy, as
> the number of Groovy catkin packages is so low.

I misunderstood, and probably read more into your statement than was intended.

> To get higher adoption, I would love to see changes to catkin, mostly revert
> to rosbuild like separation of builds (maybe with an introduction of dedicated
> build groups),
> but also "moist" workspaces where both rosbuild and catkin packages are
> build in the
> right order with a single "rosmake" command.

But, who will implement that?

> All further structuring and meta-indexing can still be added to the
> toolchain later, I think.

Maybe, but I am not convinced of it yet.

> The other issues you mentioned about rosbuild, roscreate, rosdep, rosrelease
> all also sound more important and more urgent than meta-information for
> meta-packages.

I agree, but I don't seriously expect the core developers to spend
much time on it.
--
joq

[tkruse]

On Saturday, August 11, 2012 9:34:51 PM UTC+2, Jack O'Quin wrote:

> To get higher adoption, I would love to see changes to catkin, mostly revert
> to rosbuild like separation of builds (maybe with an introduction of dedicated
> build groups),
> but also "moist" workspaces where both rosbuild and catkin packages are
> build in the
> right order with a single "rosmake" command.

But, who will implement that?

 

That would be for the ROS platform manager to decide, unless a renegade team did that outside Willow Garage.

Tully said that the catkin team is listening to the community and in the process of changing things. So I hope that at some point over the next weeks, we will get a more detailed plan of what is planned to change when (and what wont). Also I would hope that maybe during next week, there can be a post on ros-users with the results of the posts written in the SIG so far, the final(?) API reviews, or something similar.

[Jack O'Quin]

> On Sat, Aug 11, 2012 at 12:30 PM, tkruse <tibo...@googlemail.com> wrote:
>> Remember I suggested to keep meta-package names exactly as rosbuild stack
>> names.

On Sat, Aug 11, 2012 at 2:34 PM, Jack O'Quin <jack....@gmail.com> wrote:
> Will that be done in Groovy for all dry stacks, or just the wet ones?
> Will the indexer no longer include dry stacks from then on? If I want
> to create a new dry stack, will I have to ask Willow Garage to accept
> a distros.yaml pull request?

Does anyone know the answers to these questions?
--
joq

[tkruse]

Not that I would know, but I have no doubt that Tully will see to this being
handled in a reasonable fashion. I did not see any indication at all of
anything changing for dry stacks in Groovy, did you?

[Tully Foote]

For existing dry stacks nothing will change in groovy.  All of these discussions are only for things being converted to wet.  (Currently 55 of 282 released stacks are running on the prototype of catkin in fuerte. ) We will continue to support the dry toolchain for indexing and source based installs for several cycles.  The length of time depends on how quickly the community adapts.  Due to the significant extra burden of the dual build system when operating the debbuild infrastructure the binary built releases may drop support for dry as soon as Hydro, but no sooner.  

Tully

[Tully Foote]

On Sun, Aug 12, 2012 at 5:56 PM, Tully Foote <tfo...@willowgarage.com> wrote:

On Sun, Aug 12, 2012 at 3:37 PM, tkruse <tibo...@googlemail.com> wrote:

On Sunday, August 12, 2012 9:35:43 PM UTC+2, Jack O'Quin wrote:

> On Sat, Aug 11, 2012 at 12:30 PM, tkruse <tibo...@googlemail.com> wrote:
>> Remember I suggested to keep meta-package names exactly as rosbuild stack
>> names.

On Sat, Aug 11, 2012 at 2:34 PM, Jack O'Quin <jack....@gmail.com> wrote:
> Will that be done in Groovy for all dry stacks, or just the wet ones?
> Will the indexer no longer include dry stacks from then on? If I want
> to create a new dry stack, will I have to ask Willow Garage to accept
> a distros.yaml pull request?

I realized I didn't completely answer this in my last post.  For clarity this does not only apply to existing, but potentially new dry stacks.  You can still add a new stack to the rosdistro file in the release repository as before.  

Tully 

[Jack O'Quin]

I'd like to see a description, a maintainer, a review status, doc URL,
a Trac URL. Some of those could be optional. An optional deprecation
status would be very helpful for tick-tock. People are free to create
meta-packages without those data. Then, I will know to avoid them.

> - How do meta-packages relate to packages?

They are sets containing packages and other meta-packages.

The set membership graph probably needs to be acyclic to avoid
Russell's paradox (involving sets that are subsets of themselves). I
am not quite sure about that, but it seems likely. Plus, acyclic
graphs are easier to handle.

> - What should they be used for?
> (Beside generating an empty Debian package which depends on the packages
> listed for the meta-package.)

> - How do meta-packages relate to the context of developing, releasing and
> maintaining?

When depending on another meta-package (i.e. subset), I would want to
know if it is supported (maintained, reviewed, etc.). If not, I
probably don't want to depend on it.

When depending on collections of packages created by other developers,
I don't want to depend on unnecessary detail about the internal
structure of those packages. When checking out their source code, I
don't want to list every single package, just the name of the
collection (e.g. navigation, common_msgs).

If I do depend on another meta-package and my system breaks in some
future release due to changes, I know who to contact for help in
resolving that issue. For maintained meta-packages (the only ones I
care about), there needs to be an explicit tick-tock mechanism, so
dependent meta-packages can migrate to a new version over time.

This was already a problem with variants, but there are very few
variants, and for most the maintainers are known from context and
likely to provide relatively stable names. I would prefer variants to
become meta-packages with similar information, rather than the
opposite.

When releasing a collection of packages, I want the release scripts to
permit releasing all the packages in a meta-package at once,
regardless of whether they come from a DVCS repository.

Similarly, I want the pre-release tests to accept meta-packages, so I
don't have to list by hand a dozen packages when preparing to release
them.

I am basically discussing semantics here. The best syntax for
representing it is not clear, and probably mostly depends on the needs
and convenience of developing the support scripts and other tools.

Possibly, most of these semantics could be provided by creating an
empty package in the repository with a manifest but no code, only
dependencies. I am not sure, and have no idea how well that would
integrate with CMake and catkin.
--
joq

[Dirk Thomas]

On 13.08.2012 08:46, Jack O'Quin wrote:
>
> When depending on collections of packages created by other developers,
> I don't want to depend on unnecessary detail about the internal
> structure of those packages. When checking out their source code, I
> don't want to list every single package, just the name of the
> collection (e.g. navigation, common_msgs).

So rosinstall (or something similar) should know how to checkout all dependencies of a meta-package and be able to checkout all dependent packages for building from source.

Should that include only first-level dependencies explicitly listed in the meta-package?
What about transitive dependencies?

> If I do depend on another meta-package and my system breaks in some
> future release due to changes, I know who to contact for help in
> resolving that issue. For maintained meta-packages (the only ones I
> care about), there needs to be an explicit tick-tock mechanism, so
> dependent meta-packages can migrate to a new version over time.

You want to depend on a meta-package from another package?
How should the CMake inside this package look like?
Would you still find_package() all the individual packages the meta-package depends on?
Or would you like go for find_package(meta-package-name)?

> When releasing a collection of packages, I want the release scripts to
> permit releasing all the packages in a meta-package at once,
> regardless of whether they come from a DVCS repository.

Should all packages come from a single repository (independent of the type) or could they be stored in multiple different repositories?

> Similarly, I want the pre-release tests to accept meta-packages, so I
> don't have to list by hand a dozen packages when preparing to release
> them.

What should a pre-release of a meta-package do?
Current behavior would be install all dependencies from Debian packages and build all other stuff which depends on it.
I guess you would like to build all packages which the meta-package depends on from source?

- Dirk

[Jack O'Quin]

On Mon, Aug 13, 2012 at 11:37 AM, Dirk Thomas <dth...@willowgarage.com> wrote:
> On 13.08.2012 08:46, Jack O'Quin wrote:
>>
>>
>> When depending on collections of packages created by other developers,
>> I don't want to depend on unnecessary detail about the internal
>> structure of those packages. When checking out their source code, I
>> don't want to list every single package, just the name of the
>> collection (e.g. navigation, common_msgs).
>
>
> So rosinstall (or something similar) should know how to checkout all
> dependencies of a meta-package and be able to checkout all dependent
> packages for building from source.
>
> Should that include only first-level dependencies explicitly listed in the
> meta-package?
> What about transitive dependencies?

I think that requirement mostly applies to only direct package dependencies.

My guess is that recursive dependencies would generally not be desired
for rosws, although they would probably be useful as an option, if
that's not too difficult. Some of this is hard to decide without
experimenting with the new tools. This is all so unfamiliar, the
concepts are tricky to map backwards to normal work-flows.

>> If I do depend on another meta-package and my system breaks in some
>> future release due to changes, I know who to contact for help in
>> resolving that issue. For maintained meta-packages (the only ones I
>> care about), there needs to be an explicit tick-tock mechanism, so
>> dependent meta-packages can migrate to a new version over time.
>
>
> You want to depend on a meta-package from another package?

No, I did not say that. Only from meta-packages (sets containing subsets, etc.).

> How should the CMake inside this package look like?
> Would you still find_package() all the individual packages the meta-package
> depends on?
> Or would you like go for find_package(meta-package-name)?

I was not asking for that. What do you have in mind?

>> When releasing a collection of packages, I want the release scripts to
>> permit releasing all the packages in a meta-package at once,
>> regardless of whether they come from a DVCS repository.
>
> Should all packages come from a single repository (independent of the type)
> or could they be stored in multiple different repositories?

Multiple repositories would be useful. William already mentioned that.
But, I don't see it as the highest priority.

Your question about recursion and transitive dependencies also applies
here. In this case, users should be able to make appropriate choices.

>> Similarly, I want the pre-release tests to accept meta-packages, so I
>> don't have to list by hand a dozen packages when preparing to release
>> them.
>
> What should a pre-release of a meta-package do?

The goal is to establish that the code about to be released will not
break the build.

> Current behavior would be install all dependencies from Debian packages and
> build all other stuff which depends on it.
> I guess you would like to build all packages which the meta-package depends
> on from source?

I don't know enough about the implementation details to recommend an
exact procedure.

I think the current pre-release test may also run the self-test for
dependent packages. That is clearly useful in many cases.

My experience with Fuerte suggests that those tests are not working as
well as they used to for dry stacks.
--
joq

[Jonathan Bohren]

On Mon, Aug 13, 2012 at 5:46 PM, Jack O'Quin <jack....@gmail.com> wrote:

I'd like to see a description, a maintainer, a review status, doc URL,
a Trac URL. Some of those could be optional. An optional deprecation
status would be very helpful for tick-tock. People are free to create
meta-packages without those data. Then, I will know to avoid them.

Another idea. Since we're tending towards a more distributed community, howabout a tag that  indicates that this package should be indexed or hidden from indexers? Something that has the same effect as a robots.txt file. It could even be an independent file that can be easily found in searches.

I've been looking into scraping the popular source code hosts (bitbucket, github, etc) to discover previously unlisted ROS packages, but if someone doesn't want to be discovered, they should be given that option.

-j

[Jack O'Quin]

On Mon, Aug 6, 2012 at 6:11 PM, Tully Foote <tfo...@willowgarage.com> wrote:

> However as the ROS ecosystem is growing two things are going on.
> First is the hundreds of packages have grown to thousands and the
> number of stacks has grown into the hundreds. At the same time the
> ecosystem is maturing and the core components are stabilizing and
> there is more demand for fine grain control of dependencies and
> smaller release units. As we’ve done this we’ve seen more and more
> stacks be converted to unary stacks with the package as the release
> unit.

This is surely a valid Willow Garage perspective, but likely not true
for thousands of packages developed at other locations. It does not
reflect my own experience.

> Another discussion point is that so far in ROS, we have not had
> versioned dependencies or versioned .so files. Although we’re not
> planning to enable this in the current round, we need to make sure
> we don’t prevent doing so in the future. The standard solution is
> to version at the release unit which would suggest that we have
> versions of stacks. However if we consider the capability of being
> able to move packages between stacks freely, it could easily happen
> that a package moves from a 1.5.x stack to a 1.3.x stack, and
> suddenly the package has its version number decreased. And this
> breaks many assumptions of tools such as CMake’s find_package logic,
> which can detect versions of packages. And if we are considering
> adding support for versioned .so files, suddenly you have the
> possibility of having two versions of the .so file with the same
> version number because the package changed stacks and the version
> number dropped, and then reincremented back up.

Please, please do *not* confuse .so versions with package versions.
Library API and ABI interfaces rarely change in sync with release
numbers, and the needs of the loader and the linker are more complex.

The simplest approach I know of that actually works is the GNU libtool
.so version scheme (I do *not* recommend libtool, itself). Basically,
each shared library has a version of the form "current:revision:age".
From that, the linker can record the minimum version needed by each
program at run time, and the loader knows whether the currently
installed shared library binary, meets that requirement. Similarly,
the packaging system can tell whether dependent packages need updating
when a library changes. Package release versions do not provide that
information.

http://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html

> Another feature which has been mentioned in the discussions is to
> start providing the capabilities used in packaging such as replaces,
> provides, and conflicts flags for packaging. And this could be
> woven into the dual layer system, but semantically we’d either have
> to have two namespaces like we do now, or else choose only one level
> to provide these capabilities in the future.

Probably true.

--
joq