Request for comment REP 127

[Tully Foote]

Hi All, 

One of the most strongly supported features discussed in the ROS Buildsystem SIG was to release ROS packages as debian packages.  Required to do this is to have packaging and build dependencies in each package.  Based on our discussions on the SIG Dirk has put together REP 127 with a proposal to replace stack.xml and manifest.xml with package.xml http://www.ros.org/reps/rep-0127.html

Please take a look and provide feedback.  Please reply to ros-sig-b...@googlegroups.com for the discussion. 

If you're interested and not subscribed you can subscribe or view the mailing list at https://groups.google.com/forum/?fromgroups#!forum/ros-sig-buildsystem

Tully

PS: We will also be releasing REPs to update 122-125 shortly as well.  

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[tkruse]

Hi,

thanks to Dirk for creating the REP.

My review comments (sorry for providing that many):

* packages could have a namespace (stack), or "group", or similar to give more structure. E.g.: package: amcl, group: navigation
* Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP
* URL types could be more, and the semantics should be stated when no type is given.
* bugtracker url could have more information, to help later creation of cli tool that creates tickets
* The REP could mention YAML and JSON, and why they were not chosen as a replacement
* The REP could mention which ROS package / library will provide the validation / parsing, to prevent many packages from writing their own parser.
* It might be good to think ahead, that the syntax will change, and declare what version of the package.xml syntax a given package.xml adheres to.
* The REP should provide the XML schema for review, not just announce it to be available
* maintainer could be an attribute of author.
* http://ros.org/wiki/Manifest also mentions logo tag (was that ever used?)
* The structure depends, run_depends, build_depends is not future proof, more scopes might become relevant
* Also the depends semantics should be clarified, what effect will it have if I put <depends>xyz</depends> in the package.xml?
* The document structure is weird, an additional header after the example might be useful
* build_type should be put in the example
* for the export tag, the valid subtags should be listed
* The naming convention for the name tag differ from: http://www.ros.org/wiki/Naming (lowercase, dashes?, must start with letter)
* dashes in the name???
* message_generator tag description could be more verbose, also what is the API / contract for such packages?
* REP should mention which tools are known to rely on stack.xml / package.xml (possibly also in what way they will be changed)
* "For catkin packages these files will be auto generated." sounds weird/redundant, as package.xml files are only supposed to work in catkin packages.
* why is the run_depend information used to determine build order? (Copy & paste bug) -> explain what the run_depend tag is really used for

[Damon Kohler]

I agree that the depends, build_depends, and run_depends tags are
inflexible. If you look at Maven, for one example, dependencies also
have a scope but it is specified in a separate tag (though it could
easily be an attribute). As an example limitation in the current REP,
there is no support for dependencies that are only required for
testing. The more Maven-esque way to form the example from the REP
would be:

<package>
...
<dependencies>
<depends scope="compile">libboost-thread-dev</depends>
<depends scope="run">libboost-thread</depends>
<depends scope="test">libgtest-dev</depends>
<dependencies>
</package>

To make it more compact, the compile scope could be considered the default.

> _______________________________________________
> ros-users mailing list
> ros-...@code.ros.org
> https://code.ros.org/mailman/listinfo/ros-users
>

--
:wq

Google Germany GmbH
Dienerstr. 12
80331 München

AG Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

Tax ID:- 48/725/00206
VAT ID:- DE813741370

[William Woodall]

Some small typo fixes.

https://gist.github.com/3715785

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
William Woodall

Willow Garage - Software Engineer

wwoo...@willowgarage.com

[Dirk Thomas]

I like that idea.
One minor thing: I would consider to choose an other default.
Most of our dependencies are "build" and "run", so I would propose to use both when no scope is specified.

And I guess "test" is only additive to the build and run dependencies, right?
("Build" and "run" are separate sets in your example, right?)

- Dirk

[William Woodall]

On Thu, Sep 13, 2012 at 6:41 AM, tkruse <tibo...@googlemail.com> wrote:

Hi,

thanks to Dirk for creating the REP.

My review comments (sorry for providing that many):

* packages could have a namespace (stack), or "group", or similar to give more structure. E.g.: package: amcl, group: navigation
* Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP

 

I don't think making emails into tags makes much sense, because then there is no relationship between specific author/email pairs, but brief could be a separate tag, but being an attribute might encourage people to make them briefer. 

* URL types could be more, and the semantics should be stated when no type is given.
* bugtracker url could have more information, to help later creation of cli tool that creates tickets
* The REP could mention YAML and JSON, and why they were not chosen as a replacement
* The REP could mention which ROS package / library will provide the validation / parsing, to prevent many packages from writing their own parser.

* It might be good to think ahead, that the syntax will change, and declare what version of the package.xml syntax a given package.xml adheres to.

A version attribute for <package> makes sense to me.

 

* The REP should provide the XML schema for review, not just announce it to be available
* maintainer could be an attribute of author.

Maintainer might not be an author.

 

* http://ros.org/wiki/Manifest also mentions logo tag (was that ever used?)
* The structure depends, run_depends, build_depends is not future proof, more scopes might become relevant

I like Damon's suggestion of having just <depends> but having an optional scope attribute.

 

* Also the depends semantics should be clarified, what effect will it have if I put <depends>xyz</depends> in the package.xml?
* The document structure is weird, an additional header after the example might be useful
* build_type should be put in the example
* for the export tag, the valid subtags should be listed
* The naming convention for the name tag differ from: http://www.ros.org/wiki/Naming (lowercase, dashes?, must start with letter)
* dashes in the name???

I agree, when you consider the recommendation to make the containing folder match the name, dashes don't make sense.

 

* message_generator tag description could be more verbose, also what is the API / contract for such packages?
* REP should mention which tools are known to rely on stack.xml / package.xml (possibly also in what way they will be changed)
* "For catkin packages these files will be auto generated." sounds weird/redundant, as package.xml files are only supposed to work in catkin packages.

I think he means the manifest.xml files (for automatic backwards compatibility).

 

* why is the run_depend information used to determine build order? (Copy & paste bug) -> explain what the run_depend tag is really used for

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/QxPm3ouS5UQJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

On Thursday, September 13, 2012 7:06:49 PM UTC+2, William Woodall wrote:

* Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP

 

I don't think making emails into tags makes much sense, because then there is no relationship between specific author/email pairs, but brief could be a separate tag, but being an attribute might encourage people to make them briefer. 

That would be:
<author>
<name>xyz</author>
<email>x...@xyz.com</email>
</author>
 

* The naming convention for the name tag differ from: http://www.ros.org/wiki/Naming (lowercase, dashes?, must start with letter)
* dashes in the name???

I agree, when you consider the recommendation to make the containing folder match the name, dashes don't make sense.

There are more problems, such as using the package name in generated code for variable names and such.

[William Woodall]

On Thu, Sep 13, 2012 at 10:23 AM, tkruse <tibo...@googlemail.com> wrote:

On Thursday, September 13, 2012 7:06:49 PM UTC+2, William Woodall wrote:

* Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP

 

I don't think making emails into tags makes much sense, because then there is no relationship between specific author/email pairs, but brief could be a separate tag, but being an attribute might encourage people to make them briefer. 

That would be:
<author>
<name>xyz</author>
<email>x...@xyz.com</email>
</author>

Sure that makes sense.

 

* The naming convention for the name tag differ from: http://www.ros.org/wiki/Naming (lowercase, dashes?, must start with letter)
* dashes in the name???

I agree, when you consider the recommendation to make the containing folder match the name, dashes don't make sense.

There are more problems, such as using the package name in generated code for variable names and such.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/j0QUlqosYNkJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dirk Thomas]

> * packages could have a namespace (stack), or "group", or similar to give more structure. E.g.: package: amcl, group: navigation

We should stick to a flat namespace (http://www.ros.org/wiki/Naming).

> * Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP
>
> I don't think making emails into tags makes much sense, because then there is no relationship between specific author/email pairs, but brief could be a separate tag, but being an attribute might
> encourage people to make them briefer.

I agree with William here.
"email" is an additional information which belong to the person and this would not be desireable:
<author>
<name>..</name>
<email>..</email>
</author>

> * URL types could be more, and the semantics should be stated when no type is given.

What do you imagine here? Without any concrete use cases we should not allow free-form data in the type attribute.

> * bugtracker url could have more information, to help later creation of cli tool that creates tickets

What information do you think are necessary beside the URL?
Components, version and stuff like that can be encoded in the URL (at least for trac).
And especially who should the format look like to support arbitrary bug trackers in the future?

> * The REP could mention YAML and JSON, and why they were not chosen as a replacement

First there must be a reason to choose another format (which would be limitations or other negative aspects of the XML format).
Only then it makes sense to state arguments pro and contra other formats.
And the arguments against yaml were already discussed before the switch back from stack.yaml to stack.xml was done.

> * The REP could mention which ROS package / library will provide the validation / parsing, to prevent many packages from writing their own parser.

I added a paragraph to the REP that a reference implementation will be provided.
It has still to be determined if this will be integrated in any existing tool or a new standalone tool.

> * It might be good to think ahead, that the syntax will change, and declare what version of the package.xml syntax a given package.xml adheres to.
>
> A version attribute for <package> makes sense to me.

+ 1, absolutely.

> * The REP should provide the XML schema for review, not just announce it to be available

Right, we wanted to send the REP out for review asap.
The schema will follow as soon as someone takes the time to write it.
Any contributions would be welcome :-)

> * maintainer could be an attribute of author.
>
> Maintainer might not be an author.

Since "maintainer" is required tag and "author" is only optional that is not an option.

> * http://ros.org/wiki/Manifest also mentions logo tag (was that ever used?)

I added the "logo" tag to the list of dropped stuff.

>
> * Also the depends semantics should be clarified, what effect will it have if I put <depends>xyz</depends> in the package.xml?
> * The document structure is weird, an additional header after the example might be useful
> * build_type should be put in the example

The example tries to show a common package.
Therefore the "special" tags are not in the example.
I hope the descriptions of the "build_type" tag is clear enough to derive a schema from that?

> * for the export tag, the valid subtags should be listed

The point of the specification is that the subtags are unknown and not a fixed list - so they can't be "specified".

I.e. the message generation provides its own tag (with a known name) for passing information about message generators.
But plugins define there own tag on-the-flight.
The best approach would be to create a Wiki page where we ask the community to collect all currently used export subtags.

> * The naming convention for the name tag differ from: http://www.ros.org/wiki/Naming (lowercase, dashes?, must start with letter)
> * dashes in the name???

Right, I update the REP to conform with that.

> * message_generator tag description could be more verbose, also what is the API / contract for such packages?

This should not be part of this REP.

> * REP should mention which tools are known to rely on stack.xml / package.xml (possibly also in what way they will be changed)

A complete list is extremely difficult to create and changes are not 100% foreseeable.
Imho the best approach to deal with this on demand.

> * why is the run_depend information used to determine build order? (Copy & paste bug) -> explain what the run_depend tag is really used for

Removed the duplication.

- Dirk

[Dirk Thomas]

Please reply only to the buildsystem SIG as Tully suggested in his initial email...

On 13.09.2012 10:37, William Woodall wrote:

> On Thu, Sep 13, 2012 at 10:23 AM, tkruse <tibo...@googlemail.com <mailto:tibo...@googlemail.com>> wrote:
>
>
>
> On Thursday, September 13, 2012 7:06:49 PM UTC+2, William Woodall wrote:
>
> * Attributes "brief" and "email" could be tags as well, based on the rationale given in the REP
>
> I don't think making emails into tags makes much sense, because then there is no relationship between specific author/email pairs, but brief could be a separate tag, but being an attribute
> might encourage people to make them briefer.
>
>
> That would be:
> <author>
> <name>xyz</author>

> <email>x...@xyz.com <mailto:x...@xyz.com></email>
> </author>

Why would that extra verbose variant be preferable over:
<author email="x...@xyz.com">xyz</author>

And repeating information like in the above sketched email tag is not an option.
We only want to store atomic information.

- Dirk

[tkruse]

On Thursday, September 13, 2012 7:50:45 PM UTC+2, Dirk Thomas wrote:

>     * packages could have a namespace (stack), or "group", or similar to give more structure. E.g.: package: amcl, group: navigation

We should stick to a flat namespace (http://www.ros.org/wiki/Naming).

What I suggest could also be realized as "tags", like:
<tag>pr2</tag>
<tag>navigation</tag>

So this does not contradict the flat namespace.
 

>     * URL types could be more, and the semantics should be stated when no type is given.

What do you imagine here? Without any concrete use cases we should not allow free-form data in the type attribute.

homepage, root repository, wiki, bugtracker, changelog, tar-download, mailinglist, ohloh, freecode, ...
 

>     * bugtracker url could have more information, to help later creation of cli tool that creates tickets

What information do you think are necessary beside the URL?
Components, version and stuff like that can be encoded in the URL (at least for trac).
And especially who should the format look like to support arbitrary bug trackers in the future?

The type of tracker, so that hopefully from the base URL one can infer the link for a new issue as well as for certain reports.
Else I am not sure, I think Jon Bohren looked more into this.
 

>     * The REP could mention YAML and JSON, and why they were not chosen as a replacement

First there must be a reason to choose another format (which would be limitations or other negative aspects of the XML format).
Only then it makes sense to state arguments pro and contra other formats.
And the arguments against yaml were already discussed before the switch back from stack.yaml to stack.xml was done.

Time has passed since then, but if the arguments still hold, a link to that discussion would be fine.

 

>     That would be:
>     <author>
>     <name>xyz</author>
>     <email>x...@xyz.com <mailto:x...@xyz.com></email>
>     </author>

Why would that extra verbose variant be preferable over:
   <author email="x...@xyz.com">xyz</

author>

And repeating information like in the above sketched email tag is not an option.
We only want to store atomic information.

The "mailto" part was added by a mailreader, not me.
and the "extra verbose" variant just foollows the REP:

> Tags vs. attributes

> While plain text information can be stored in either attributes or tags the later is preferred for readability reasons.

Verbosity is just XML, and else I follow guielines such as:http://www.ibm.com/developerworks/xml/library/x-eleatt/index.html
But it's not a big deal either way.

[Jack O'Quin]

On Thu, Sep 13, 2012 at 3:06 PM, Jochen Sprickerhof
<ros-...@jochen.sprickerhof.de> wrote:
> * Tully Foote <tfo...@willowgarage.com> [2012-09-13 03:53]:
>> Hi All,
>
> Hi Tully,

>
>> One of the most strongly supported features discussed in the ROS
>> Buildsystem SIG was to release ROS packages as debian packages. Required
>> to do this is to have packaging and build dependencies in each package.
>> Based on our discussions on the SIG Dirk has put together REP 127 with a
>> proposal to replace stack.xml and manifest.xml with package.xml
>> http://www.ros.org/reps/rep-0127.html
>

> This looks almost like the information found in a Debian
> (source-)package but with some information missing. I would propose to
> either go directly with their file formats or at least add all the
> missing information (like versioned dependencies, recommends...), so we
> can generate more standard compliant packages. You can find more
> information here:
> http://en.wikipedia.org/wiki/Debian_build_toolchain
> http://www.debian.org/doc/manuals/maint-guide/
> http://www.debian.org/doc/debian-policy/

+1

I am not convinced that *everything* defined in a Debian control file
should necessarily be provided (I have doubts about Pre-Depends, for
example). But, *most* package relations are needed, such as: Breaks,
Conflicts, Enhances, Replaces, Recommends, Suggests.

Package version dependencies are definitely necessary. Failing to
implement that now will almost certainly cause considerable breakage
when we eventually end up doing it.
--
joq

[William Woodall]

I agree with Jack, I am not convinced that much of that information belongs in the package.xml. Why not make the package.xml file the common denominator and put platform specifics in the gbp release repository?

I am all for adding a piece of meta information if it translates into debian, rpm, homebrew, source install, etc.., but if it really only applies to debian releases why put it in there?  I just don't want to see the package.xml file get too complicated, I agree that this information is necessary, I just don't know if it belongs in the package.xml file.

For instance, if all of a sudden we decide we need Homebrew specific cmake arguments, do we add a <homebrew_cmake_args> tag to the package.xml file rather than putting that in the release repository?

On the other hand, adding a version attribute to the <depends> tag, even if it is not used by some platforms, is an acceptable suggestion in my opinion.

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Jack O'Quin]

On Thu, Sep 13, 2012 at 12:50 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
>> * packages could have a namespace (stack), or "group", or similar to
>> give more structure. E.g.: package: amcl, group: navigation
>
> We should stick to a flat namespace (http://www.ros.org/wiki/Naming).

Agreed about the flat namespace.

But, note that http://www.ros.org/wiki/Naming was never enforced, and
we do in fact have packages with names that start with a number.

Are we planning to start enforcing that now?

>> * The REP could mention YAML and JSON, and why they were not chosen as
>> a replacement
>
> First there must be a reason to choose another format (which would be
> limitations or other negative aspects of the XML format).
> Only then it makes sense to state arguments pro and contra other formats.
> And the arguments against yaml were already discussed before the switch back
> from stack.yaml to stack.xml was done.

The negative aspects of XML are clear to us all.

Since YAML was an obvious enough choice to have been discussed
already, that rationale should be included in the REP. Some of us were
not part of that discussion and wonder why YAML was rejected. Future
readers of the REP may also wonder.

>> * Also the depends semantics should be clarified, what effect will it
>> have if I put <depends>xyz</depends> in the package.xml?
>> * The document structure is weird, an additional header after the
>> example might be useful
>> * build_type should be put in the example
>
> The example tries to show a common package.
> Therefore the "special" tags are not in the example.
> I hope the descriptions of the "build_type" tag is clear enough to derive a
> schema from that?

Not particularly clear to me.

>> * REP should mention which tools are known to rely on stack.xml /
>> package.xml (possibly also in what way they will be changed)
>
> A complete list is extremely difficult to create and changes are not 100%
> foreseeable.
> Imho the best approach to deal with this on demand.

Even if not 100% complete, listing the known tools would be extremely
valuable to people reading this REP.

--
joq

[Dirk Thomas]

On 13.09.2012 14:13, Jack O'Quin wrote:
> On Thu, Sep 13, 2012 at 12:50 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
>
> But, note that http://www.ros.org/wiki/Naming was never enforced, and
> we do in fact have packages with names that start with a number.
>
> Are we planning to start enforcing that now?

The most important point is that out naming must be in line with where the package name is used:
* naming namespaces
* naming variables
* xml-tags

And how packages are processed in the build and packaging steps (see the Debian and Fedora naming policies for that):
* http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source
* http://fedoraproject.org/wiki/Packaging:NamingGuidelines#Common_Character_Set_for_Package_Naming

The conclusion based on these:
* Only lower case letters (a-z), numbers (0-9) and one delimiting character (_) can be used.
Reason: some of the above cases do not support case sensitivity and only underscore can be used in most scenarios
(even if that one must be replace with a dash for Debian packages, but this mapping affects the user/developer the least).

* And the first letter must be a letter (and not a number or _).
Reason: some languages do not allow to name a namespace/package with a number in the first character.

So I would propose, yes, we should "enforce" that policy for package names.
This does not imply that we have to bail out and i.e. deny building a package like that (if the tools can handle it well).

But if people are not following these conventions they can just not expect that this is supported.
E.g. A creates a package "1foo" and calls his Python package "one_foo" (which looks reasonable).
B creates a package "one_foo" and names his Python package accordingly...

>>> * The REP could mention YAML and JSON, and why they were not chosen as
>>> a replacement
>>
>> First there must be a reason to choose another format (which would be
>> limitations or other negative aspects of the XML format).
>> Only then it makes sense to state arguments pro and contra other formats.
>> And the arguments against yaml were already discussed before the switch back
>> from stack.yaml to stack.xml was done.
>
> The negative aspects of XML are clear to us all.
>
> Since YAML was an obvious enough choice to have been discussed
> already, that rationale should be included in the REP. Some of us were
> not part of that discussion and wonder why YAML was rejected. Future
> readers of the REP may also wonder.

http://lists.willowgarage.com/pipermail/catkin/2012-March/000111.html
https://github.com/ros/catkin/issues/128

While reason 1-3 were implementation related issues reason number 4 is a valid one.

>>> * build_type should be put in the example
>>
>> The example tries to show a common package.
>> Therefore the "special" tags are not in the example.
>> I hope the descriptions of the "build_type" tag is clear enough to derive a
>> schema from that?
>
> Not particularly clear to me.

I added the default to the package.xml example and an example block with variants below the tag description.

>>> * REP should mention which tools are known to rely on stack.xml /
>>> package.xml (possibly also in what way they will be changed)
>>
>> A complete list is extremely difficult to create and changes are not 100%
>> foreseeable.
>> Imho the best approach to deal with this on demand.
>
> Even if not 100% complete, listing the known tools would be extremely
> valuable to people reading this REP.

The problem is that this REP deals only with the specification of the meta data of a package.
I added a list of affected software to the REP for that.
But the majority of the tools are affected because of the underlying transition from packaging packages instead of stacks.

- Dirk

[Jack O'Quin]

I agree with you. We should enforce it.

There are not very many such packages listed on the Browse Software
page. They will need to change their names while converting to catkin
(if they ever do).

That implies the tools should reject invalid names with a clear error message.

>>>> * The REP could mention YAML and JSON, and why they were not chosen
>>>> as a replacement
>>>
>>> First there must be a reason to choose another format (which would be
>>> limitations or other negative aspects of the XML format).
>>> Only then it makes sense to state arguments pro and contra other formats.
>>> And the arguments against yaml were already discussed before the switch
>>> back from stack.yaml to stack.xml was done.
>>
>> The negative aspects of XML are clear to us all.
>>
>> Since YAML was an obvious enough choice to have been discussed
>> already, that rationale should be included in the REP. Some of us were
>> not part of that discussion and wonder why YAML was rejected. Future
>> readers of the REP may also wonder.
>
>
> http://lists.willowgarage.com/pipermail/catkin/2012-March/000111.html
> https://github.com/ros/catkin/issues/128
>
> While reason 1-3 were implementation related issues reason number 4 is a
> valid one.

So, it's simple. YAML is easier for humans to read and XML is easier
for computers.

A 10x parser speed-up is significant and should be documented in the
REP as a rationale.

Is that the only reason? Some tools must support both wet and dry
packages: they might be slightly simpler for only needing one parser.
--
joq

[Jack O'Quin]

On Thu, Sep 13, 2012 at 5:53 AM, Tully Foote <tfo...@willowgarage.com> wrote:
>
> One of the most strongly supported features discussed in the ROS Buildsystem
> SIG was to release ROS packages as debian packages. Required to do this is
> to have packaging and build dependencies in each package. Based on our
> discussions on the SIG Dirk has put together REP 127 with a proposal to
> replace stack.xml and manifest.xml with package.xml
> http://www.ros.org/reps/rep-0127.html
>
> Please take a look and provide feedback. Please reply to
> ros-sig-b...@googlegroups.com for the discussion.

Some miscellaneous comments not already mentioned in other messages:

Abstract:
=======

Make it clear that this REP applies only to "wet" (i.e. catkin) packages.

Optional Tags; Dependencies:
========================

We need something more flexible, perhaps like Damon's suggestion.
There are many kinds of dependencies that are worth describing, even
if some underlying packaging systems ignore or cannot support them.

Example:
=======

Is catkin really both a build and run-time dependency? Why not only a
build_depends?

<version>:
=========

To me, the ABI version number is a property of each library. It should
be specified using the CMake SOVERSION property, not at the
package.xml level.

Dirk and I had an interesting discussion about this in a separate
thread. I now believe he is proposing to use something at the package
level that is similar to library so-versions as a more flexible way to
describe package version dependencies. That may be possible. I don't
fully understand the implications. But, for now I prefer using the
Debian scheme for specifying package version dependencies. We need to
be able to generate them in the Debian control file, anyway.

<maintainer>:
===========

I dislike *requiring* the maintainer to specify an e-mail address. It
seems like a spammer magnet, and I would prefer not to provide mine in
such a widely-known, easily-parsed format.

I would rather encourage maintainers to provide a working bug tracker
URL. If they do not, that provides a valuable clue to potential users
that they should maybe look elsewhere.

<depends>:
==========

I can't figure out a use case for a convenience tag that is both a
<build_depends> and a <run_depends>. My recommendation: rename
<run_depends> to <depends>. That should be the usual case, and it
matches Debian terminology.

I would like to specify most of the Debian package dependency
semantics, including: Breaks, Conflicts, Enhances, Replaces,
Recommends, Suggests. As someone else suggested, "<tests>" might also
be useful.

Modifications to previous specifications:
===============================

Why is the license information now optional?

Are <depend> and <rosdep> really the same? Or, only to catkin at build time?

Future enhancements:
==================

I believe we need full Debian-style package version dependency
semantics now. I expect that adding them later will create a mess,
requiring considerable re-work of both packages and tools.

Further, we need at least some of those semantics to provide something
similar to dry stack package version coherence. Without that,
component maintenance becomes a much bigger combinatorics problem.
--
joq

[Dirk Thomas]

On 13.09.2012 18:55, Jack O'Quin wrote:
> On Thu, Sep 13, 2012 at 5:53 AM, Tully Foote <tfo...@willowgarage.com> wrote:
>
> Make it clear that this REP applies only to "wet" (i.e. catkin) packages.

Added.

> Optional Tags; Dependencies:
> ========================
>
> We need something more flexible, perhaps like Damon's suggestion.
> There are many kinds of dependencies that are worth describing, even
> if some underlying packaging systems ignore or cannot support them.

The feedback of Damon has been integrated to the REP.

> Is catkin really both a build and run-time dependency? Why not only a
> build_depends?

Catkin is a build dependency because it provide CMake and Python which is mandatory for the build process of packages.
And catkin is also a run-time dependency because it provides the functionality and tools which are used at runtime to work with the multiple separate workspaces (i.e. finding resources like binaries).

It is arguable why both parts are provided in one package.
Splitting them into "catkin-dev" (build dep.) and "catkin" (run-time dep.) would be a feasible option.

> <version>:
> =========
>
> To me, the ABI version number is a property of each library. It should
> be specified using the CMake SOVERSION property, not at the
> package.xml level.
>
> Dirk and I had an interesting discussion about this in a separate
> thread. I now believe he is proposing to use something at the package
> level that is similar to library so-versions as a more flexible way to
> describe package version dependencies. That may be possible. I don't
> fully understand the implications. But, for now I prefer using the
> Debian scheme for specifying package version dependencies. We need to
> be able to generate them in the Debian control file, anyway.

We do have the imminent need to be able to decide if a new package release must trigger a rebuilt of all dependent packages.
Currently the rebuild is triggered always.
This also applies to packages which do not contain a single so file so it can not be solved with any kind of so versioning.

Consider the use case that a single comment line is changed in catkin.
When the package is released de-facto all ROS packages are needlessly rebuilt.
Since this takes several hours of round-trip time on the build farm and bloats the round-trip time we want to avoid that asap.

The currently proposed ABI version attribute solves that particular problem (but might not be enough/the right way for other use cases).
Since judging ABI compatibility is not trivial this feature will likely only be used by a couple or low-level packages which are affected most be a wrong rebuilt trigger.


Besides that we have two additional topics: so versioning and versioned dependencies.
The problem is that we can not specify the above ABI version approach with a clear conscience without knowing how it affects the other two topics (now or in the future).

(More on that with the last paragraph.)

> <maintainer>:
> ===========
>
> I dislike *requiring* the maintainer to specify an e-mail address. It
> seems like a spammer magnet, and I would prefer not to provide mine in
> such a widely-known, easily-parsed format.
>
> I would rather encourage maintainers to provide a working bug tracker
> URL. If they do not, that provides a valuable clue to potential users
> that they should maybe look elsewhere.

The point here is again: what does the toolchain require?
Debian packages require a valid email address to be set.
If we do not have these information in the manifest we can never release official Debian packages (which is the goal for even more distributions in the future).
http://www.debian.org/doc/debian-policy/ch-binary.html#s-maintainer

> <depends>:
> ==========
>
> I can't figure out a use case for a convenience tag that is both a
> <build_depends> and a <run_depends>. My recommendation: rename
> <run_depends> to <depends>. That should be the usual case, and it
> matches Debian terminology.
>
> I would like to specify most of the Debian package dependency
> semantics, including: Breaks, Conflicts, Enhances, Replaces,
> Recommends, Suggests. As someone else suggested, "<tests>" might also
> be useful.
>
> Modifications to previous specifications:
> ===============================
>
> Why is the license information now optional?

The idea is to make all information required which we need to perform the necessary steps.
And on the other hand make all information which is not necessary optional (since people would fill them with nonsense anyway).
It should still be recommended to set the license information.

> Are <depend> and <rosdep> really the same? Or, only to catkin at build time?

They are already unified in Fuerte.
Which case do you see where that would not be the case?

> Future enhancements:
> ==================
>
> I believe we need full Debian-style package version dependency
> semantics now. I expect that adding them later will create a mess,
> requiring considerable re-work of both packages and tools.
>
> Further, we need at least some of those semantics to provide something
> similar to dry stack package version coherence. Without that,
> component maintenance becomes a much bigger combinatorics problem.

I do understand the need for that.
But it is not a must-have feature for Groovy.
As long as we continue to release ROS packages version-locked as we do it currently nothing is going to change - no version problems will pop up out of a sudden.


For both open topics (so versioning and versioned dependencies) we lack any kind of overall design.
We do not have any spec what information we want to store nor how these information are used and effect the existing toolchain.
It does not have to be something official - any informal collection on these topic would hopefully get us further.

Either we need to make a lot of progress on these topics in a short amount of time or we should postpone them for the next release cycle if we don't have enough information to realize them now.
Postponing does not mean that we should stop thinking about it - it would still be beneficial if we would have a concrete sketch on who to tackle these (even if we don't get them integrated for Groovy).

- Dirk

[Dirk Thomas]

I have updated the REP:

- updated <depend> tag to include the types mentioned by Tully in his last email ("conflict" and "replace")
- update version attributes to be "atomic" yet readable and combinable
- allow license to appear multiple times
- added meta-package flag (necessary functionality for ROS wiki pages to generate stack-ish package relations)
- clarified rules for tags under <export>

Please take a close look to the document again: http://www.ros.org/reps/rep-0127.html
and post any feedback so that we get a consensus on all the details.

We will likely have to move forward soon (due to the advanced time), start updating the toolchain to support package.xml and upgrade the existing wet stacks...

Thank you
- Dirk


On 13.09.2012 20:22, Bill Morris wrote:

> On Thu, 2012-09-13 at 03:53 -0700, Tully Foote wrote:
>> Hi All,
>>
>>
>> One of the most strongly supported features discussed in the ROS
>> Buildsystem SIG was to release ROS packages as debian packages.
>> Required to do this is to have packaging and build dependencies in
>> each package. Based on our discussions on the SIG Dirk has put
>> together REP 127 with a proposal to replace stack.xml and manifest.xml
>> with package.xml http://www.ros.org/reps/rep-0127.html
>>
>>
>> Please take a look and provide feedback. Please reply
>> to ros-sig-b...@googlegroups.com for the discussion.
>

> Since you are getting rid of manifest.xml we would like to reiterate and
> formalize support for extending the export statement. We have been
> working on extending manifest.xml to support exporting URDF files and
> launch files.
>
> https://github.com/IHeartRobotics/iheart-ros-pkg/tree/master/ihe_hardware/urdf_compose
>
> manifest.xml
>
> <package>
> ....
> <export>
> <urdf_compose urdf="urdf/cup_holder.urdf"/>
> </export>
> </package>
>
> Some hardware we are developing has launch files that should be started
> at bootup. To make this user friendly we would like to make a launch
> manager gui, packages could then export launch files that they would
> like to offer the user to start at bootup
>
> https://github.com/TurtleBot-Mfg/ros-system-tools/blob/master/rosmetalaunch
>
> manifest.xml
>
> <package>
> ....
> <export>
> <launch_manager launch="button_interface.launch"/>
> </export>
> </package>
>
> For some reason it looks like both of these use cases currently work and
> we would like to maintain and formalize support for this.
>
> Also, this approach may work well for the appmanager to locate .app
> files provided by various packages.
>
> Thanks,
> Bill
>

[Jack O'Quin]

On Tue, Sep 18, 2012 at 11:50 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
> I have updated the REP:
>
> - updated <depend> tag to include the types mentioned by Tully in his last
> email ("conflict" and "replace")
> - update version attributes to be "atomic" yet readable and combinable
> - allow license to appear multiple times
> - added meta-package flag (necessary functionality for ROS wiki pages to
> generate stack-ish package relations)
> - clarified rules for tags under <export>
>
> Please take a close look to the document again:
> http://www.ros.org/reps/rep-0127.html
> and post any feedback so that we get a consensus on all the details.
>
> We will likely have to move forward soon (due to the advanced time), start
> updating the toolchain to support package.xml and upgrade the existing wet
> stacks...

Looks like good progress. Comments by section follow:

Rationale:
========

Still missing the "Why not YAML?" comparison (mostly machine-readable
vs. human-readable, but maybe there are other reasons).

<version>; Attributes:
================

I understood you to say that Groovy will *not* implement
"libtool-style" package versioning. If so, let's remove this section,
with a possible mention later on as potential future work.

In any case, the term "ABI" is confusing and misleading in the context
of package versions. It really applies to the library-level so-version
scheme. Package versions are not just ABI.

(I would very much like to see a section in the catkin docs
recommending that people set the SOVERSION property with an example of
how to do that.)

<description>; Attributes:
===================

Minor wording edit: instead of "to (sic) verbose to be displayed" I
suggest something like "verbose enough to need a summary".

<depend>:
========

I like the improved flexibility of this section.

But the proposed syntax is ugly, even by XML standards. Surely there
is a better representation. I thought the original proposal with
nested dependency tags was cleaner. Using empty attributes rather than
tags seems confusing and error-prone for users.

I disagree with both "build" and "run" being the default attribute. I
would be astonished if that were the "most common use case". Surely
"run" is the most common? So far, "catkin" itself is the only example
identified of a dependency that is both. I would prefer dividing
catkin into separate build and run-time packages and *not* *allowing*
any single dependency to be both.

The "conflict" attribute is repeated in the text. I think the second
instance was intended to be something like "replaces".

Have you decided to implement package version dependencies? I had thought not.

I am happy to see it included in Groovy. I don't much like the
"version_gte" syntax style, but can live with it to get that important
capability. A little more semantic detail is needed: how do multiple
version attributes (like >= 2 and <4) work? Can they be connected with
AND and OR? Although that depends on the underlying packaging systems,
it needs to be spelled out for ROS, too.

<architecture_independent/>:
======================

This seems a bit simplistic. The Debian equivalent actually specifies
which architectures the package supports, with "any" or "all" as
useful options.

<meta_package/>:
==============

Is this being implemented for Groovy? If so, I doubt an empty tag will
be sufficient.

As mentioned in the review meeting, I don't like that name, although
it was useful for earlier discussions to distinguish them from dry
stacks. I recommend keeping the old "stack" terminology for users,
because it is already familiar to the ROS community.

Concerns:
========

I would prefer to keep the "review" tag. It is one of the best ways to
quickly identify poorly-maintained packages.

Backwards Compatibility:
===================

As someone mentioned earlier, saying the `manifest.xml` will be auto
generated is mostly just confusing in this context. While some tools
may work that way, users (presumably) don't see those files and don't
care about them. Right?
--
joq

[Tully Foote]

Hi Jack, 

I've replied inline:

At the moment these are all constraints, thus would imply AND logic.  As far as I know this is all the debian provides.  You can get NOT using the conflicts.  OR capabilities seem to be done using capabilities which is something we can consider adding later, but for now I suggest we leave it out. 

 

<architecture_independent/>:
 ======================

This seems a bit simplistic. The Debian equivalent actually specifies
which architectures the package supports, with "any" or "all" as
useful options.

For others not familiar see the Architecture section: http://www.debian.org/doc/debian-policy/ch-controlfields.html

Our current infrastructure only supports the "all" semantics.  But thinking about this it seems to me that this is a very packaging specific attribute.  And as such it likely should be considered not to be contained in the package.xml but be in the bloom configuration. 

 

<meta_package/>:
==============

Is this being implemented for Groovy? If so, I doubt an empty tag will
be sufficient.

As mentioned in the review meeting, I don't like that name, although
it was useful for earlier discussions to distinguish them from dry
stacks. I recommend keeping the old "stack" terminology for users,
because it is already familiar to the ROS community.

With respect to the naming, if we want to do this keeping the stack name will be valuable in documentation and usage.  However I'd suggest as we look to move forward toward more standard packaging concepts, metapackage draws a better parallel to other systems, such as debian type packaging.  

What information would be needed to make it sufficient? 

 

Concerns:
========

I would prefer to keep the "review" tag. It is one of the best ways to
quickly identify poorly-maintained packages.

Backwards Compatibility:
===================

As someone mentioned earlier, saying the `manifest.xml` will be auto
generated is mostly just confusing in this context. While some tools
may work that way, users (presumably) don't see those files and don't
care about them. Right?
--
 joq

Tully 

[Jack O'Quin]

On Wed, Sep 19, 2012 at 9:10 PM, Tully Foote <tfo...@willowgarage.com> wrote:
>> Have you decided to implement package version dependencies? I had thought
>> not.
>>
>> I am happy to see it included in Groovy. I don't much like the
>> "version_gte" syntax style, but can live with it to get that important
>> capability. A little more semantic detail is needed: how do multiple
>> version attributes (like >= 2 and <4) work? Can they be connected with
>> AND and OR? Although that depends on the underlying packaging systems,
>> it needs to be spelled out for ROS, too.
>
> At the moment these are all constraints, thus would imply AND logic. As far
> as I know this is all the debian provides. You can get NOT using the
> conflicts. OR capabilities seem to be done using capabilities which is
> something we can consider adding later, but for now I suggest we leave it
> out.

Seems good. The AND (constraint) logic should be documented, that's
all. Otherwise, people won't know how to use it.

>> <architecture_independent/>:
>> ======================
>>
>> This seems a bit simplistic. The Debian equivalent actually specifies
>> which architectures the package supports, with "any" or "all" as
>> useful options.
>
>
> For others not familiar see the Architecture section:
> http://www.debian.org/doc/debian-policy/ch-controlfields.html
>
> Our current infrastructure only supports the "all" semantics. But thinking
> about this it seems to me that this is a very packaging specific attribute.
> And as such it likely should be considered not to be contained in the
> package.xml but be in the bloom configuration.

I could see doing it either way. Any architecture dependencies or
independence are essentially properties of the implementation.

>>
>> <meta_package/>:
>> ==============
>>
>> Is this being implemented for Groovy? If so, I doubt an empty tag will
>> be sufficient.
>>
>> As mentioned in the review meeting, I don't like that name, although
>> it was useful for earlier discussions to distinguish them from dry
>> stacks. I recommend keeping the old "stack" terminology for users,
>> because it is already familiar to the ROS community.
>
>
> With respect to the naming, if we want to do this keeping the stack name
> will be valuable in documentation and usage. However I'd suggest as we look
> to move forward toward more standard packaging concepts, metapackage draws a
> better parallel to other systems, such as debian type packaging.
>
> What information would be needed to make it sufficient?

I really don't know, and I doubt anyone will until we actually implement it.

Is it in-plan for Groovy?

--
joq

[Brian Gerkey]

[Arriving late to the party, digging through this thoughtful discussion.]

On Thu, Sep 13, 2012 at 8:45 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
> On 13.09.2012 18:55, Jack O'Quin wrote:
>> Why is the license information now optional?
>
> The idea is to make all information required which we need to perform the
> necessary steps.
> And on the other hand make all information which is not necessary optional
> (since people would fill them with nonsense anyway).
> It should still be recommended to set the license information.

License should absolutely be required. Whether or not the tools needs
that data, it's vital that we have all the code properly tagged with
license statements. Otherwise we will end up with unlicensed released
code, which sensible people (surely all companies) won't consider
relying on.

I'd rather authors be forced to enter nonsense than be allowed to
forget to give the license.

What's the reason to support multiple <license> tags? If we're now
packaging at the package level, it wouldn't be too much of a
constraint to say that all the constituent code must be governed by a
single license.

brian.

[Brian Gerkey]

On Thu, Sep 13, 2012 at 3:53 AM, Tully Foote <tfo...@willowgarage.com> wrote:
> One of the most strongly supported features discussed in the ROS Buildsystem
> SIG was to release ROS packages as debian packages. Required to do this is
> to have packaging and build dependencies in each package. Based on our
> discussions on the SIG Dirk has put together REP 127 with a proposal to
> replace stack.xml and manifest.xml with package.xml
> http://www.ros.org/reps/rep-0127.html

Comments:

* I suggest removing the 'brief' attribute from the <description> tag.
Historically it's never been properly used; in the cases where it's
set in a manifest.xml, 99% of the time it's equal to the name of the
package.

* Multiple <copyright> tags should be allowed. Copyright generally
accrues to the authors (or their employers), and is extended as the
author list grows. If there can be multiple authors, there can be
multiple copyright holders. But:

* I don't see the reason to include <copyright> in this file. It's a
great opportunity for out-of-date and inconsistent data. It's more
important to maintain author and license data.

* I agree with Jack that the version_*** attribute syntax for
dependencies is pretty bad. But then I'm offended by pretty much all
XML. I assume that a better format was sought out and not found?

Otherwise, this is looking good. After all the push-and-pull over the
last year of manifest.xml, stack.xml, stack.yaml, etc., it's great to
see a sensible, agreed-upon format emerging.

brian.

[tkruse]

Some more suggestions:

For posteriority, also reference the discussions on the REP, both this one as well as the one where the step towards packages was discussed.

Regarding the 'format' attribute,
<package format='1'>
Either this should be mandatory (and self-explaining like catkin_version='0.6'), or we should drop this. By now I think this should rather be dropped, and structure changes be introduced by tic-toc deprecation.


Regarding creating a schema, a good starting point is a schema-from-example generation tool like trang:
$ sudo apt-get install trang
$ trang package1.xml package2.xml package.xsd
By using multiple suitable examples, trang can infer what things are optional and what sequence order does not matter.
It cannot infer the semantics of the export tag, so that has to be replaced with:
<xs:element name="export">
    <xs:complexType>
      <xs:sequence>
        <xs:any minOccurs="0"  maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
The validation can be done like this:
$ xmllint --schema package.xsd package.xml --noout


For dependency versions, I would prefer "version=..." instead of version_gte, and version_exact instead or version_eq.
I am not sure whether version_lte or version_gt are useful options. Would the following be insufficient:

<build_depend version="1.1">...</build_depend>

<build_depend version="1.1" version_lower_than="2.1">...</build_depend>
<build_depend version_exact="1.1">...</build_depend>


As to the depend tags, I would still prefer in the REP for each tag an explanation and example for both 
c++ and python, of what changes to the tag causes what changes in the build/run/packaging. Currently the REP only says:
"The build and buildtool dependencies are used to determine the build order of multiple packages." (which is a bit 
confusing as well, given catkin only seems to use those for configure-order of packages)

How is this used to determine the build order of anything:
<build_depend>libboost-thread-dev</build_depend>
unless somebody wrote a libboost-thread-dev ROS package?

What happens if a developer forgets to specify:
<test_depend>gtest</test_depend>
or

<buildtool_depend>...</buildtool_depend>

?

Will this:
<run_depend>libboost-thread</run_depend>
be parsed and used to specify linker flags, or will it also/only be used when creating a debian package?

While the REP should not specify how catkin will work, it needs to justify why having those tags at all 
(as opposed to keeping that information somewhere else),
and to make it clear I would like explicit examples of what tag goes into what process how.


I think the REP needs to state whether transitive dependencies will be resolved automatically or have to be stated again.
Meaning if ros package A <..._depends> on ros package B, how do the build-, buildtool-, run-, and test-dependencies 
of B affect the dependencies of A.


+1 to removing the brief, also see http://www.ros.org/browse/list.php which shows how the full description can be cut off 
and still be useful when people give a short and meaningful first sentence.


-1 to keeping review tag, i think that was agreed upon in the review meeting (http://www.ros.org/wiki/catkin/Reviews/2012-08-14_API_Review)

[Jonathan Bohren]

Sorry, I've been out of the loop, ICRA madness and all...

So one big thing that I remember discussing with Tully back in July, was making the manifest file have a ros-specific name, so that things like webcrawlers could easily find something that looks like a ros package on source servers. For example, instead of "package.xml" just calling it "rospackage.xml" would mean that I could tell if something was trying to be a ROS package just by looking at its files. This will also decrease the likelihood of naming collisions with a package that also wants to have some other file called "package.xml"

On Thu, Sep 20, 2012 at 8:11 AM, tkruse <tibo...@googlemail.com> wrote:

Regarding the 'format' attribute,
<package format='1'>
Either this should be mandatory (and self-explaining like catkin_version='0.6'), or we should drop this. By now I think this should rather be dropped, and structure changes be introduced by tic-toc deprecation.

Yeah, I think it's reasonable for this to be mandatory. That being said "package format=1" could be confusing because it's talking about the package spec and not the package itself. If we change the xml filename to "rospackage.xml" then in the root tag we could do:

<rospackage format='1'>

</rospackage>

 

+1 to removing the brief, also see http://www.ros.org/browse/list.php which shows how the full description can be cut off 
and still be useful when people give a short and meaningful first sentence.

+1 as well

-1 to keeping review tag, i think that was agreed upon in the review meeting (http://www.ros.org/wiki/catkin/Reviews/2012-08-14_API_Review)

-1 as well 

-j

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[Jack O'Quin]

On Thu, Sep 20, 2012 at 9:00 AM, Jonathan Bohren
<jonatha...@gmail.com> wrote:
> Sorry, I've been out of the loop, ICRA madness and all...
>
> So one big thing that I remember discussing with Tully back in July, was
> making the manifest file have a ros-specific name, so that things like
> webcrawlers could easily find something that looks like a ros package on
> source servers. For example, instead of "package.xml" just calling it
> "rospackage.xml" would mean that I could tell if something was trying to be
> a ROS package just by looking at its files. This will also decrease the
> likelihood of naming collisions with a package that also wants to have some
> other file called "package.xml"

+1

> On Thu, Sep 20, 2012 at 8:11 AM, tkruse <tibo...@googlemail.com> wrote:
>>
>> Regarding the 'format' attribute,
>> <package format='1'>
>> Either this should be mandatory (and self-explaining like
>> catkin_version='0.6'), or we should drop this. By now I think this should
>> rather be dropped, and structure changes be introduced by tic-toc
>> deprecation.
>
> Yeah, I think it's reasonable for this to be mandatory. That being said
> "package format=1" could be confusing because it's talking about the package
> spec and not the package itself. If we change the xml filename to
> "rospackage.xml" then in the root tag we could do:
>
> <rospackage format='1'>
> </rospackage>

+1

My initial reaction was to prefer "version" rather than "format", but
I quickly recognized that "version" is heavily over-loaded in this
context and should be avoided for describing the file format.

>> -1 to keeping review tag, i think that was agreed upon in the review
>> meeting (http://www.ros.org/wiki/catkin/Reviews/2012-08-14_API_Review)
>>
> -1 as well

I don't recall any discussion about the "review" tag during the review
meeting, but I was distracted by a number of other, more important,
issues and probably forgot.

I want to collect QA information automatically. Parsing arbitrary web
pages for that information looks a lot harder, reliably determining
whether it is present or missing. Providing an optional "review" tag
permits developers to declare a commitment to quality and
compatibility. I hope we can combine that with other measures of
package quality and reusability to provide meaningful guidance for
users.
--
joq

[Jack O'Quin]

On Thu, Sep 20, 2012 at 1:22 AM, Brian Gerkey <ger...@osrfoundation.org> wrote:
> [Arriving late to the party, digging through this thoughtful discussion.]
>
> On Thu, Sep 13, 2012 at 8:45 PM, Dirk Thomas <dth...@willowgarage.com> wrote:
>> On 13.09.2012 18:55, Jack O'Quin wrote:
>>> Why is the license information now optional?
>>
>> The idea is to make all information required which we need to perform the
>> necessary steps.
>> And on the other hand make all information which is not necessary optional
>> (since people would fill them with nonsense anyway).
>> It should still be recommended to set the license information.
>
> License should absolutely be required. Whether or not the tools needs
> that data, it's vital that we have all the code properly tagged with
> license statements. Otherwise we will end up with unlicensed released
> code, which sensible people (surely all companies) won't consider
> relying on.
>
> I'd rather authors be forced to enter nonsense than be allowed to
> forget to give the license.

+1
--
joq

[Jack O'Quin]

On Thu, Sep 20, 2012 at 1:37 AM, Brian Gerkey <ger...@osrfoundation.org> wrote:
>
> * I don't see the reason to include <copyright> in this file. It's a
> great opportunity for out-of-date and inconsistent data. It's more
> important to maintain author and license data.

+1

> * I agree with Jack that the version_*** attribute syntax for
> dependencies is pretty bad. But then I'm offended by pretty much all
> XML. I assume that a better format was sought out and not found?

YAML was the only other format seriously considered. Ken Conley did
some performance tests of the available Python parsers and found that
XML was 10x faster than YAML. I don't know which parsers he used. He
seemed to regard that difference as significant for some uses of the
file, I don't know which ones. But, I certainly respect Ken's opinion.

Since this information is not widely known or documented, I recommend
adding "Why XML?" or "Why not YAML?" to the Rationale section.
--
joq

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 11:24 AM, Jack O'Quin <jack....@gmail.com> wrote:

I want to collect QA information automatically. Parsing arbitrary web
pages for that information looks a lot harder, reliably determining
whether it is present or missing. Providing an optional "review" tag
permits developers to declare a commitment to quality and
compatibility. I hope we can combine that with other measures of
package quality and reusability to provide meaningful guidance for
users.

Yeah, so I started hacking on something I call "roscensus" earlier in the summer, and now plan to dive back into it. I started off just collecting information from package manifests, but what I'd like to do is build a web backend that collects all sorts of information about packages in the ros ecosystem. One thing that I'd REALLY like it to do is help people find package forks., but also help people figure out which forks are alive and which are dead. I'd also like to integrate it with the ROS wiki, for which the package listing pages need a huge overhaul. I'm planning on really getting into this discussion (with a prototype system) post-groovy release.

[William Woodall]

On Thu, Sep 20, 2012 at 7:00 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

Sorry, I've been out of the loop, ICRA madness and all...

So one big thing that I remember discussing with Tully back in July, was making the manifest file have a ros-specific name, so that things like webcrawlers could easily find something that looks like a ros package on source servers. For example, instead of "package.xml" just calling it "rospackage.xml" would mean that I could tell if something was trying to be a ROS package just by looking at its files. This will also decrease the likelihood of naming collisions with a package that also wants to have some other file called "package.xml"

-1

While in practice most catkin projects are ROS packages, they do not have to be.  I can, for example, write a sensor interface library using catkin which does not rely on ROS and can be used outside of ROS.  Ideally, Catkin is completely ROS agnostic. Catkin is simply an extension of cmake's capabilities, therefore making its only required file in a repository (package.xml) ROS specific makes no sense to me.

 

On Thu, Sep 20, 2012 at 8:11 AM, tkruse <tibo...@googlemail.com> wrote:

Regarding the 'format' attribute,
<package format='1'>
Either this should be mandatory (and self-explaining like catkin_version='0.6'), or we should drop this. By now I think this should rather be dropped, and structure changes be introduced by tic-toc deprecation.

Yeah, I think it's reasonable for this to be mandatory. That being said "package format=1" could be confusing because it's talking about the package spec and not the package itself. If we change the xml filename to "rospackage.xml" then in the root tag we could do:

<rospackage format='1'>

</rospackage>

+1 (except the renaming)

The only detraction from making it mandatory is that writing a package.xml by hand becomes more difficult, but as long as there are good tools for generating them, I am fine with this, because it opens the door for migration tools and better legacy compatibility.

 

 

+1 to removing the brief, also see http://www.ros.org/browse/list.php which shows how the full description can be cut off 
and still be useful when people give a short and meaningful first sentence.

+1 as well

+1 

-1 to keeping review tag, i think that was agreed upon in the review meeting (http://www.ros.org/wiki/catkin/Reviews/2012-08-14_API_Review)

-1 as well 

0

I don't see this as a required tag for the package.xml or even a good place for it, but I can see the QA crawling use case.

 

-j

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 7:00 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

Sorry, I've been out of the loop, ICRA madness and all...

So one big thing that I remember discussing with Tully back in July, was making the manifest file have a ros-specific name, so that things like webcrawlers could easily find something that looks like a ros package on source servers. For example, instead of "package.xml" just calling it "rospackage.xml" would mean that I could tell if something was trying to be a ROS package just by looking at its files. This will also decrease the likelihood of naming collisions with a package that also wants to have some other file called "package.xml"

On Thu, Sep 20, 2012 at 12:45 PM, William Woodall <wwoo...@willowgarage.com> wrote:

-1

While in practice most catkin projects are ROS packages, they do not have to be.  I can, for example, write a sensor interface library using catkin which does not rely on ROS and can be used outside of ROS.  Ideally, Catkin is completely ROS agnostic. Catkin is simply an extension of cmake's capabilities, therefore making its only required file in a repository (package.xml) ROS specific makes no sense to me.

I understand that perspective, ROS uses Catkin, not the other way around. So then what about "catkin.xml" or "catkinfo.xml" or something of that nature, just to set it aside?

[William Woodall]

On Thu, Sep 20, 2012 at 9:52 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

On Thu, Sep 20, 2012 at 7:00 AM, Jonathan Bohren <jonatha...@gmail.com> wrote:

Sorry, I've been out of the loop, ICRA madness and all...

So one big thing that I remember discussing with Tully back in July, was making the manifest file have a ros-specific name, so that things like webcrawlers could easily find something that looks like a ros package on source servers. For example, instead of "package.xml" just calling it "rospackage.xml" would mean that I could tell if something was trying to be a ROS package just by looking at its files. This will also decrease the likelihood of naming collisions with a package that also wants to have some other file called "package.xml"

On Thu, Sep 20, 2012 at 12:45 PM, William Woodall <wwoo...@willowgarage.com> wrote:

-1

While in practice most catkin projects are ROS packages, they do not have to be.  I can, for example, write a sensor interface library using catkin which does not rely on ROS and can be used outside of ROS.  Ideally, Catkin is completely ROS agnostic. Catkin is simply an extension of cmake's capabilities, therefore making its only required file in a repository (package.xml) ROS specific makes no sense to me.

I understand that perspective, ROS uses Catkin, not the other way around. So then what about "catkin.xml" or "catkinfo.xml" or something of that nature, just to set it aside?

We just had a conversation about this.

(catkin.xml, catkinfo.xml): This is not preferable, because if we ever change from catkin to something else (we don't plan to, but...), this file will be out of place, package.xml is more generic.

manifest.xml: We could have this file called manifest.xml, but it would collide with existing rosbuild manifest.xml and it isn't really a manifest, i.e. it doesn't give a list of the package contents.

project.xml: A good name too, but no real difference from package.xml, more likely to collide

package.xml: A good name, given the context, it is agnostic to the build system and ROS

Ultimately, we decided that package.xml is probably the best name for being generic and for future proofing.

 

-j

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 1:44 PM, William Woodall <wwoo...@willowgarage.com> wrote:

We just had a conversation about this.

(catkin.xml, catkinfo.xml): This is not preferable, because if we ever change from catkin to something else (we don't plan to, but...), this file will be out of place, package.xml is more generic.

Now this seems like it's contradicting your previous point. Before it sounded like this file is totally bound and a part of Catkin, but tell me if I'm inferring correctly that the "we" in your above statement refers to "ROS users" in which case, we're not talking about a filename for catkin, but a filename for ROS. And if this is the case, I re-assert that we should call it something like "rospackage.xml"

-j

[William Woodall]

I would say I meant "we" to be catkin users.  For example, we might choose to change the name of catkin, but not change its function (unlikely).  I still don't believe it should be prefixed with "ros".  package.xml is simply more generic than catkin.xml, as package.xml describes the associated package which is worked on by catkin, but definitely is not ROS exclusive.  Additionally, other tools will likely use this file to gather information, like indexers, or QA scripts, and they will not be called catkin.

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tully Foote]

There are also other projects which have adopted the manifest/stack format and as we clean up our format I would not be surprised if they decide to follow.  Such as autoproj http://www.orocos.org/wiki/orocos/toolchain/using-autoproj (I hope our work is generic and that they can use the same infrastructure.

Tully

Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[Dirk Thomas]

The REP 127 has been updated based on the latest feedback (http://www.ros.org/reps/rep-0127.html).

Summarizing the latest changes:
- made at least on license required (keeping option to have multiple)
- dropped copyright
- dropped brief description
- the mailing list thread is linked as a reference
- clarified "maintainer" for unmaintained packages
- mention what multiple licenses mean
- added plugin export example

- Dirk

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 2:52 PM, William Woodall <wwoo...@willowgarage.com> wrote:

I would say I meant "we" to be catkin users.  For example, we might choose to change the name of catkin, but not change its function (unlikely).

I'd expect this would be _really_ unlikely.

 

I still don't believe it should be prefixed with "ros".  package.xml is simply more generic than catkin.xml, as package.xml describes the associated package which is worked on by catkin, but definitely is not ROS exclusive. 

I understand it's generic, but genericity alone isn't always better.

Additionally, other tools will likely use this file to gather information, like indexers, or QA scripts, and they will not be called catkin.

I've been looking into package discovery and indexers, and it would be VERY useful if this file has some sort machine-readable filename or XML tag in it that makes it obvious that it's a _Catkin_ package file and not some other kind of package. Doing this will require we give it a name that has to do with Catkin or ROS, at least something with some meaning that won't generate too many false-positives.

Personally, I think "catkinfo.xml" has a nice ring to it.

-j

[Dirk Thomas]

As stated by William before: "catkin" (or "ros") do not seem like reasonable parts for the filename.

This file exists to store meta information about the package.
It is not fixed to one specific tool and therefore a "generic" name is preferred.

Tuly has mentioned why the other "generic" filename have not been selected (manifest, project, stack).

For your use case you should obviously not rely on the filename alone.
Independent of which filename is used you need to make sure that the content conforms to the REP before really using it.

- Dirk


On 20.09.2012 15:24, Jonathan Bohren wrote:

> (707) 520-4736 <tel:%28707%29%20520-4736>
> j...@jhu.edu <mailto:j...@jhu.edu>

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 6:35 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

As stated by William before: "catkin" (or "ros") do not seem like reasonable parts for the filename.

This file exists to store meta information about the package.
It is not fixed to one specific tool and therefore a "generic" name is preferred.

I'm sorry, then the text of the REP is really confusing. Specifically the following:

Catkin-only - no rosbuild information inside

The file only specifies information needed for a ROS ecosystem which is entirely based on catkin. Any additional information necessary for backward compatibility with rosbuild is kept separate (i.e. in the legacy manifest.xml files). This enable to easily remove the legacy files in the future.

This makes it sound like:

1. The file only specifies information needed for a ROS ecosystem
2. The file only specifies information needed for a package which is entirely based on catkin
3. Any additional information is kept separate

Is this not the purpose of this file?

 On Thu, Sep 20, 2012 at 6:35 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

 Tuly has mentioned why the other "generic" filename have not been selected (manifest, project, stack).

I'm fine with a generic name, I just want some way to set the package.xml aside that an automated system can depend on. Even an explicit tag like:

<indexing ecosystem="ros" allow="automated"/>

-j

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[Dirk Thomas]

On 20.09.2012 16:40, Jonathan Bohren wrote:

> On Thu, Sep 20, 2012 at 6:35 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dth...@willowgarage.com>> wrote:
>
> As stated by William before: "catkin" (or "ros") do not seem like reasonable parts for the filename.
>
> This file exists to store meta information about the package.
> It is not fixed to one specific tool and therefore a "generic" name is preferred.
>
>
> I'm sorry, then the text of the REP is really confusing. Specifically the following:
>
>

> Catkin-only - no rosbuild information inside <http://www.ros.org/reps/rep-0127.html#id30>

>
> The file only specifies information needed for a ROS ecosystem which is entirely based on catkin. Any additional information necessary for backward compatibility with rosbuild is kept separate
> (i.e. in the legacy manifest.xml files). This enable to easily remove the legacy files in the future.
>
> This makes it sound like:
>

> 1. The file only specifies information needed for a ROS ecosystem
> 2. The file only specifies information needed for a package which is entirely based on catkin
> 3. Any additional information is kept separate

The formulation was intended to make clear that package.xml should not contain anything related to the old buildsystem "rosbuild".
Stuff only necessary for backward compatibility should not be part of the new specification.
I will update that block to make it more clear.

> I'm fine with a generic name, I just want some way to set the package.xml aside that an automated system can depend on. Even an explicit tag like:
>
> <indexing ecosystem="ros" allow="automated"/>

What should that tag be used for?
How would that work in contrast to the current indexing?

- Dirk

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 8:21 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

The formulation was intended to make clear that package.xml should not contain anything related to the old buildsystem "rosbuild".
Stuff only necessary for backward compatibility should not be part of the new specification.
I will update that block to make it more clear.

Cool, I've been out of the loop, so thanks for clearing things up!

<indexing ecosystem="ros" allow="automated"/>

On Thu, Sep 20, 2012 at 8:21 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

What should that tag be used for?
How would that work in contrast to the current indexing?

Well right now the current indexing is very explicit in some ways, and not at all in others. First, it requires people to announce their packages on ROS-users, and for someone at Willow (or OSRF or whoever controls the wiki at that time) to manually add an entry to the indexer. This works when every entry is a unique ROS package, but it fails when it comes to forks of projects or people who don't want to make a formal announcement because of the implications it makes on responsibility. There are a ton of ROS packages and ROS package forks on github and bitbucket, and it's hard for them to get exposure without making an executive (centralized) decision about which one is given priority on the wiki. I think if we want to really make the ros ecosystem analytics scalable and distributed, then we should rely on a more automated method of package discovery.

So this tag would be an optional (but maybe default) tag that declares the indexing permissions for this package, with an additional attribute saying that it's a ROS package so that we don't include anything with the word "ROS" in the schema as per the previous points.

[Jonathan Bohren]

I should mention that I think this deserves a whole other discussion, but I wanted to make sure that minimal features existed in the package.xml file that could support such systems.

[Dirk Thomas]

One option how this could be realized with the current spec:

You create a package "ros_web_indexer".
Then you can use a tag under export with the same name to embed arbitrary information.
<export>
<ros_web_indexer ecosystem="ros" allow="automated"/>
</export>
You indexer can then look for packages on the web and perform arbitrary operations with the meta information found.

Anyway, with the need for maintainers to embed that exta information the mount of packages which will have that defined might be pretty low...

- Dirk

On 20.09.2012 17:49, Jonathan Bohren wrote:
> I should mention that I think this deserves a whole other discussion, but I wanted to make sure that minimal features existed in the package.xml file that could support such systems.
>

> On Thu, Sep 20, 2012 at 8:47 PM, Jonathan Bohren <jonatha...@gmail.com <mailto:jonatha...@gmail.com>> wrote:

>
> On Thu, Sep 20, 2012 at 8:21 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dth...@willowgarage.com>> wrote:
>
> The formulation was intended to make clear that package.xml should not contain anything related to the old buildsystem "rosbuild".
> Stuff only necessary for backward compatibility should not be part of the new specification.
> I will update that block to make it more clear.
>
>
> Cool, I've been out of the loop, so thanks for clearing things up!
>
>
> <indexing ecosystem="ros" allow="automated"/>
>
>
> On Thu, Sep 20, 2012 at 8:21 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dth...@willowgarage.com>> wrote:
>
> What should that tag be used for?
> How would that work in contrast to the current indexing?
>
>
> Well right now the current indexing is very explicit in some ways, and not at all in others. First, it requires people to announce their packages on ROS-users, and for someone at Willow (or OSRF
> or whoever controls the wiki at that time) to manually add an entry to the indexer. This works when every entry is a unique ROS package, but it fails when it comes to forks of projects or people
> who don't want to make a formal announcement because of the implications it makes on responsibility. There are a ton of ROS packages and ROS package forks on github and bitbucket, and it's hard
> for them to get exposure without making an executive (centralized) decision about which one is given priority on the wiki. I think if we want to really make the ros ecosystem analytics scalable
> and distributed, then we should rely on a more automated method of package discovery.
>
> So this tag would be an optional (but maybe default) tag that declares the indexing permissions for this package, with an additional attribute saying that it's a ROS package so that we don't
> include anything with the word "ROS" in the schema as per the previous points.
>
> -j
>
>
> --
> Jonathan Bohren
> PhD Student
> Dynamical Systems and Control Laboratory
> Laboratory for Computational Sensing and Robotics
> The Johns Hopkins University
>

> (707) 520-4736 <tel:%28707%29%20520-4736>
> j...@jhu.edu <mailto:j...@jhu.edu>

>
>
>
>
> --
> Jonathan Bohren
> PhD Student
> Dynamical Systems and Control Laboratory
> Laboratory for Computational Sensing and Robotics
> The Johns Hopkins University
>
> (707) 520-4736

[Jonathan Bohren]

On Thu, Sep 20, 2012 at 9:46 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

One option how this could be realized with the current spec:

You create a package "ros_web_indexer".
Then you can use a tag under export with the same name to embed arbitrary information.
  <export>
    <ros_web_indexer ecosystem="ros" allow="automated"/>
  </export>
You indexer can then look for packages on the web and perform arbitrary operations with the meta information found.

Anyway, with the need for maintainers to embed that exta information the mount of packages which will have that defined might be pretty low...

Right, but I do like this method! What if roscreate-pkg automatically adds this by default?

-j

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[Dirk Thomas]

On 20.09.2012 18:48, Jonathan Bohren wrote:

> On Thu, Sep 20, 2012 at 9:46 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dth...@willowgarage.com>> wrote:
>
> One option how this could be realized with the current spec:
>
> You create a package "ros_web_indexer".
> Then you can use a tag under export with the same name to embed arbitrary information.
> <export>
> <ros_web_indexer ecosystem="ros" allow="automated"/>
> </export>
> You indexer can then look for packages on the web and perform arbitrary operations with the meta information found.
>
> Anyway, with the need for maintainers to embed that exta information the mount of packages which will have that defined might be pretty low...
>
>
> Right, but I do like this method! What if roscreate-pkg automatically adds this by default?

Why bloating all package manifests by default?
If you would like to have opt-out, lets add the tag when the user does NOT want his package being indexed.
(The appropriate line could be included as a comment by roscreate-pkg...)

- Dirk

[Jonathan Bohren]

On Sep 20, 2012 9:57 PM, "Dirk Thomas" <dth...@willowgarage.com> wrote:

On 20.09.2012 18:48, Jonathan Bohren wrote:

On Thu, Sep 20, 2012 at 9:46 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dthomas@willowgarage.com>> wrote:

    One option how this could be realized with the current spec:

    You create a package "ros_web_indexer".
    Then you can use a tag under export with the same name to embed arbitrary information.
       <export>
         <ros_web_indexer ecosystem="ros" allow="automated"/>
       </export>
    You indexer can then look for packages on the web and perform arbitrary operations with the meta information found.

    Anyway, with the need for maintainers to embed that exta information the mount of packages which will have that defined might be pretty low...


Right, but I do like this method! What if roscreate-pkg automatically adds this by default?

Why bloating all package manifests by default?
If you would like to have opt-out, lets add the tag when the user does NOT want his package being indexed.
(The appropriate line could be included as a comment by roscreate-pkg...)

Yeah, an opt-out tag like the way robots.txt works would be good to put in as a commented-out tag, but I'd still like some sort of sentinel tag that lets you declare "this is a ros package" 

-j

[tkruse]

Hi, I have more ideads/concerns:

Concerns:
Why do we move away from using the name of the folder as the package name? (this can cause confusion and breakage)

Why do we explicitly allow xhtml in the description? (This causes technical confusion about the interpretation of the contents)

The justifications belong in the REP.


Ideas:
autoproj defines project types.
The concept of a type allows to reduce the necessary information in a manifest, because all packages of the same type would share that information, and it allows bulk update of many packages by updating the type.
Types could be roscpp-lib, rospy-scripts, rviz-plugin, rosgui-plugin. etc.
I understand that's more overhead to implement, and it makes manifests less self-sustained.

The version of a package is often defined in several locations, CMakelists, Makefiles, source code (for --version), manifest...
Anything that helps reducing the number of places is useful. Maybe a convention fo a file named VERSION in the root of the project that can be parsed even quicker than html might be useful. Then in the package.xml, the default would be not to specify a version, only if no VERSION file exists.

The same argument goes for description. This is also often redundantly defined in a manifest, a README.rst for github, and in the source code (for --help). Again it is useful to rely on just one source where possible.


And the same holds for license information, see http://www.catb.org/esr/Licensing-HOWTO.html
"

By convention, this file should be called LICENSE or COPYING and live in the root directory of your source distribution. It is not necessary to include a copy of the license in every source file, but it is a good idea for the header comment to refer readers to the license file with a comment like this:

This program is open source.  For license terms, see the LICENSE file."


Finally I believe it could be beneficial to have the maintainer information by a username instead of a full name, referencing
some external registry of known mantainers. This allows easier tracking of who maintains what, and of changing that if a
maintainer moves on to new things than ROS. This could also be an attribute of author/maintainer, like rosid='kwc'.

[Dirk Thomas]

On 21.09.2012 05:58, tkruse wrote:
> Hi, I have more ideads/concerns:
>
> Concerns:
> Why do we move away from using the name of the folder as the package name? (this can cause confusion and breakage)

We are not moving away from that.
There is just technically no need for the folder name to be the same as the package name.
The guideline should still strongly suggest to use the same name.

Before it was a must have because a package did not have a name.
So the folder name was used instead.

> Why do we explicitly allow xhtml in the description? (This causes technical confusion about the interpretation of the contents)

That is not change in the spec - this has been the case all the time.

- Dirk

[tkruse]

On Friday, September 21, 2012 7:49:02 PM UTC+2, Dirk Thomas wrote:

On 21.09.2012 05:58, tkruse wrote:
> Hi, I have more ideads/concerns:
>
> Concerns:
> Why do we move away from using the name of the folder as the package name? (this can cause confusion and breakage)

We are not moving away from that.
There is just technically no need for the folder name to be the same as the package name.
The guideline should still strongly suggest to use the same name.

Before it was a must have because a package did not have a name.
So the folder name was used instead.

Yes, the question is: Why do we not just keep that strategy, leaving the name out of the package.xml, and using the folder name, just as we did for manifest.xml and stack.xml?
Is there a good use-case that would justify opening the possibility of users being confused by tool behavior because the folder name and the name declared in package.xml do not match?

> Why do we explicitly allow xhtml in the description? (This causes technical confusion about the interpretation of the contents)

That is not change in the spec - this has been the case all the time.

That's also worth mentioning in the REP.

[Dirk Thomas]

Using the folder name as the package name seems to be the wrong approach.
Let me sketch some examples:

A)
You have a repository containing a single package.
In the case of SVN your URL might look like http://domain/svn/ containing trunk/branches/tags
If you check out trunk the folder is called "trunk" - not "pkgname".
That would be one case where the folder name is not automatically equal to the package name

B)
For the above example it is not even possible to determine the name of the package contained in the SVN repository

C)
You have checked out a package "pkgname".
After some changes you rename the folder to "pkgname-mod" (i.e. to fetch the original package again).
This will fail badly because only the folder name changed - everything inside the package is still using the original name.

- Dirk

[William Woodall]

I agree that this shouldn't be a hard and fast rule, but for C) won't cmake fail anyways if 'pkgname' and 'pkgname-mod' are in the same workspace?

--

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-buildsystem@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Jonathan Bohren]

On Fri, Sep 28, 2012 at 1:22 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

Using the folder name as the package name seems to be the wrong approach.
Let me sketch some examples:

A)
You have a repository containing a single package.
In the case of SVN your URL might look like http://domain/svn/ containing trunk/branches/tags
If you check out trunk the folder is called "trunk" - not "pkgname".
That would be one case where the folder name is not automatically equal to the package name

B)
For the above example it is not even possible to determine the name of the package contained in the SVN repository

C)
You have checked out a package "pkgname".
After some changes you rename the folder to "pkgname-mod" (i.e. to fetch the original package again).
This will fail badly because only the folder name changed - everything inside the package is still using the original name.

I like having the package name in the metadata, especially to support use-case C. As long as roscd still takes a user to the package with the appropriate name, I think would be an improvement.

-j

[Jonathan Bohren]

On Fri, Sep 28, 2012 at 1:25 PM, William Woodall <wwoo...@willowgarage.com> wrote:

I agree that this shouldn't be a hard and fast rule, but for C) won't cmake fail anyways if 'pkgname' and 'pkgname-mod' are in the same workspace?

Maybe it would fail quietly and it will simply build against the package that comes first in the list.

[Dirk Thomas]

On 28.09.2012 10:27, Jonathan Bohren wrote:

> On Fri, Sep 28, 2012 at 1:25 PM, William Woodall <wwoo...@willowgarage.com <mailto:wwoo...@willowgarage.com>> wrote:
>
> I agree that this shouldn't be a hard and fast rule, but for C) won't cmake fail anyways if 'pkgname' and 'pkgname-mod' are in the same workspace?
>
>
> Maybe it would fail quietly and it will simply build against the package that comes first in the list.

That will definitely fail when calling CMake.
But that should be the case since you can not expect that all the CMake code will magically switch to the new package name when the directory is renamed.

- Dirk

[tkruse]

Maybe this had already been discussed for catkin in fuerte, when the stack.xml was changed in catkin to be different from the stack.xml in rosbuild, (I don't think I have seen a discussion online though, or a REP).

I think so far in SVN we used structures/URLs that allowed to infer the name of the stack / package (using svn info, or by looking at the description in the manifest, really) even if you by mistake had checked out something into a folder named "trunk", which is in any case not a very useful name to have in a workspace of multiple projects.

Also it seems currently most projects move towards DVCSes anyway, the limitations of the SVN canonical form should not dictate our workflows.

You introduce the use-case of renaming the folder because you need to free the parent folder namespace for something else of that name (i.e. the original package).
I have no idea how this is a valid usecase in a catkin workspace, having the same package in two versions in the same workspace seems like a difficult situation in any case, because different tools may have different preference orderings of which of the two same packages in different folders are "active". And cmake you agree that this use-case fails anyways, and I assume it will not fail with a nice error message. So this use-case does not seem that important to me.
Personally for this usecase I just move the folder to a different location, rather than renaming it.

Also once the declared name in the package manifest becomes the name authority, then a tool like roscd has to not only crawl directories to search for packages, but also to parse all manifests it finds, assuming we want roscd to work on packagenames, not foldernames.

A compromise could be to cause strict failure for all tools if folder name and declared name do not match, however I don't know if that requirement collides with your use-cases of naming folders "trunk" or "pkgname-mod". But this is always problematic because it leads to confusion as soon as any such tool does not validate, and an inconsistency exists.

The only real backdraws I see is that some repository hosts do not allow underscores in project names (indefero), and that scanning just the manifest does not allow a machine to infer the packagename (humans can use the description usually).

My personal feeling still is that having a declared name in the manifest is a step back, using only the foldername worked well for rosbuild. But I will admit that it is a bit a matter of tastes and personal workflows. If there is no strict technical reason or overwhelming advantage, this could be decided more democratically. In any case I believe the justifications belong in the REP, even if this does not change between catkin_fuerte and catkin_groovy, because it never was justified in docs for fuerte (I believe).

[tkruse]

This thread has gone silent. Still i think that REP 127 is not in a good state yet.

Other than removing the name tag, I also think it would be good to have separate standards for build time and release time.

The creation of a hello_world package should not feel like filling out a tax report. So all the information that is currently mandatory for releasing purposes should in my opinion only be mandatory at release time. This means maintainer, license, ulrs.
I intend to finish catkin_create_pkg to make initial creation of packages easier, but still even then I think a hello_world package should not cause failures  for lack of a missing maintainer email or license. And filling "fixme" or "todo" values in the fields just makes it less likely that at release time people will remember to update those values.
An alternative would be to have separate files and separate standards for releasing, such as a "release.xml" read by bloom, but I cannot tell which alternative is better.

Then I think REP127 could explain more on the ABI attribute (e.g. con one state that one depends on an ABI version?). If the attribute is dead for now, then it might be better not to have it at all for now, until we use it. Adding it later in a backward compatible way should be easy enough.

Also, now that the current REP127 version has materialized in the source, it seems to force to give a lot of redundant information such as:
  <build_depend>catkin</build_depend>
  <build_depend>cpp_common</build_depend>
  <run_depend>cpp_common</run_depend>
That a catkin package depends on catkin for build should be implicit, and having to specify build+run dependencies like cpp_common twice also looks bad to me. As an example, in actionlib.package.xml or roscpp/package.xml, this causes a long list of entries making it difficult to find duplictate, missing or obsolete entries.

Note also that for groovy release it might be nice to actually use the dependency version attributes of the REP, such as generating early failures if one tries to compile incompatible versions of catkin packages. If we find out in the process that this is unsuitable, then REP127 should probably be changed.

Finally the concept of meta-package should be extended in my opinion. If it does not make sense for meta-packages to declare other dependencies than the packages they "contain", then the standard should forbid this (and validate that), meaning the meta-package tag should not be in the exports section and should have an alternative syntax where things are not allowed. E.g. may a meta-package have a CMakeLists, have sources, generate executables, etc.? If not, the REP needs to say so, in my opinion.

[Jonathan Bohren]

On Mon, Oct 15, 2012 at 10:31 AM, tkruse <tibo...@googlemail.com> wrote:

The creation of a hello_world package should not feel like filling out a tax report. So all the information that is currently mandatory for releasing purposes should in my opinion only be mandatory at release time.

+1

[Jack O'Quin]

On Mon, Oct 15, 2012 at 9:31 AM, tkruse <tibo...@googlemail.com> wrote:
> This thread has gone silent. Still i think that REP 127 is not in a good
> state yet.
>
> Other than removing the name tag, I also think it would be good to have
> separate standards for build time and release time.
>
> The creation of a hello_world package should not feel like filling out a tax
> report. So all the information that is currently mandatory for releasing
> purposes should in my opinion only be mandatory at release time. This means
> maintainer, license, ulrs.
> I intend to finish catkin_create_pkg to make initial creation of packages
> easier, but still even then I think a hello_world package should not cause
> failures for lack of a missing maintainer email or license. And filling
> "fixme" or "todo" values in the fields just makes it less likely that at
> release time people will remember to update those values.
> An alternative would be to have separate files and separate standards for
> releasing, such as a "release.xml" read by bloom, but I cannot tell which
> alternative is better.

+1; an excellent idea.

> Then I think REP127 could explain more on the ABI attribute (e.g. con one
> state that one depends on an ABI version?). If the attribute is dead for
> now, then it might be better not to have it at all for now, until we use it.
> Adding it later in a backward compatible way should be easy enough.

+1

I am under the impression that the developers had decided "package
ABI" versions could *not* be implemented in time for Groovy. I agree
that we should not document this idea until the details have been
worked out and someone is ready to implement them.

Normal linker and loader ABI versions are *already* available in
CMake. What is needed for Groovy is clear documentation recommending
their use and showing how to set the SOVERSION in a CMake file. And
please, let's not confuse those two concepts.

> Note also that for groovy release it might be nice to actually use the
> dependency version attributes of the REP, such as generating early failures
> if one tries to compile incompatible versions of catkin packages. If we find
> out in the process that this is unsuitable, then REP127 should probably be
> changed.

+1

> Finally the concept of meta-package should be extended in my opinion. If it
> does not make sense for meta-packages to declare other dependencies than the
> packages they "contain", then the standard should forbid this (and validate
> that), meaning the meta-package tag should not be in the exports section and
> should have an alternative syntax where things are not allowed. E.g. may a
> meta-package have a CMakeLists, have sources, generate executables, etc.? If
> not, the REP needs to say so, in my opinion.

I was under the impression that "meta-packages" would not be supported
in Groovy. Is that wrong?

If not, I recommend removing that concept from REP 127 completely,
until such time as we are ready to actually do it. We can write a new
REP for it, when those requirements are better understood. I am happy
to work on that, but I want to get the timing right.

I don't even like the term "meta-package". I still think we should
call them "stacks".
--
joq

[tkruse]

On Monday, October 15, 2012 9:16:54 PM UTC+2, Jack O'Quin wrote:

I was under the impression that "meta-packages" would not be supported
in Groovy. Is that wrong?

Previous stacks in released Willow Grage stacks have being migrated to meta-packages, so as far as files go, this support exists. As to automatic releasing and generation of debians packages, I would not know, though that is less urgent. For now the wiki consistency is more important, meaning stack packages may remain valid as meta-package pages and wiki links can theoretically be generated.
 

I don't even like the term "meta-package". I still think we should
call them "stacks".

So far nobody gave you +1 one that, and I am -1 for the reasons Tully gave.
You can always make sure both your and Tullys arguments are represented as rationale in the REP, so that we avoid talking in circles.

[Dirk Thomas]

On 15.10.2012 15:11, tkruse wrote:
>
> On Monday, October 15, 2012 9:16:54 PM UTC+2, Jack O'Quin wrote:
>
> I was under the impression that "meta-packages" would not be supported
> in Groovy. Is that wrong?
>
>
> Previous stacks in released Willow Grage stacks have being migrated to meta-packages, so as far as files go, this support exists. As to automatic releasing and generation of debians packages, I would
> not know, though that is less urgent. For now the wiki consistency is more important, meaning stack packages may remain valid as meta-package pages and wiki links can theoretically be generated.

Metapackages are handled not special in the release process but as any normal package.

Bloom releases ROS packages as Debian packages.
If multiple packages are stored in one DVCS they all have the same version number and are released simultaneously.

> I don't even like the term "meta-package". I still think we should
> call them "stacks".
>
>
> So far nobody gave you +1 one that, and I am -1 for the reasons Tully gave.
> You can always make sure both your and Tullys arguments are represented as rationale in the REP, so that we avoid talking in circles.

The term "metapackage" (note the missing hyphen :-) ) has been finalized based on the previous discussion on the mailing list.
As stated in the REP a metapackage contains only a package.xml and nothing else (no CMakeLists.txt, no libraries, binaries or any other files).
Besides the meta information it therefore only models dependencies.
Former wet stacks have already been converted to metapackages.
This keeps BC where stack names are referenced and maintains the relations required for the ROS wiki.

- Dirk

[tkruse]

On Tuesday, October 16, 2012 4:12:08 AM UTC+2, Dirk Thomas wrote:

As stated in the REP a metapackage contains only a package.xml and nothing else (no CMakeLists.txt, no libraries, binaries or any other files).
Besides the meta information it therefore only models dependencies.

Sorry I missed that bit.

In that case, the package.xml should not be allowed to have entries that only make sense if other file were present.
So i think there should not be any of:
buildtool_depend, test_depend, replace and export

License and maintainer are questionable. Since metapackages are not stacks, they cannot replace information from packages, so we'd have a license and maintainer just for the package.xml itself, which seems a bit silly. An alternative is that at release time, license and maintainer information is gathered from the packages.

Whether the packages collected by the metapackage should be run_depend or build_depend tags or a new tag of their own i am not sure. However it might be good to also enforce that only known catkin packages are allowed, not other system libraries (unless that could be used for some useful purpose). Also metapackages without packages or with just one package seem like something that should not be supported, so maybe that should also be checked and lead to errors, at least at release time.

Finally I wonder whether we can get rid of some of the clutter in manifests by having additional convenience meta-packages. So instead of depending on common_msgs, langs, ros, ros_comm, rosconsole and rostime (or some similar set), you could just depend on ros-base, and have all of them (if they usually come together).
This may lead to some unnecessary dependencies in some cases, but the added convenience might outweight that.

[Tully Foote]

On Mon, Oct 15, 2012 at 12:16 PM, Jack O'Quin <jack....@gmail.com> wrote:

On Mon, Oct 15, 2012 at 9:31 AM, tkruse <tibo...@googlemail.com> wrote:
> This thread has gone silent. Still i think that REP 127 is not in a good
> state yet.
>
> Other than removing the name tag, I also think it would be good to have
> separate standards for build time and release time.
>
> The creation of a hello_world package should not feel like filling out a tax
> report. So all the information that is currently mandatory for releasing
> purposes should in my opinion only be mandatory at release time. This means
> maintainer, license, ulrs.
> I intend to finish catkin_create_pkg to make initial creation of packages
> easier, but still even then I think a hello_world package should not cause
> failures  for lack of a missing maintainer email or license. And filling
> "fixme" or "todo" values in the fields just makes it less likely that at
> release time people will remember to update those values.
> An alternative would be to have separate files and separate standards for
> releasing, such as a "release.xml" read by bloom, but I cannot tell which
> alternative is better.

+1; an excellent idea.

I agree that filling in a package.xml should not be "like filling in a tax report"  However as there are only 5 required fields, it's not at all like a tax report ( at least here in the US) We've previously reviewed all those fields on this mailing list. And the consensus on this list was to make these elements required. A couple of them are required for release, but they are also important for documentation and indexing.  The dependencies are verbose, but if you keep them alphabetized they're not terribly hard to manage.  And having been manipulating them recently I believe it's actually easier to have the longer lists to clearly understand what's for build and what's for runtime, there's not an issue with sorting priority of build and run vs build vs run vs the alphabet.  And copy and paste makes adding a dependency only a copy and paste and change a few characters.  

I think that making two standards would make things even more complicated and harder for the user.  

> Then I think REP127 could explain more on the ABI attribute (e.g. con one
> state that one depends on an ABI version?). If the attribute is dead for
> now, then it might be better not to have it at all for now, until we use it.
> Adding it later in a backward compatible way should be easy enough.

+1

+1

We haven't had time to make this happen and adding an attribute to the standard is definitely easy to add later.  

 

I am under the impression that the developers had decided "package
ABI" versions could *not* be implemented in time for Groovy. I agree
that we should not document this idea until the details have been
worked out and someone is ready to implement them.

Normal linker and loader ABI versions are *already* available in
CMake. What is needed for Groovy is clear documentation recommending
their use and showing how to set the SOVERSION in a CMake file. And
please, let's not confuse those two concepts.

> Note also that for groovy release it might be nice to actually use the
> dependency version attributes of the REP, such as generating early failures
> if one tries to compile incompatible versions of catkin packages. If we find
> out in the process that this is unsuitable, then REP127 should probably be
> changed.

+1

We definitely want to use this soon.  The first primary test case for this will be to have catkin depend on a leveled version of catkin_pkg.  

 

> Finally the concept of meta-package should be extended in my opinion. If it
> does not make sense for meta-packages to declare other dependencies than the
> packages they "contain", then the standard should forbid this (and validate
> that), meaning the meta-package tag should not be in the exports section and
> should have an alternative syntax where things are not allowed. E.g. may a
> meta-package have a CMakeLists, have sources, generate executables, etc.? If
> not, the REP needs to say so, in my opinion.

I was under the impression that "meta-packages" would not be supported
in Groovy. Is that wrong?

If not, I recommend removing that concept from REP 127 completely,
until such time as we are ready to actually do it. We can write a new
REP for it, when those requirements are better understood. I am happy
to work on that, but I want to get the timing right.

I don't even like the term "meta-package". I still think we should
call them "stacks".
--
 joq

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[Jack O'Quin]

I misinterpreted Tully's remark as continued discussion. I somehow
missed the fact that it was really a final decision.

I have no desire to waste your time or mine reopening matters that
have already been decided. All I ask is that such messages carry a
fairly explicit "commit" indicator so we can all move on to the
remaining issues.
--
joq

[Jack O'Quin]

On Tue, Oct 16, 2012 at 5:53 AM, tkruse <tibo...@googlemail.com> wrote:
>
> In that case, the package.xml should not be allowed to have entries that
> only make sense if other file were present.
> So i think there should not be any of:
> buildtool_depend, test_depend, replace and export

One metapackage may need to "replace" another, so it should be
allowed. It should even be possible for a metapackage to "replace" a
package, if the tools can support that.

There are legitimate reasons to break an existing package into
separate pieces, such as providing finer granularity of dependencies.
The straightforward way to do that with tick-tock is to introduce a
metapackage with the old name that depends on the newly-created
"sub-packages", providing backwards compatibility while the various
packages depending on it are being updated. All that takes time.

> License and maintainer are questionable. Since metapackages are not stacks,
> they cannot replace information from packages, so we'd have a license and
> maintainer just for the package.xml itself, which seems a bit silly. An
> alternative is that at release time, license and maintainer information is
> gathered from the packages.

Maintainer is important for metapackages that others depend on (see below).

> Also metapackages without packages or with just one package seem like
> something that should not be supported, so maybe that should also be checked
> and lead to errors, at least at release time.

They should be allowed for tick-tock, whenever an old package name is
being phased out for some reason.

> Finally I wonder whether we can get rid of some of the clutter in manifests
> by having additional convenience meta-packages. So instead of depending on
> common_msgs, langs, ros, ros_comm, rosconsole and rostime (or some similar
> set), you could just depend on ros-base, and have all of them (if they
> usually come together).
> This may lead to some unnecessary dependencies in some cases, but the added
> convenience might outweight that.

+1; This is why metapackages need maintainers. Otherwise, no one else
can depend on them to provide stable contents and semantics.
--
joq

[Dirk Thomas]

That's why a metapackage is nothing special.

It is just a normal package with the "metapackage" flag in the exports section and without a CMakeLists.txt file.
The only special treatment is that no CMakeLists.txt is included when being processed by catkin.

So all the features and meta information available are the same as for packages, i.e.:
- allowing conflicting/replacing packages
- listing maintainers and url

- Dirk

[Jonathan Bohren]

On Tue, Oct 16, 2012 at 5:41 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

That's why a metapackage is nothing special.

It is just a normal package with the "metapackage" flag in the exports section and without a CMakeLists.txt file.
The only special treatment is that no CMakeLists.txt is included when being processed by catkin.

So all the features and meta information available are the same as for packages, i.e.:
- allowing conflicting/replacing packages
- listing maintainers and url

One apparent difference between stacks and metapackages is that stacks can be checked out from a VCS with their constituent packages, and metapackages can only be used for creating a distribution-level dependency. Is this correct?

-j

[Dirk Thomas]

On 16.10.2012 14:53, Jonathan Bohren wrote:

Yes: metapackages are used for distribution-level dependencies.
No: the options for checking out code have not changed.

Since currently all metapackages are usually former stacks (to keep BC and correlation of Wiki pages) they live in the same repository as the packages of that former stack.
So nothing has changed and you can still checkout the repo as a whole and get all the packages (including the metapackage).

- Dirk

[William Woodall]

On Tue, Oct 16, 2012 at 4:32 PM, Dirk Thomas <dth...@willowgarage.com> wrote:

On 16.10.2012 14:53, Jonathan Bohren wrote:

On Tue, Oct 16, 2012 at 5:41 PM, Dirk Thomas <dth...@willowgarage.com <mailto:dthomas@willowgarage.com>> wrote:

    That's why a metapackage is nothing special.

    It is just a normal package with the "metapackage" flag in the exports section and without a CMakeLists.txt file.
    The only special treatment is that no CMakeLists.txt is included when being processed by catkin.

    So all the features and meta information available are the same as for packages, i.e.:
    - allowing conflicting/replacing packages
    - listing maintainers and url


One apparent difference between stacks and metapackages is that stacks can be checked out from a VCS with their constituent packages, and metapackages can only be used for creating a
distribution-level dependency. Is this correct?

Yes: metapackages are used for distribution-level dependencies.
No: the options for checking out code have not changed.

Since currently all metapackages are usually former stacks (to keep BC and correlation of Wiki pages) they live in the same repository as the packages of that former stack.
So nothing has changed and you can still checkout the repo as a whole and get all the packages (including the metapackage).

Metapackages can, however, be in a different repository from its "packages", which was not possible before with stacks, i.e. all packages of a stack had to be in the same folder and there for the same repository.

Metapackages can also have different version numbers from the packages it depends on, but I should note you are not allowed to release packages of different versions from the same repository.

 

- Dirk

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To post to this group, send email to ros-sig-buildsystem@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--