Rosdep and ROS2

[Brad Baillio]

Does rosdep still work on the ROS2 package.xml variant?

Is this still going to be the strategy moving forward for resolving system dependencies?

Thanks.

[William Woodall]

`rosdep` can parse the `package.xml` we use in ROS 2, which is just `package.xml` "format 2" as described in REP-0140: http://www.ros.org/reps/rep-0140.html

`rosdep` will always be an option. However, we haven't been tackling the automatic dependency installation issue yet ROS 2. We've just been listing the dependencies in the installation instructions, mostly because the list is short and we have to do so anyways for Windows.

A while back (quite a while back now actually), Nikolaus Demmel did a GSoC (or an internship) with me at OSRF and he did pretty much all of the design work and almost all the implementation work on a new tool called `xylem`: http://xylem.readthedocs.io/en/latest/

It was meant to do two things: fix the conceptual issues with `rosdep`'s database format and make it ROS agnostic.

It's a really great design, but I haven't had the time, or the need, to revive the work yet. It was always my intention to use it for ROS 2, but it would be more expedient to pick up `rosdep` and use that.

Cheers,

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

William Woodall

ROS Development Team

wil...@osrfoundation.org

http://wjwwood.io/

[Asmodehn Shade]

Hi,

Oh Thank you for mentioning xylem, I wasn't aware of it and it looks very promising.

The main feature that would interest me in there (if I understood the design docs properly) is the ability to manage pip packages.

That would mean that we do not have to build a deb for a ros package and a deb for each of its dependencies, right ?

This is the main drawback to rosdep currently from my perspective, as it forces python coders to fork the original code and maintain a separate version of _each_ of their dependencies.

I agree completely with the goal of a ROS independent meta package manager and I might jump in in some of the discussions there.

Thanks again !

[William Woodall]

On Mon, May 23, 2016 at 5:55 PM, Asmodehn Shade <asmo...@gmail.com> wrote:

Hi,

Oh Thank you for mentioning xylem, I wasn't aware of it and it looks very promising.

The main feature that would interest me in there (if I understood the design docs properly) is the ability to manage pip packages.

It will offer no distinct advantage in managing pip over rosdep AFAIK.

That would mean that we do not have to build a deb for a ros package and a deb for each of its dependencies, right ?

No, that's a limitation of apt. No apt package (.deb) can depend implicitly on something that is not provided by another `.deb`.

 

This is the main drawback to rosdep currently from my perspective, as it forces python coders to fork the original code and maintain a separate version of _each_ of their dependencies.

Sorry, I don't follow what you mean here. Python packages which are provided by apt, e.g. python-yaml, require no extra action from the Python developer.

 

I agree completely with the goal of a ROS independent meta package manager and I might jump in in some of the discussions there.

Sorry if I got your hopes up about a different path for packaging Python code alongside non-Python packages in apt. But I still think Xylem is awesome either way. It does, however, explicitly mention:

From http://xylem.readthedocs.io/en/latest/design.html#anti-goals:

xylem does not aim to replace package managers or package software itself. While support for package-manager-less platforms can be achieved with installer plugins such as the source installer, it is not an objective of xylem to systematically maintain such installation scripts.

Which I think covers your connection to an alternative way to depend on custom Python packages from apt packages.

[Asmodehn Shade]

Thanks for the reply.
Maybe I got my hopes up too fast, and I didn't read it all in details yet, but I will try to be a bit more detailed about my understanding so far, answers inline...

On May 24, 2016 10:22 AM, "William Woodall" <wil...@osrfoundation.org> wrote:
>
> On Mon, May 23, 2016 at 5:55 PM, Asmodehn Shade <asmo...@gmail.com> wrote:
>>
>> Hi,
>>
>> Oh Thank you for mentioning xylem, I wasn't aware of it and it looks very promising.
>>
>> The main feature that would interest me in there (if I understood the design docs properly) is the ability to manage pip packages.
>
> It will offer no distinct advantage in managing pip over rosdep AFAIK.

Rosdep is not really a package manager, but more like a "dependency manager":
- rosdep user cannot install a ROS package ( only the dependencies of a package )
- rosdep user cannot remove a ROS package

I see rosdep more as an extra layer on top of the system package manager (apt), with a tentative approach to pip.

Since xylem aims to be an interface to many package managers, my understanding is that it will improve this situation (ref http://xylem.readthedocs.io/en/latest/design.html#commands)
We might need that remove command as well in xylem though...

Also managing package versions is an important part of a package manager, so I am glad xylem plans to address this : http://xylem.readthedocs.io/en/latest/design.html#versions-in-rules-files

So that makes already xylem awesome.
I just hope the awesomeness goes further :-)

>>
>> That would mean that we do not have to build a deb for a ros package and a deb for each of its dependencies, right ?
>
>
> No, that's a limitation of apt. No apt package (.deb) can depend implicitly on something that is not provided by another `.deb`.

ROS software is distributed via deb. Rosdep manages dependencies in deb.

But xylem promise to interface with multiple package manager.
From then I assume it means we can have some ROS software from pip, and some ROS software from deb, for example ?

If that is true it means that, when building a python (pip) package that has python (pip) dependencies, integrating them all in the ROS ecosystem, would be much simpler than currently.

Is that "ROS packages from pip" a correct assumption ? Or maybe that's where I got the wrong idea...

>  
>>
>> This is the main drawback to rosdep currently from my perspective, as it forces python coders to fork the original code and maintain a separate version of _each_ of their dependencies.
>
>
> Sorry, I don't follow what you mean here. Python packages which are provided by apt, e.g. python-yaml, require no extra action from the Python developer.

"Which are provided by apt" is a big condition :
Pypi packages : 81191

apt-cache search python | wc -l
4080

>  
>>
>> I agree completely with the goal of a ROS independent meta package manager and I might jump in in some of the discussions there.
>
>
> Sorry if I got your hopes up about a different path for packaging Python code alongside non-Python packages in apt. But I still think Xylem is awesome either way. It does, however, explicitly mention:
>
> From http://xylem.readthedocs.io/en/latest/design.html#anti-goals:
>
>> xylem does not aim to replace package managers or package software itself. While support for package-manager-less platforms can be achieved with installer plugins such as the source installer, it is not an objective of xylem to systematically maintain such installation scripts.
>
>
> Which I think covers your connection to an alternative way to depend on custom Python packages from apt packages.

I m actually more thinking the other way around :
- many ROS packages are written in python
- These packages could be released as pip packages
- xylem (bloom ?) would provide a way for these packages to depend on apt packages (from ROS or system).

Ideally I would like a ROS package to depend on some pip package and some deb package ( what rosdep is trying/tried to do, but not really working as far as I know...)
And both would be in the ROS ecosystem, one downloadable via xylem-pip plugin, the other one via xylem-apt plugin.
And I could build this package with bloom and release it, to have a complete ROS package release pipeline.

As a ROS developer, In the same way that now I can have a ROS package depending on another ROS package or a system package (via rosdep keys in package.xml), I could make a ROS package that depends on both a pip package and a deb package, via "rosdep->xylem" keys in the package.xml.

The trick here is what to allow/forbid in package dependencies provided by different package managers, so that we can guarantee that whatever works in source also works from installed (built) packages...

Thanks for letting me know what part is actually planned, and what part I am just dreaming about.

Cheers !

[William Woodall]

On Tue, May 24, 2016 at 6:43 PM, Asmodehn Shade <asmo...@gmail.com> wrote:

Thanks for the reply.
Maybe I got my hopes up too fast, and I didn't read it all in details yet, but I will try to be a bit more detailed about my understanding so far, answers inline...

On May 24, 2016 10:22 AM, "William Woodall" <wil...@osrfoundation.org> wrote:
>
> On Mon, May 23, 2016 at 5:55 PM, Asmodehn Shade <asmo...@gmail.com> wrote:
>>
>> Hi,
>>
>> Oh Thank you for mentioning xylem, I wasn't aware of it and it looks very promising.
>>
>> The main feature that would interest me in there (if I understood the design docs properly) is the ability to manage pip packages.
>
> It will offer no distinct advantage in managing pip over rosdep AFAIK.

Rosdep is not really a package manager, but more like a "dependency manager":
- rosdep user cannot install a ROS package ( only the dependencies of a package )
- rosdep user cannot remove a ROS package

I would characterize it as a package manager abstraction. But then again, I'm not entirely sure what qualifies something to be a "dependency manager".

 

I see rosdep more as an extra layer on top of the system package manager (apt), with a tentative approach to pip.

I do not believe rosdep provides any extra functionality for apt that it does not provide for pip. 

Since xylem aims to be an interface to many package managers, my understanding is that it will improve this situation (ref http://xylem.readthedocs.io/en/latest/design.html#commands)
We might need that remove command as well in xylem though...

Also managing package versions is an important part of a package manager, so I am glad xylem plans to address this : http://xylem.readthedocs.io/en/latest/design.html#versions-in-rules-files

So that makes already xylem awesome.
I just hope the awesomeness goes further :-)

>>
>> That would mean that we do not have to build a deb for a ros package and a deb for each of its dependencies, right ?
>
>
> No, that's a limitation of apt. No apt package (.deb) can depend implicitly on something that is not provided by another `.deb`.

ROS software is distributed via deb. Rosdep manages dependencies in deb.

But xylem promise to interface with multiple package manager.
From then I assume it means we can have some ROS software from pip, and some ROS software from deb, for example ?

No, unfortunately not. When installing ROS from apt, you just do one call to apt, e.g. `sudo apt-get install ros-indigo-desktop`. There is no invocation of rosdep. For xylem that would not change. apt packages (.deb files) are responsible for installing all of their dependencies directly, and since they cannot depend on anything that is not also a `.deb` file, you cannot have a dependency which is distributed only by pip. rosdep is not involved at this point and neither would xylem.

 

If that is true it means that, when building a python (pip) package that has python (pip) dependencies, integrating them all in the ROS ecosystem, would be much simpler than currently.

Again, I do not believe there is anything about xylem that would improve the current situation with respect to distributing packages via apt which depend on something distributed via pip. You can already have a package which depends on something from apt and pip, and you can get rosdep to install those dependencies, but you cannot generate a `.deb` file for that package because all of its dependencies are not available as `.deb` files via apt. 

Is that "ROS packages from pip" a correct assumption ? Or maybe that's where I got the wrong idea...

>  
>>
>> This is the main drawback to rosdep currently from my perspective, as it forces python coders to fork the original code and maintain a separate version of _each_ of their dependencies.
>
>
> Sorry, I don't follow what you mean here. Python packages which are provided by apt, e.g. python-yaml, require no extra action from the Python developer.

"Which are provided by apt" is a big condition :
Pypi packages : 81191

apt-cache search python | wc -l
4080

I agree, but that's out of our hands. A more common problem is that the versions provided in apt are typically much older than those in pip. Again, xylem or rosdep has no control over this.

>  
>>
>> I agree completely with the goal of a ROS independent meta package manager and I might jump in in some of the discussions there.
>
>
> Sorry if I got your hopes up about a different path for packaging Python code alongside non-Python packages in apt. But I still think Xylem is awesome either way. It does, however, explicitly mention:
>
> From http://xylem.readthedocs.io/en/latest/design.html#anti-goals:
>
>> xylem does not aim to replace package managers or package software itself. While support for package-manager-less platforms can be achieved with installer plugins such as the source installer, it is not an objective of xylem to systematically maintain such installation scripts.
>
>
> Which I think covers your connection to an alternative way to depend on custom Python packages from apt packages.

I m actually more thinking the other way around :
- many ROS packages are written in python
- These packages could be released as pip packages
- xylem (bloom ?) would provide a way for these packages to depend on apt packages (from ROS or system).

That's not the job of xylem. That would be the job of the package manager. You can already have the described situation for source packages, just not binary packages for apt.

Ideally I would like a ROS package to depend on some pip package and some deb package ( what rosdep is trying/tried to do, but not really working as far as I know...)
And both would be in the ROS ecosystem, one downloadable via xylem-pip plugin, the other one via xylem-apt plugin.
And I could build this package with bloom and release it, to have a complete ROS package release pipeline.

Again, this assumes that installation involves the user calling xylem at all. Currently, the user only invokes apt. That is the issue, full stop. 

As a ROS developer, In the same way that now I can have a ROS package depending on another ROS package or a system package (via rosdep keys in package.xml), I could make a ROS package that depends on both a pip package and a deb package, via "rosdep->xylem" keys in the package.xml.

The necessary functionality here is that packages that are in pip would need to have `.deb` files generated for them. We generate `.deb` files for our Python packages, but the process is not very easy and fairly fragile. For example, you can install catkin_pkg on Ubuntu with either `sudo apt-get install python-catkin-pkg` or `sudo -H python -m pip install -U catkin_pkg`. Both will work and that is the _only_ reason we can depend on catkin_pkg from ROS packages and then create `.deb` files for those ROS packages (i.e. release them).

The trick here is what to allow/forbid in package dependencies provided by different package managers, so that we can guarantee that whatever works in source also works from installed (built) packages...

Thanks for letting me know what part is actually planned, and what part I am just dreaming about.

Sorry to poke holes in your email. I don't mean to be negative :D, but I think the problem you want to tackle is orthogonal to what Xylem is trying to improve.

Thanks, 

Cheers !

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Asmodehn Shade]

Ah I see... I agree about the orthogonality of the issue I was thinking about compared to xylem goal.

I am by now used to work from source in ROS with python, but I m still struggling to tackle the pip dependencies problem in binary packages...

I ll have to find another solution to that, so that my source and binary packages can use the same dependencies...

Thanks for taking the time to reply in details!

--
AlexV