Changes to $(find ...)

[Jonathan Bohren]

Just found out about this:

https://github.com/ros/ros_comm/pull/228

-j

[Thibault Kruse]

Hi all,

so to summarize the discussion. Stepping back a bit, this is about ROS packages sharing files. Meaning ROS package B requires some file that is provided by ROS package A.

There are standard procedures for
- c header / libraries
- python modules

However other files to provide / consume are
- executables / scripts
- ROS message definitions (.msg)
- urdfs
- roslaunch files
- image files
- anyother file may think of

Additionally with cakin (and rosbuild), we want packages A to provide such files to packages B even before package A has been installed, though after having been build (as opposed to how the rest of the world creates software). With rosbuild, a straightforwads (and dirty)( way was to just use package lookup and the relative path from the package root to the desired file. This is dirty because it establishes a dependency to the internal file structure of pakcga A that package B should not know about (Else anytime the maintainer ofpackage A decides to change the internal package structure not knowin about the existence of packages B, all packages B that relied on some relative path break).
With catkin, this does not work in a straightforward way, because packaging does not replicate the source folder structure, but instead does whatever the CMakeLists.txt decides to install, ideally some FHS compliant layout, but in reality whatever the maintainer fancied.

So when looked at it like that, this is not a roslaunch-specific problem, but it can be solved for each filetype separately, and the discussion on github was caused by an issue that arose because catkin was introduced without taking into account existing launch files using the $(find ...) shortcut to locate files, which works for the catkin devel space, but does not easily map to FHS layouts.

So to solve this, several design decisions have to be taken. Here those are in some arbitrary order:

1. Should packages B be allowed random access to files in packages A at all, as opposed to restricted access to files that are somehow marked as public?
If we decide for yes, then packagename/relativepath would remain the key to locate a file in source, and the question is how that would map to the install layout

2. Should packages A be allowed to install files wherever they want?
This is not just about FHS layout, but also about packages A being free to rename subfolders/files or extend/reduce the subfolder structure. Since we rely on such a poorly designed build language as cmake (not offering much in the way of reasoning about the build definition), there is little possibility to ever make sure this happens, so there will be breakage if we try to enforce this, but it might be bearable.

3. Should packages B be allowed to assume to know the install layout of packages A
This would mean all packages A that provide files would need to follow strict conventions on where to install files, such that packages B can reliably refer to files in both source and install layout.
If we decide against that, then packages A decide where files get installed to, and thus another mapping mechanism would be required.

4. Do we want a generic solution for all filetypes or specific solutions for executables, .msg files, .action files, urdf files, launch files, etc. each?

Just for brainstroming purposes, here are some possibilities for generic solutions with:

Mapping Alternative: Packages A that want to provide files need to generate a map of public files via a cmake macro, the map maps keys to path, such that packages B can resolve paths from keys. This mapping file would be generated both during build and install, and go into the catkin develspace /installspace respectively.

Public subfolder alternative: Packages A may have a subfolder "public" within which they may place any (non-achitecture-dependent) files for public consumption, and packages B may refer to any files in that folder. All the contents of this folder are also installed into the FHS layout under a fixed location under .../share/packagename[/public], and a find mechanism resolves relatives paths know where to look for them in both install and source layouts. For architecture dependent files the same happens but the files are installed to .../lib/... and packages B decide what kind of resource they want.

Public packageroot alternative: Packages B may have arbitrary access to any files in the source folders of packages A, but really should be careful to only use files that seem like they are kind of intended to be public, and maintainers of packages A should install all such files to the same relative paths in the FHS layout ounder share or lib.

A special problem is backwards compatibility, because package authors in the past have put executables foo in diverse places such as packageroot/bin/foo, packageroot/scripts/foo, packageroot/nodes/foo, yet with catkin those executables all go into .../lib/packagename/foo

Due to this, the initially suggested solution in the github issue was to just strip the relative path when looking up executables in the FHS layout, but keep it when looking up an executable in the develspace/sourcespace. That however is intransparent, inconsistent, and generally dirty. As an example, this would seem as if the path was part of the public API of packages A for sharing executables foo. However the path is nt really part of the ApI, because if it was, it would be possible for a package to provide  executables v.10/foo and v2.0/foo at the same time, but due to the stripping of the path this would fail one way or the other.

The other problem is that the ROS ecosystem provides the rosrun tool which crawls packages for executables, such that "rosrun A foo" will run the executable foo in package a wherever it is located (and interactively ask for disambiguation when more than one such file is found). It might be nice for tool consistency if such an API was usable accross many tools, but concerns have been raised that crawling package folders may have very large performance impact as packages grow. If that were so, rosrun should rather be deprecated and replaced by a non-crawling solution, or a solution that only crawls specific locations (e.g. only bin, script and nodes subfolders) which have litttle risk of ever becoming large.


So to me it is not clear whether this is an issue for the roslaunch SIG at all, since it mostly relates to issues caused by the introduction of FHS layouts for ROS packages with catkin.