python script catkin install

[tkruse]

Hi,

actually the question raised by Tommy Perssson on ros-users stumped me. If we use setup.py with catkin to declare scripts, they end up in the PATH.
As an example, tf/view_frames was only runnable using rosrun in fuerte, in groovy it has been promoted to a global binary, despite the rather nondescript name.

Is this by intention, or an oversight?

cheers,
  Thibault

[Jack O'Quin]

It does seem unfortunate to me. 

--
 joq

[Tully Foote]

This is an oversight.  It should be installed package local.  

Please open a ticket for that. 

Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[Jonathan Bohren]

Just view_frames should be installed package local, or anything in the setup.py 'scripts' element should be installed package local?

-j

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

[Tully Foote]

In general executables should be installed package local, just like in the past. Only the few things that we previously promoted to $ROS_ROOT/bin should be on the $PATH like before.  

Tully

[Jonathan Bohren]

On Sun, Feb 3, 2013 at 4:38 PM, Tully Foote <tfo...@willowgarage.com> wrote:

In general executables should be installed package local, just like in the past. Only the few things that we previously promoted to $ROS_ROOT/bin should be on the $PATH like before.  

Could that cause confusion, since modules in the setup.py are installed to the install target lib/python... path?

-j 

[William Woodall]

The script tf/view_frames should not be in the path, that is the bug.  Anything listed in scripts of the setup.py will be put in the path, which is consistent with conventional Python development.  If you want to install a python script and not have it on the path it should be absent from the scripts section of the setup.py and should in stead be installed some place using a CMake install rule.

On Sun, Feb 3, 2013 at 1:41 PM, Jonathan Bohren <jonatha...@gmail.com> wrote:

On Sun, Feb 3, 2013 at 4:38 PM, Tully Foote <tfo...@willowgarage.com> wrote:

In general executables should be installed package local, just like in the past. Only the few things that we previously promoted to $ROS_ROOT/bin should be on the $PATH like before.  

Could that cause confusion, since modules in the setup.py are installed to the install target lib/python... path?

Again you would expect, from a conventional python development standpoint, for things listed in modules to be on the PYTHONPATH after installation.  So I would say it is correct, expected behavior.

 

-j 

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
William Woodall

Willow Garage - Software Engineer

wwoo...@willowgarage.com

[tkruse]

TF ticket here:
https://code.ros.org/trac/ros-pkg/ticket/5610
But comparing /opt/ros/fuerte/bin  to /opt/ros/groovy/bin, I guess there may be some more candidates.

that also means the catkin documentation and migration guide have to be adapted. They now should tell python developers that they should indeed use setup.py just like for normal python development, but should really stay away from using the scripts argument in there to deploy scripts, in total contrast to normal python development. This also means crippling the setup.py even more (e.g. making it even harder to use it to deploy to pypi as well).

I guess many users may get this wrong.

There are alternatives of course, that have already been discussed for other reasons. Writing a cmake macro that generates a setup.py is one solution. The other would be to wrap the call to setup() in setup.py, by somethig like catkin_setup(). Such a wrapper function could allow users to give both global as well as non-global scripts, and it would be under our control where stuff goes, transparent to the user.

Previous discussions here:
https://groups.google.com/d/topic/ros-sig-buildsystem/wtYhPumot0A/discussion
https://github.com/ros-infrastructure/catkin_pkg/issues/20

On Sunday, February 3, 2013 11:26:14 PM UTC+1, William Woodall wrote:

The script tf/view_frames should not be in the path, that is the bug.  Anything listed in scripts of the setup.py will be put in the path, which is consistent with conventional Python development.  If you want to install a python script and not have it on the path it should be absent from the scripts section of the setup.py and should in stead be installed some place using a CMake install rule.

On Sun, Feb 3, 2013 at 1:41 PM, Jonathan Bohren <jonatha...@gmail.com> wrote:

On Sun, Feb 3, 2013 at 4:38 PM, Tully Foote <tfo...@willowgarage.com> wrote:

In general executables should be installed package local, just like in the past. Only the few things that we previously promoted to $ROS_ROOT/bin should be on the $PATH like before.  

Could that cause confusion, since modules in the setup.py are installed to the install target lib/python... path?

Again you would expect, from a conventional python development standpoint, for things listed in modules to be on the PYTHONPATH after installation.  So I would say it is correct, expected behavior.

 

-j 

--

Jonathan Bohren

PhD Student

Dynamical Systems and Control Laboratory

Laboratory for Computational Sensing and Robotics

The Johns Hopkins University

(707) 520-4736

j...@jhu.edu

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsystem+unsub...@googlegroups.com.

To post to this group, send email to ros-sig-b...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

Catkin could be extended to accept this setup invocation though:

setup(
  version='0.0.1',
  # scripts=['scripts/foo'],
  data_files=[('lib/foo', ['scripts/foo'])],
  package_dir={'': 'src'})

Funnily, this even works for this special case anyway right now. This makes script foo runnable even "in" the develspace, since we also look in the source space.
However for consistency's sake, I guess catkin should then really support the data_files element, which probably means copying over resources to the develspace.
Suggested here. https://github.com/ros/catkin/issues/345

[tkruse]

And while we are at it, another thing I found:

In rostopic, the rostopic script is both deployed using the setup.py method to global bin, as well as using the install macro to the non-global bin folder.
Is that also an oversight, or by design to keep the rostopic script executable with rosrun? Because in the global bin folder, executables are not found using rosrun.

Also, why do python scripts go into lib, not share? I though lib was for architecture-dependent stuff only. Not sure whether that relates to python3 compatibility. But the layout of scripts going into lib/packagename/ does not seem to allow providing separate python2 and python3 compatible scripts of the same name.

[tkruse]

On Monday, February 4, 2013 1:38:19 PM UTC+1, tkruse wrote:

Also, why do python scripts go into lib, not share? I though lib was for architecture-dependent stuff only. Not sure whether that relates to python3 compatibility. But the layout of scripts going into lib/packagename/ does not seem to allow providing separate python2 and python3 compatible scripts of the same name.

It seems the web community is also divided on that isse. Since the FHS says /share is for "data" only, some people interpret this meaning "non-executables".
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=134762
Others say lib is for architecture dependent-data only, so share is the right place for scripts that have no place in bin:
http://webapps-common.alioth.debian.org/draft/html/ch-issues.html

I guess putting shareable files under /lib is less harmful than putting non-shareable files under /share. So I'm not saying this needs changing, but it should be mentioned in REP122 i believe, since FHS is ambiguous.

[William Woodall]

On some systems libexec is used to store these executables, but since Ubuntu doesn't support libexec they go into lib in stead.

 

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.

To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/k_zAmDU3B4QJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[William Woodall]

There is no reason for users to shy away from using the Python setup.py scripts section when deploying scripts, they just need to be aware that they are doing this.  The whole idea of putting these executables somewhere not on the path and using rosrun to execute them is a purely ROS concept, and has nothing to do with catkin.  If you want to preserve the behavior of a converted ROS stack or install scripts without polluting the PATH then use either the data_files entry within the setup.py or a CMake install rule to install it.  The only thing I am not sure about is how data_files handles permissions, there might be some special logic in scripts to ensure the installed file is executable.

I think mirroring data_files in the devel space might be a reasonable thing to do.

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.

To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/CnpyadABMtkJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

On Monday, February 4, 2013 8:51:22 PM UTC+1, William Woodall wrote:

There is no reason for users to shy away from using the Python setup.py scripts section when deploying scripts, they just need to be aware that they are doing this.  The whole idea of putting these executables somewhere not on the path and using rosrun to execute them is a purely ROS concept, and has nothing to do with catkin.  

Then why would catkin provide a CATKIN_PACKAGE_BIN_DESTINATION ? Shouldn't that then be a ROS_PACKAGE_BIN_DESTINATION ?

If you want to preserve the behavior of a converted ROS stack or install scripts without polluting the PATH then use either the data_files entry within the setup.py or a CMake install rule to install it.

The cmake install rule does not put things into the devel space. I know that's not an issue for rosrun, but it's not clean either.

Is there maybe some value in recommending just one approach? Which should go into out documentation and tutorials?
 

 The only thing I am not sure about is how data_files handles permissions, there might be some special logic in scripts to ensure the installed file is executable.

 

distutils also manipulates the hashbang line when installing. Not sure how much that will trouble us, in particular with python3 in ringtail.

[William Woodall]

On Mon, Feb 4, 2013 at 11:58 AM, tkruse <tibo...@googlemail.com> wrote:

On Monday, February 4, 2013 8:51:22 PM UTC+1, William Woodall wrote:

There is no reason for users to shy away from using the Python setup.py scripts section when deploying scripts, they just need to be aware that they are doing this.  The whole idea of putting these executables somewhere not on the path and using rosrun to execute them is a purely ROS concept, and has nothing to do with catkin.  

Then why would catkin provide a CATKIN_PACKAGE_BIN_DESTINATION ? Shouldn't that then be a ROS_PACKAGE_BIN_DESTINATION ?

This is a use variable for being FHS compliant.  This is where you should put executables which don't belong on the path.  For example, you might have an executable which should be discovered using pkg-config or the location is templated into another script or config file at build time.  There might be room for discussion on the name of this variable, but it has value outside of the ROS scope.

 

If you want to preserve the behavior of a converted ROS stack or install scripts without polluting the PATH then use either the data_files entry within the setup.py or a CMake install rule to install it.

The cmake install rule does not put things into the devel space. I know that's not an issue for rosrun, but it's not clean either.

Is there maybe some value in recommending just one approach? Which should go into out documentation and tutorials?

I would agree there should be one, clear way to do this, I would be inclined to say the CMake method, but perhaps not.  I don't think it should be in the devel space because we generally only put generated files in the devel space, though a redirection placeholder might work.  If you install a script to the devel folder then you loose the benefit of being able to source the devel space, modify the script in source, and test it without re-running make.

 

 

 The only thing I am not sure about is how data_files handles permissions, there might be some special logic in scripts to ensure the installed file is executable.

 

distutils also manipulates the hashbang line when installing. Not sure how much that will trouble us, in particular with python3 in ringtail.

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/HrGbKuWD7d0J.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jack O'Quin]

Longstanding Debian practice is to use /usr/lib/pythonX.Y/dist-packages. I don't know why that is, or what interpretation of the FHS it implies. 

There may be several factors:

 * The original definition of /usr/share/ was for files that could be shared read-only across multiple machines in an NFS network. Even though Python *.pyc files are architecture-independent, they are not Python-version-independent. So, they can't be shared across a network of dissimilar systems with different Python versions.

 * Debian systems support multiple Python versions concurrently on a single system.

 * Some Python modules are implemented in C or C++, so they are not architecture-independent. It is probably more convenient to search for all the dist-packages for a particular Python version in one place. 

 * Some Python-oriented distribution methods install source files in /usr/share, but "compile" them to .pyc files for all relevant Python versions into /usr/lib/pythonX.Y/dist-packages.

--
 joq

[tkruse]

On Monday, February 4, 2013 9:49:20 PM UTC+1, William Woodall wrote:

On Mon, Feb 4, 2013 at 11:58 AM, tkruse <tibo...@googlemail.com> wrote:

On Monday, February 4, 2013 8:51:22 PM UTC+1, William Woodall wrote:

There is no reason for users to shy away from using the Python setup.py scripts section when deploying scripts, they just need to be aware that they are doing this.  The whole idea of putting these executables somewhere not on the path and using rosrun to execute them is a purely ROS concept, and has nothing to do with catkin.  

Then why would catkin provide a CATKIN_PACKAGE_BIN_DESTINATION ? Shouldn't that then be a ROS_PACKAGE_BIN_DESTINATION ?

This is a use variable for being FHS compliant.  This is where you should put executables which don't belong on the path.

All I am saying is that catkin could support this better for setup.py as well, the same way it does for cmake. Else, users could also be left to build up target destinations in cmake clumsily. distutils does support moving files anywhere, the same way cmake does. But for cmake apparently someone thought it wise to provide users with CATKIN_PACKAGE_BIN_DESTINATION, so why not provide something equivalent for setup.py?


[tkruse]

On Tuesday, February 5, 2013 1:37:39 AM UTC+1, Jack O'Quin wrote:

Longstanding Debian practice is to use /usr/lib/pythonX.Y/dist-packages. I don't know why that is, or what interpretation of the FHS it implies. 

There may be several factors:

 * The original definition of /usr/share/ was for files that could be shared read-only across multiple machines in an NFS network. Even though Python *.pyc files are architecture-independent, they are not Python-version-independent. So, they can't be shared across a network of dissimilar systems with different Python versions.

Yes, so that still leaves open whether it is wise to install python scripts to /usr/lib/packagename, rather than /usr/lib/pythonX.Y/dist-packages/[something/]packagename

I could live with the current way if the scripts were just python2 and python3 compatible 3-liners invoking a main function in some python module, that could be installed in parallel in two different locations. But view_frames as an example isn't.

[Justin Bronder]

On 04/02/13 16:56 -0800, tkruse wrote:
>
>
> On Tuesday, February 5, 2013 1:37:39 AM UTC+1, Jack O'Quin wrote:
> >
> > Longstanding Debian practice is to use /usr/lib/pythonX.Y/dist-packages. I
> > don't know why that is, or what interpretation of the FHS it implies.
> >
> > There may be several factors:
> >
> > * The original definition of /usr/share/ was for files that could be
> > shared read-only across multiple machines in an NFS network. Even though
> > Python *.pyc files are architecture-independent, they are not
> > Python-version-independent. So, they can't be shared across a network of
> > dissimilar systems with different Python versions.
> >
>
> Yes, so that still leaves open whether it is wise to install python scripts
> to /usr/lib/packagename, rather than
> /usr/lib/pythonX.Y/dist-packages/[something/]packagename

The latter would be nice. However in that instance, it would be better to
use the following as not everyone uses dist-packages.

python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"


--
Justin Bronder

[tkruse]

So discussing open issues in a meeting, here are some results:

executables are to remain in lib/packagename for now (instead of lib/pythonx.y/...; share is discarded for reasons already mentioned in this thread):
* as supporting side-by-side installs pytho2/python3 is very tedious with very little real benefit for us
* because that's also where other executables are (no separate namespaces)

ROS packages should create all their executables (python or other) to lib/packagename preferrably, as that works with rosrun (package-namespace)
* The global bin folder of the installspace is problematic because it does not work with rosrun
* So some packages (core ros stuff) may ALSO generate to global bin folder, when there is very good reason to, yet should use descriptive, likely unique names (So rostopic is a valid use-case, though maybe not the best example)
* In very rare cases, a ROS package may also only install to the global bin folder, but no good example other than roscore came to mind.
* Generally though, catkin packages that are not ROS packages (if such a thing were to exist one day), are free to put anything into the bin folder like any other cmake project
This is to be documented on the ROS wiki in the catkin/ROS tutorials



The preferred way to install python scripts shall be the setup.py data_files argument to distutils.setup(). catkin shall however not copy files as listed using data_files into the develspace, nor give an error message. One problem here is that the file permissions are not changed to make a script executable, but it should be any way to be rosrunnable (If it is, we are confident the permissions remain intact via either way, to be confirmed). The larger problem with both this approach as well as the cmake approach is that the hashbang line is not being changed as it would be by distutils, to point to the correct location of the "python" executable. To be investigated further whether we can do anything about that using lower-level distutils.

Another general problem here is that the catkin develspace semantics are not well defined. In a way, the develspace enhances the sourcespace to act as a runtime environment, so that the install step is not required to run executables. However, there apparently is no strict requirement that running anything in the develspace shall behave exactly as in the installspace, even if developers should try to make this happen the best they can, without duplicating source files.

So by not copying files into the develspace, we can cause executables to fail in the develspace due to a missing resource where the resource is present in the install space. If we'd copy files over into the develspace, develspace and installspace would behave more similar in that respect, but source files would get duplicated. The same problem exists when using the install macro in cmake, but there maybe it is more transparent to users, given the install macro of cmake is only used for install, while much of the setup.py information is also used without invoking make install.

So I feel this might not be a good solution, to me the duplicated source files seem like the lesser evil, and more opinions are welcome.

[Jonathan Bohren]

As an aside, the TF is still not installing view_frames etc into the lib/tf directory.

https://kforge.ros.org/geometry/geometry/file/2eee66fd724e/tf/setup.py#l9

-j

[tkruse]

Yes, i recently bumped the issue and assigned Tully, not sure whether anyone else wants to fix this:
https://code.ros.org/trac/ros-pkg/ticket/5610

[Tully Foote]

Ioan just moved geometry to github: https://github.com/ros/geometry  I hope to get to it shortly, the rosdistro effort REP 137 is currently our focus though.  But a pull request would make it happen much quicker.  Also I'll be migrating all the tickets to github soon too. 

Tully

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-buildsy...@googlegroups.com.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/iL5FBG80njUJ.

[Jonathan Bohren]

Here you go: https://github.com/ros/geometry/pull/2

--

Jonathan Bohren

Laboratory for Computational Sensing and Robotics

http://dscl.lcsr.jhu.edu/People/JonathanBohren