export python modules in catkin package

[tkruse]

Hi,

I was curious as to how catkin deals with python modules. Unless I am mistaken, the current support in catkin for python modules in catkin packages works via setup.py only. This means if one wants to create a catkin package which contains python modules to be imported in other catkin packages, one has to define a setup.py and use the catkin_python_setup() cmake macro.

The catkin_python_setup() will make sure that some things declared in the setup.py will go into the devel/install space.

However, A setup.py contains some redundant information with the CMakeLists.txt and package.xml fie. Also, all catkin requires from it at this time is, unless I am mistaken, the version, scripts, packages and package_dirs.

A user might not be able though to create a fully functional setup.py that could e.g. create a valid package on pypi. So such a user would have to define a crippled setup.py just so that he can use the catkin  catkin_python_setup() macro.

Am I missing something here, or else should there be some mechanism to provide catkin with (scripts, packages and package_dirs) without a setup.py?

cheers,
  Thibault

[tkruse]

Note I just checked on how things are done with existing ros_comm python files, and I see that catkin_pkg provides a python function 'parse_package_for_distutils' to reuse information from package.xml in setup.py. So providing redundant information can be avoided.
That API is currently not documented anywhere and not part of tutorials (will put on my todo list).

But that still leaves the question whether such a setup.py should be the only way to declare python resources to install in catkin.

[Tully Foote]

We chose to use/require setup.py to keep things pythonic.  This way we don't have to reimplement the python install logic in catkin. Catkin is using setup.py under the hood, for example if you ask catkin to install to /foo and you have python code it will call setup.py install with /foo as the correct argument.  Catkin uses  the mentioned "redundant" data to be able to do the workspace chaining logic and integration with autogenerated code.  But otherwise the goal is to not force python programmers to use c++ tools for python.  

Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/FDJyBE3Z1SEJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Tully Foote
tfo...@willowgarage.com
(650) 475-2827

[tkruse]

I get the well-meant intention, but the actual logic is reimplemented in catkin anyway, since we really just use setup.py as a container for lists of strings. i.e. we do not invoke it to install stuff to devel or install folders, AFAIK.

And that is a bad thing, since we now force python developers to write a setup.py even if they have no desire or need to do so. And I dislike seeing ROS projects being delivered with a defunct setup.py, that kinda looks like a valid setup.py, but cannot work. That's no a pythonic way neither, only superficially.

Writing a catkin macro that allows to capture a few strings and pass them to the catkin functions we have implemented already anyway should not be much effort.

So we might keep te catkin_python_setup macro, but really only use it with valid setup.pys (not sure whether we have any), and otherwise just use a plain cmake macro.

On Tuesday, November 27, 2012 10:15:35 PM UTC+1, Tully Foote wrote:

We chose to use/require setup.py to keep things pythonic.  This way we don't have to reimplement the python install logic in catkin. Catkin is using setup.py under the hood, for example if you ask catkin to install to /foo and you have python code it will call setup.py install with /foo as the correct argument.  Catkin uses  the mentioned "redundant" data to be able to do the workspace chaining logic and integration with autogenerated code.  But otherwise the goal is to not force python programmers to use c++ tools for python.  

Tully

On Tue, Nov 27, 2012 at 6:55 AM, tkruse <tibo...@googlemail.com> wrote:

Note I just checked on how things are done with existing ros_comm python files, and I see that catkin_pkg provides a python function 'parse_package_for_distutils' to reuse information from package.xml in setup.py. So providing redundant information can be avoided.
That API is currently not documented anywhere and not part of tutorials (will put on my todo list).

But that still leaves the question whether such a setup.py should be the only way to declare python resources to install in catkin.

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/FDJyBE3Z1SEJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

Checking again, it seems our setup.pys are mostly functional, except for maybe the install_requires section.

And maybe we do use it for the install space, after all, so I spoke too soon.

[tkruse]

So, some discussion happened here:
https://github.com/ros-infrastructure/catkin_pkg/issues/20

My current thinking now is that catkin users should not have a setup.py file. Instead, I suggest a macro like:

catkin_python_install(
  PACKAGES roslaunch
  SCRIPTS scripts/roscore scripts/roslaunch scripts/roslaunch-deps scripts/roslaunch-logs
  PACKAGE_DIR "{'': 'src'}"
  REQUIRES = genmsg  genpy roslib rospkg
)

hopefully the requires section can be fully replaced by catkin one day. Not sure whether the PACKAGE_DIR section can be made prettier, though maybe it can have a default value as stated there.
The macro should then generate a setup.py somewhere in the build space, which might look something like:

from distutils.core import setup
from catkin_pkg.package import parse_package_for_distutils

d = parse_package_for_distutils()
d['packages'] = ['roslaunch']
d['package_dir'] = {'': '/home/user/groovy_underlay/src/ros/tools/roslaunch/src'}
d['scripts'] = ['scripts/roscore',
                'scripts/roslaunch',
                'scripts/roslaunch-deps',
                'scripts/roslaunch-logs']
d['install_requires'] = ['genmsg', 'genpy', 'roslib', 'rospkg']

setup(**d)

Note the absolute path to the source folder.
This worked for me with setup.py targets build, bdist and install. sdist did not work (though it worked with setuptools), not sure whether sdist is crucial to catkin.

That way, catkin users will not need a third file next to package.xml and CMakeLists.txt in which to specify dependencies, users will not need to learn distutils/setuptools/distribute/distutils2. The catkin_pkg API for the support will be hidden mostly from users (unless they go looking for it). The approach should also be much more robust, compared to the current approach of executing a user-written setup.py to get those values I put in the cmake macro draft above.

[Tully Foote]

We definitely don't want to forray into writing the contents of setup.py into CMake and then generating the setup.py.  This creates a very large cognative disconnect for people who are focused on python only.  With the current behaviour the python programmer just needs to know to add one line to say, "i'm using python" to integrate into a catkin package.  

I think that the cleanest solution is using the method to create the dict to pass to the native setup.  

This keeps the intrusion into setup.py down to one extra method.  The method can do sanity checking if we want to allow us to provide at least warnings about collisions. We probably don't want to get into merge logic but could if we wanted.  This lets the user not type out the things that are auto generated.  And obviously they have an opportunity to mutate it later, but by recommendation we can do sanity checking like asserting that they don't have different versions in setup.py and in package.xml.  After all this is a convenience function and is not necessary at all.  With respect to keeping the syntax as close to setup.py as possible, it's a tossup between calling **args inside setup.py with the other keyword args and passing the keyword args to the generator.  However going back to the fact that this is a convenience function being able to provide helpful warnings and errors if the keywords are passed to our function should allow us to provide a better user experience.  

With respect to distutils vs setuptools, as we currently only support distutils lets keep it specific to that there can be a parallel implementation for setuptools if we come to support that in the future.  Also following the theme of being a convenience function using the **args is much more convenient than requiring name = args['name'], ... for all the arguments.  It is more obvious, but it's much harder to maintain longer term and can easily lead to things like forgetting to manually copy in one of the arguments.  

So I'd suggest we go with Williams proposal in the ticket https://github.com/ros-infrastructure/catkin_pkg/issues/20#issuecomment-10782894 with the updates from dirk for the naming.  

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/AKDeOMctrWQJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

I was not suggesting replacing catkin_python_setup, but having a second way of doing things. So that one the one hand python programmers could have their setup.py, but our rosbuild-spoiled user does not need to learn distutils, and understand how catkin monkey-patches distutils, so that a user may debug errors in his setup.py. This would also avoid all errors due to users actually using setup.py, not catkin, preferredly by invoking "sudo setup.py install", because that's what the internet says about setup.py.

About our strategic vision, will we try to make the setup.py become fully functional such as invoking it would also generate ROS messages?
Also, do we consider putting all (or core) ROS packages on pypi one day, such that the dependencies (e.g. on rospy) we already put into setup.py become meaningful one day? Because right now, it's a bit difficult to explain in the tutorials that a user should take care to write the dependencies in their setup.py, but that those will currently not be used, or checked.

Next, we do provide a convenience function to use package.xml values for setup.py, which as you say is voluntary for users. Currently catkin only checks whether the version in package.xml matches the one in setup.py. It does not check name, dependencies, maintainer, license, etc. Should those then also be checked if they do not meet the expectation that the values correspond to how our convenience function transforms package.xml values for setup.py?

To unsubscribe from this group, send email to ros-sig-buildsystem+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/AKDeOMctrWQJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tkruse]

Also, though nothing is clear in the Python Packaging world, there seems to be a motion to have distutils2 replace distutils.
http://alexis.notmyidea.org/distutils2/setupcfg.html
distutils2 uses a setup.cfg file, meaning not an executable file. For the current catkin design to support such a python build system, there would be no more room for a convenience python function to avoid replicating redundant data from package.xml.

If one wanted in the future to try whether we could switch all ROS packages to distutils, with the current design such an attempt would have to go through all catkin catkin packages with setup.py's and manually switch to another build system. If we however generated the python setup file that we want to support, this could be tested (and later migrated) without touching the source of all catkin packages (this regardless of whether we want to try distutils2 or any other tool).

I'm just trying to point out benefits and disadvantages. Maybe one day we could also have a REP for the design decisions around catkin_python_setup with the motivations summarized for posteriority.

[tkruse]

I meant "switch all ROS packages to distutils2 ..."

[tkruse]

Actually we don't even need to generate a setup.py, catkin can invoke distutils setup() method with the values from the package.xml and whatever the user declares as his modules and scripts.

I see the benefit of supporting custom setup.pys mostly when we talk about using other disutils functions. Yet, catkin currently cannot handle additional disutils functions, such as the

py_modules and data_files parameters or extention modules. Those things will fail to have effect in the devel space, currently, without even a warning or error. (And no documentation as of today)
Are there plans to either forbid those arguments or provide an equivalent implementation for the devel space?

If we'd install to devel space using setup.py on "make", that would be clean, but we'd lose the ability to work with python without make invocation. 
And I'll guess hacking into distutils to install relay files in the "make" scenario when invoking setup.py is also something we don't want to do.

[Tully Foote]

Although this is a neat ability to be able to do this, however it runs completely opposite of the design goal of keeping developers using the standard tools.  It would force the python developer to not use setup.py and learn an entirely different system.  Whereas with the other approach we provide tools to help the developer not provide duplicate information.  If they don't want to use our helper methods they can simply use setup.py manually themselves.  To integrate the system we require telling catkin to look for the python toolchain, but we don't require them to completely learn a new custom toolchain.  With In our method we can catch the arguments and warn the user they're using unsupported arguments.  

With this in mind I don't think that we should change our approach and push all the metadata into cmake/catkin, and we should augment the argument sanity checking to add warnings for unsupported arguments to setup.py.  

Tully

--

You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.

To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/ros-sig-buildsystem/-/N9z3QkmBkLcJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Geoffrey Biggs]

On Dec 4, 2012, at 8:28 AM, Tully Foote wrote:
> Although this is a neat ability to be able to do this, however it runs completely opposite of the design goal of keeping developers using the standard tools. It would force the python developer to not use setup.py and learn an entirely different system. Whereas with the other approach we provide tools to help the developer not provide duplicate information. If they don't want to use our helper methods they can simply use setup.py manually themselves. To integrate the system we require telling catkin to look for the python toolchain, but we don't require them to completely learn a new custom toolchain. With In our method we can catch the arguments and warn the user they're using unsupported arguments.

I'm sorry, but I find this whole paragraph completely confusing. I'm probably missing some information. I thought the whole point of Catkin was to provide a unified approach rather than using the standard tools for whatever the developer is developing. Why is the approach for C++, Python and packing tools so different?

Geof

[tkruse]

On Tuesday, December 4, 2012 12:28:43 AM UTC+1, Tully Foote wrote:

Although this is a neat ability to be able to do this, however it runs completely opposite of the design goal of keeping developers using the standard tools.  It would force the python developer to not use setup.py and learn an entirely different system.

We currently do force python developers to not use setup.py (not invoke setup.py manually) and we force them to learn an entirely different system (package.xml and CMakelists, cmake commands, catkin workspaces, etc).

I believe for this design goal, catkin should make it possible to develop python projects using just a package.xml and a setup.py, providing addons for setup.py just as we currently provide additional macros for cmake (mostly genmsg). The cmake calls that are necessary (such as generating XXXConfig.cmake files) should not be in project files, but in generic catkin files. In a mixed c++ / python project, the CMakeLists.txt file should be written agnostic of the setup.py part.

That would be more work, but i would not oppose following that design goal. But if we follow it, I would like us to follow it all the way through.


BTW: Note I started documentation of the limitations of the current approach:
https://github.com/ros/catkin/blob/master/doc/user_guide/setup_dot_py.rst
Not sure how much of it would fit into the tutorials.

[Tully Foote]

Hi Geof, 

Sorry my last emails was a little bit hurried and was mostly a reply to ongoing discussions with Thibault.  

For clarity we've been trying to use as standard a toolchain as possible.  This allows us to leverage the mature standard tools.  For example, rospy being a pure python package can be written as a pure python project, with setup.py. To be able to integrate into the larger system we do require package.xml and a minimal boilerplate cmake file.  

The required CMakeLists.txt is:

{{{

cmake_minimum_required(VERSION 2.8.3)

project(rospy)

find_package(catkin REQUIRED)

catkin_package()

catkin_python_setup()

}}}

If you read the file in rospy there are some extra lines below for backwards compatibility with rosbuild, but those are not necessary for most packages.  

The setup.py for rospy reads like this:

{{{

from distutils.core import setup

from catkin_pkg.python_setup import generate_distutils_setup




d = generate_distutils_setup(

    packages=['rospy', 'rospy.impl'],

    package_dir={'': 'src'},

    requires=['genpy', 'numpy', 'rosgraph', 'roslib', 'rospkg']

)




setup(**d)

}}}




Where d evaluates to: 

{{{


{'author': u'Ken Conley',
 'description': 'rospy is a pure Python client library for ROS. The rospy client\n    API enables Python programmers to quickly interface with ROS <a href="http://ros.org/wiki/Topics">Topics</a>, <a href="http://ros.or',
 'license': 'BSD',
 'long_description': 'rospy is a pure Python client library for ROS. The rospy client\n    API enables Python programmers to quickly interface with ROS <a href="http://ros.org/wiki/Topics">Topics</a>, <a href="http://ros.org/wiki/Services">Services</a>, and <a href="http://ros.org/wiki/Parameter Server">Parameters</a>. The\n    design of rospy favors implementation speed (i.e. developer\n    time) over runtime performance so that algorithms can be quickly\n    prototyped and tested within ROS. It is also ideal for\n    non-critical-path code, such as configuration and initialization\n    code. Many of the ROS tools are written in rospy to take\n    advantage of the type introspection capabilities.\n\n    Many of the ROS tools, such\n    as <a href="http://ros.org/wiki/rostopic">rostopic</a>\n    and <a href="http://ros.org/wiki/rosservice">rosservice</a>, are\n    built on top of rospy.',
 'maintainer': u'Dirk Thomas',
 'maintainer_email': 'dth...@willowgarage.com',
 'name': 'rospy',
 'package_dir': {'': 'src'},
 'packages': ['rospy', 'rospy.impl'],
 'requires': ['genpy', 'numpy', 'rosgraph', 'roslib', 'rospkg'],
 'url': 'http://ros.org/wiki/rospy',
 'version': '1.9.39'}


}}}

In this way there's a convenient way to avoid duplicating data between the package.xml and the setup.py file.  

In the CMakeLists.txt you can see the one line to tell catkin that this package has a setup.py file.  And from that catkin can invoke setup.py with the right arguments to install packages into the right places, and distutils takes care of all the heavy lifting.  Things like the logic for setting the path to install into based on what version of python you're using, don't need to be replicated.  And after it's installed the only requirement to use the python library is to set the PYTHONPATH to the installed directory, which catkin builds into the setup.(ba)sh files as well.  This way to convert a pure python package from catkin to pip based installation, or the reverse, it's just a requirement to add or remove the short CMakeLists.txt boilerplate and a packages.xml.  To convert into a catkin package you don't even need to switch to uses the catkin_pkg helper function if you don't want, you can keep using the fully written out setup.py.  

Likewise on the C++ side we're leveraging CMake as much as possible.  We've added helper methods on top to automate the most boring code and package metadata related to managing multiple packages.  But the end result is to let cmake do the majority of the work.  This means that much less of the code is custom and the standard tools can be relied on for a larger part of the system.  Catkin packages, once installed, can be used from any cmake project by simply using the FindPackage infrastructure, automatically generated by catkin, meaning that it is not even necessary to use catkin to build on top of catkin packages. 

This is getting longer than I planned.  But I hope it's a little more clear about the design intentions.  

Tully

--
You received this message because you are subscribed to the Google Groups "ROS Buildsystem Special Interest Group" group.
To post to this group, send email to ros-sig-b...@googlegroups.com.
To unsubscribe from this group, send email to ros-sig-buildsy...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Geoffrey Biggs]

Hi Tully,

Thanks for the explanation. It makes more sense now, and I can see that things are being done in a consistent way.

Geoff

[tkruse]

On Friday, January 4, 2013 9:41:19 AM UTC+1, Tully Foote wrote:

This way to convert a pure python package from catkin to pip based installation, or the reverse, it's just a requirement to add or remove the short CMakeLists.txt boilerplate and a packages.xml.

Let's note for posteriority that a package using ROS message generation (containing .msg, .srv or .action files) cannot be ported to pip-based installs that way, and that dependencies listed in your rospy setup.py example (genpy, rosgraph, roslib) are not known to pip (and are currently never checked or used by catkin). Similarly there are several arguments that could be used in a given setup.py which will not work with the catkin devel space.

http://ros.org/doc/groovy/api/catkin/html/user_guide/setup_dot_py.html

So converting is not generally quite as trivial as described above, though in some cases it may be.