Re: [ros-users] clarifying package installation 'best practices'?

16 views
Skip to first unread message

Bill Morris

unread,
Mar 4, 2015, 6:21:24 PM3/4/15
to ros-sig...@googlegroups.com
I have used Chef with ROS on production systems and I'm not thrilled
with the results, on the other hand I do know some colleagues who are
using Ansible with ROS and they seem to think it may be a better choice.
Below is some of my recollections on using Chef + ROS. Obviously, some
of these problems could probably be fixed by a better configuration of Chef.

On the positive, Chef + ROS seems to work very well for cloud based
processing. It makes it very easy to spin up an unlimited number of
known working ec2 instances for parallel offline processing of data in
bag files and without physical hardware to worry about Chef works great.

Unfortunately I experienced several problems with using Chef with ROS.

The biggest problem is that configuration is often opaque to everyone
except devops, whereas devops is often unfamiliar with hardware issues.
In my experience the chef recipes and configuration files are usually
stored/managed in multiple repositories and as a result it can often
take a significant amount of time to track down which role or recipe is
managing a given configuration element. If Chef development and
production configurations are stored in the same repository there is
also an increased testing overhead needed to prevent changes to the
development recipes and configurations from breaking production.
Installing separate configurations on to individual hardware platforms
seems to create a linear increase in complexity of the Chef
configuration as the number of robots increases, though it may be
possible to put some of the configuration in a database I imagine it
only adds to the number of places to look when things go wrong.

As a concrete example. at one point we had sensor calibration files
installed via chef, this caused significant confusion as to which
calibration should be installed on which robot hardware platform. It
also required additional overhead when the calibrations were updated, as
roboticists were making pull requests to an unfamiliar repo. Eventually,
we found it easier to store the configurations in a ROS package and load
them dynamically based on the serial number.

Network configuration management also proved to be problematic. The
hardware platform had multiple ethernet interfaces that needed IP
address configuration based on MAC addresses. When the hardware team
needed to replace two two-port cards with a four-port network card, it
also temporarily broke the chef configuration until someone on the
devops side could updated Chef.

While Chef was great for rolling out a separate development and
production version, the infrastructure is difficult to subclass. I found
that Chef made it difficult to test out ideas without making commits to
the primary Chef repo. To simplify things we had chef install
production.rosinstall and development.rosinstall files and use rosws to
install to ~/development_ws and ~/production_ws
This helped segment ROS development from Chef and made everyone happier
and tagging git releases in the production.rosinstall file was easier
than managing it in Chef.

In my experience Chef + Jenkins didn't help much with
hardware-in-the-loop testing and it really required a complete OS
reinstall to confirm that all of the local changes were correctly pushed
back into the Chef configuration.

As far as I can tell Chef requires that the owner of the Chef
infrastructure is the same as the owner of the robot. It seems like this
would be reasonable for equipment that is leased or is not considered
user-serviceable. If an "upgrade" breaks the system configuration, It
was often difficult to roll back changes and was almost always easier to
reinstall the OS + Chef from scratch once the bug is fixed.

Once everything is working it was fairly easy to generate an ISO
installer that contains the Chef tools needed to bootstrap a system.
Debian packages were also be copied to the apt cache on the USB drive to
cut down on network traffic for the install.

Overall I found development with Chef painful, maintenance and testing
tedious and static production deployments to multiple systems is where
it starts to become useful.

On 03/04/2015 03:08 PM, Ryan Hass wrote:
> Mike,
>
> I think this is where the Docker, and the Chef solutions fit in well. At
> the core of the problem, you are effectively describing a configuration
> management issue. Docker makes it easy to ship a pre-built and
> pre-configured set of things -- and the Chef solution allows to abstract
> the process into a DSL. They both have their pros and cons, but I think
> they are not mutually exclusive either. I am very interested in this
> discussion, so if you have a private thread going please loop me in!
>
> -Ryan H.
>
> On Wed, Mar 4, 2015 at 5:19 AM, Mike Purvis <mpu...@clearpathrobotics.com>
> wrote:
>
>> As an OEM who ships a lot of robots to first-time ROS users, one of the
>> things I/we have thought about at Clearpath is exactly how to configure the
>> robots for the best combination of ease-of-production on our side,
>> ease-of-getting-started on the user's side, and ease-of-maintenance on the
>> user's side. The current state of affairs is:
>>
>> - Everything on the robots is installed from debs, from a combination
>> of packages.ros.org (buildfarm) and packages.clearpathrobotics.com
>> (in-house buildbot-ros builds).
>> - An /etc/ros/setup.bash file falls through to
>> /opt/ros/indigo/setup.bash. This is what is sourced for the robot_upstart
>> job which brings up the base software.
>> - The homedir is populated with a README file explaining how to create
>> a workspace with a new package in it...
>> - ... and further how to change that "master" setup.bash, in the event
>> that your new package contains nodelets, plugins, or things you want to
>> launch as part of the robot_upstart background job.
>> - Because all the "system" software is installed from debs, upgrading
>> is a fairly sane update/dist-upgrade story.
>> - But, if there's a repo of special customization
>> launchers/URDF/whatever that's specific to that one platform, it is
>> configured as a package and placed in a source workspace in the homedir.
>> - The desktop and simulator software is also distributed as debs, and
>> so provide a sane story for getting started on the user's supporting
>> computer (but if the user wants the meshes or other stuff from a
>> customization package, it must be manually copied over, le sigh).
>>
>> We also resell a number of other vendors' products, and many do not make
>> it any farther than a source workspace for everything— often with a manual
>> apt-get step for dependencies (rather than using rosdep). On the one hand,
>> this is perhaps easier for hacking on and exploring the platform's own
>> software, but it sucks for pretty much everything else.
>>
>> Anyhow, if anyone's interested in these issues or has thoughts about how
>> to improve the state of affairs for OEMs or users of OEM robots, please get
>> in touch off-list, would love to discuss further; at some point we could
>> also consider establishing a SIG for vendors. In any case, I can be found
>> at mpu...@clearpathrobotics.com
>>
>> M.
>>
>> On 4 March 2015 at 07:42, Jonathan Bohren <jonatha...@gmail.com>
>> wrote:
>>
>>> I think this is a great idea. I can't say the likelihood that this gets
>>> added to the wiki, but I'd be happy to accept a PR to add this to
>>> http://rosindex.github.io
>>>
>>> https://github.com/rosindex/rosindex/issues/142
>>>
>>> -j
>>>
>>> On Wed, Mar 4, 2015, 03:44 G.A. vd. Hoorn - 3ME <
>>> g.a.van...@tudelft.nl> wrote:
>>>
>>>> All,
>>>>
>>>>
>>>> just wondering whether we should somehow clarify what the current 'best
>>>> practices' are for installing packages onto a ROS pc. I see a lot of
>>>> people that assume that building from source (after cloning from a
>>>> repository) is _the_ way to do things.
>>>>
>>>> Apart from the fact that this is essentially a waste of time and effort,
>>>> it also often leads to problems (as they forget to check for and / or
>>>> install all dependencies first), which then results in numerous ROS
>>>> Answers questions. Having the sources locally also seems to invite some
>>>> users to start editing / hard coding parameters (such as IPs / serial
>>>> ports) into nodes, which is obviously unwanted.
>>>>
>>>> Somehow I have a feeling that the prominent placing of the repository
>>>> URL in the Package Summary contributes to the confusion.
>>>>
>>>> Would it perhaps be an idea to add a one-liner to the 'Package Summary'
>>>> on wiki pages of released packages that shows users how to install it
>>>> ("Installation: sudo apt-get install ros-$release-pkg-name", although
>>>> that is distribution specific)? Or a link to a (new) wiki page that
>>>> explains how to install packages in general (with the from-source option
>>>> shown last)?
>>>>
>>>>
>>>> Gijs
>>>> _______________________________________________
>>>> ros-users mailing list
>>>> ros-...@lists.ros.org
>>>> http://lists.ros.org/mailman/listinfo/ros-users
>>>>
>>> _______________________________________________
>>> ros-users mailing list
>>> ros-...@lists.ros.org
>>> http://lists.ros.org/mailman/listinfo/ros-users
>>>
>>>
>> _______________________________________________
>> ros-users mailing list
>> ros-...@lists.ros.org
>> http://lists.ros.org/mailman/listinfo/ros-users
>>
>>
>
>
> _______________________________________________
> ros-users mailing list
> ros-...@lists.ros.org
> http://lists.ros.org/mailman/listinfo/ros-users

Reply all
Reply to author
Forward
0 new messages