Weird issue with ROS Control and IGH EtherCat Master

[Shawn Schaerer]

Hi,

I built a custom robot using ROS Control from ROS Control Boilerplate template and I am having an issue.

For background.  I have ros control and moveit working together and verified that it all works right down to the hardware read and write (using fake data).

After this verification I integrated my EtherCat hardware using IGH EtherCat Master and then started to get some weird errors / results

"""

process[zzz/ros_control_controller_manager-2]: started with pid [29074]

[ERROR] [1443750457.900432227]: [registerService] Failed to contact master at [localhost:11311].  Retrying...

[ERROR] [1443750457.956230612]: [registerService] Failed to contact master at [localhost:11311].  Retrying...

process[zzz/robot_state_publisher-3]: started with pid [29078]

[ERROR] [1443750458.012246939]: [registerService] Failed to contact master at [localhost:11311].  Retrying...

[ERROR] [1443750458.121226678]: [registerService] Failed to contact master at [localhost:11311].  Retrying...

Loaded joint_state_controller

[ERROR] [1443750458.392289430]: [registerService] Failed to contact master at [localhost:11311].  Retrying...

Loaded position_trajectory_controller

"""

Things don't run and when I try connecting to the ROS Control program using RQT_GUI's joint trajectory interface it freezes and I have to ctrl-c the program.

Any ideas?

Thanks

[G.A. vd. Hoorn - 3ME]

On 2-10-2015 5:16, Shawn Schaerer wrote:
> Hi,
>
> I built a custom robot using ROS Control from ROS Control Boilerplate
> template and I am having an issue.
> For background. I have ros control and moveit working together and
> verified that it all works right down to the hardware read and write (using
> fake data).
>
> After this verification I integrated my EtherCat hardware using IGH
> EtherCat Master and then started to get some weird errors / results
>
>
> """
> process[zzz/ros_control_controller_manager-2]: started with pid [29074]
> [ERROR] [1443750457.900432227]: [registerService] Failed to contact master
> at [localhost:11311]. Retrying...
> [ERROR] [1443750457.956230612]: [registerService] Failed to contact master
> at [localhost:11311]. Retrying...
> process[zzz/robot_state_publisher-3]: started with pid [29078]
> [ERROR] [1443750458.012246939]: [registerService] Failed to contact master
> at [localhost:11311]. Retrying...
> [ERROR] [1443750458.121226678]: [registerService] Failed to contact master
> at [localhost:11311]. Retrying...
> Loaded joint_state_controller
> [ERROR] [1443750458.392289430]: [registerService] Failed to contact master
> at [localhost:11311]. Retrying...
> Loaded position_trajectory_controller
>
> """

The ctrlr mgr can't reach the master. As that is all TCP/IP based, this
looks more like a general networking issue than something ros_control
specific.

How is your system / network set up? How many NICs do you have, where
are you running your ethercat master and how is it configured?

> Things don't run and when I try connecting to the ROS Control program using
> RQT_GUI's joint trajectory interface it freezes and I have to ctrl-c the
> program.

As this also probably can't contact the master (and any other nodes),
that is to be expected.


Gijs

[Shawn Schaerer]

Thanks for the reply.

I have this running on two different machines to test (not networked together, just two machines to test the same codebase)

Machine 1

 Ubuntu 14.04

 One Ethernet card connected to the network with DHCP.  ROS and EtherCat Master runs on this card as well

 -EtherCat Master conf?

Machine 2

ubuntu 14.04

Two ethernet cards.  One is wireless and connected to the corporate network, second is wired and connected to the ethercat hardware with a fixed IP on a different subnet

-EtherCat master Conf?

If I run the code on Machine 2 with the etherCat hardware off or on (if its on I only configure the master, no other hardware init)  RQT_GUI will hang when I talk to the JointTragictoryController.  

It hangs when my ROS Control program outputs:  list controller service called.

I can track it down to s.call(LoadControllerRequest(name)) in controller_manager_interface.py which points to transport.receive.once() in tcpros_service.py

So yes it looks like some kind of TCP issue.

The strange thing is that on machine 2, I get the Failed to connect to Master Errors, but the system just keeps going and RQT_GUI and MoveIt can connect to my ROS Control program and pass joints down to the hardware layer. Now I dont have hardware connected to machine 2, but the results are different than machine 1 when no hardware is connected.

To further this I have a normal ROS Client Server that I wrote which integrates the EtherCat Master and I do not get any Failed to Connect to Master errors at all. 

any other ideas or areas that I should look at?

Thanks

[Shawn Schaerer]

More results:

I tried a pure C++ ROS program integrated with the IgH EtherCat waste and no errors ( Failed to contact master at [localhost:11311])  Everything works as expected.

Again I tried the customized ROS Control boilerplate (with EtherCat) program and I get the same errors and behaviours.  I replicated this on Machine 2 and the RQT_GUI just hangs waiting for a response.

Next I ran the two programs at the same time on the machine. 1st C++ ROS with EtherCat program, 2nd The ROS Control one.  Results.  Program one can control my etherCat motors no problem and no errors.  

Also the ROS Control program starts up and runs with no errors.  The the EtherCat calls are commented out, but this proves that there are no issues with networking or interactions with EtherCat and ROS.

Any ideas?  My thoughts are that there is something in the Python networking code that does not play well with the EtherCat master when they are initialized from the same executable and in the same process.

Thank you for any suggestions, pointers, etc. I am going to try and debug the python code or just abandon ROS Control or build a hack, bridging EtherCat and ROS via socket (seems to defeat the purpose of ROS).  

:)

[Peter Milani]

Hi Shawn,

I've successfully integrated ROS Control, Xenomai and Ethercat using the IgH Ethercat Master. However I am not sharing cards between ROS and Ethercat. I have a separate dedicated Ethernet Card for EtherCAT running in Xenomai Kernel space and ROS running on the localhost network. ROS control and the Xenomai RTOS communicate via a shared Heap and Mutex.

I'm not using python but purely ROS Control C++ libraries, and I dont rely on wifi. Are you running the generic driver for EtherCAT?

My suggestion would be to ensure there is dedicated hardware for the etherCAT, and not to use wifi. I've not tried integrating Ethernet over Ethercat (EoE) which I assume would be required to send normal TCP across etherCAT.

 cheers

Peter

--
You received this message because you are subscribed to the Google Groups "ROS/Orocos Robot Control Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-robot-co...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shawn Schaerer]

Hi Peter,

Thank you for the reply.

I have one NIC in my machine and right now its connected directly to my Ethercat hardware.  I am using the generic driver (I think).  I just built it with make, no special config settings.

The thing that is weird is the it works on my normal C++ ROS program.

Maybe I need to separate the localhost or some other networking trick?

[Shawn Schaerer]

Peter,

Are you using the controller manager package at all?

On Wednesday, October 7, 2015 at 4:40:11 PM UTC-5, Peter Milani wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-robot-control+unsub...@googlegroups.com.

[Peter Milani]

Hi Shawn,

My package  uses the controller_manager, hardware_interface and controller_interface from ros_control. I've implemented plugins as well as the controller manager in that package, though you may have developed your plugins in a separate package.

Peter

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-robot-co...@googlegroups.com.

[Shawn Schaerer]

I'm going to try a separate NIC.  I have tried everything else that I could think of.

[Shawn Schaerer]

I did not try a separate nic, but I did create a fix the problem by launching my ehtercat subsystem in a separate executable and then exchanging joint position and feedback by ROS Pusblish and subscribers.  

Seems a bit redundant but it works.

I don't know what the underlying problem is but it has something to do with initializing the ehtercat master inside a ROS Control executable.

weird.

[Piyush Sharma]

Hey,

How you are writing and reading from ethercat slave using ros_control_boilerplate.

Can you suggest something on that.

Thanks

[Shawn Schaerer]

When I started to do the initial integration, I setup the ethercat master in the constructor / init.  After that I did a read and write in the spots where you interface with the hardware.  Given I had issues getting this to work, I created another node that handled all ethercat communications and then interfaced to ROS control via ROS messaging.

--

[Bence Magyar]

Hi Shawn,

I think that may be a bit dangerous if you have to satisfy realtime constraints too as it adds a big delay. What were the issues? You should read the actuator command interfaces, read feedback from ethercat, do the update on controllers, then move forward with writing things. Sorry for the briefness, that my general advice given what I know of your issue :)

Cheers,

Bence

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-robot-control+unsubscri...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ROS/Orocos Robot Control Special Interest Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-robot-control+unsubscri...@googlegroups.com.