LCM not receiving messages I know are getting to the machine

262 views
Skip to first unread message

Jonathan Howland

unread,
Oct 9, 2019, 11:38:57 AM10/9/19
to Lightweight Communications and Marshalling
I'm not completely sure that I am dealing with a LCM problem--it might be a routing issue, but I'm not sure.  
envi

In an attempt to shield low bandwidth (100 megabit) devices from high rate broadcast of camera images (all over gigabit) I have purchased a small Ubiquity router that supports IGMP Proxy for multicast.  Ultimately, I want to be running on a Nvidia TX2 over a fiber optic interface, but in an attempt to get things going, I have set up things on a small wired network.  In summary, even though I am convinced that the routing is happening properly, I am not able to recieve lcm packets

I have two laptops set up--one at 192.168.1.205, and the other at 192.168.2.200.  They are both wired to appropriate ports on the router, which is configured to support IGMP protocol on the  ports.  All firewalls are turned off

On 192.168.2.205, I run the lcm_send example program, with LCM_DEFAULT_URL set to 239.255.76.67:7667?ttl=2.  I have added a route with
"sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eno1"

on 192.168.1.200, I am running the listener_async example program.  I have LCM_DEFAULT_URL set to 239.255.76.67:7667?ttl=2 and a route set up using
"sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev enp1s0f1"

listener_async receives nothing.  I run lcm-spy on the same machine, and it receives nothing.

However, when I run wireshark on the 192.168.1.200 machine, I see the LCM packets showing up.  I see the string for the channel name, and the "example string" that is part of the data type.  Similarly, when I look at the diagnostic console on my router, I can see that the router is seeing incoming messages on its 192.168.2 interface, and is sending them to its 192.168.1 interface.  So I am pretty sure that the messages are actually being routed OK, and that I do not have a TTL issue.

When I run lcm_send on the 192.168.1.200 machine (local to the receiver and the lcm-spy) everything works--both the listener_async and lcm-spy see the messages.

I have monkeyed with a lot of settings on the router, and can make things not work (no data flow between the interfaces) but cant make LCM work.  For the above reasons (see data on wireshark) I am fairly sure this is not a routing problem,but since things aren't working, I'm not positive.

environment:  Ubuntu 16.04 on the 192.168.1 machine, running lcm 1.3.1, Ubuntu  18.04 on the 192.168.2 machine, running lcm 1.4.0

Any ideas?  

Jonathan Howland

unread,
Oct 9, 2019, 3:02:24 PM10/9/19
to Lightweight Communications and Marshalling
An update:   I still don't know what's going on, but have more information.  On the machine that I wanted to be the receiver (192.168.1.) I ran iperf, which allowed me to arbitrarily monitor network ports.  It couldn't see the incoming lcm messages from the 192.168.2 network.  So even though I could see it coming in on wireshark, it wasn't actually getting anywhere.  When I ran send-message locally, it showed up in iperf with no issues.

So apparently the routing to the interface/socket number isn't right.  So I took a third laptop,changed its address appropriately,  plugged it into a switch on the 192.168.1 network, and used send-message on that machine.  Not only did my target machine show the data (on listener-async) but it also started showing the messages being sent from the 192.168.2 network!  I could unplug the third laptop, and the messages continued to flow from the 192.168.2 network.

However, when I ^C 'd the listener, and restarted it, the data from the 192.168.2 network stopped.  I repeated the entire sequence to make sure that's what happened, and verified that I couldn't receive the messages from the other subnet until I first received one from the local net.  

I still dont really understand, and don't necessarily know how to make my application work.  Any insights would be appreciated

Jonathan Howland

unread,
Oct 9, 2019, 3:51:13 PM10/9/19
to Lightweight Communications and Marshalling
Another piece of information that might be related.  When the receiver is getting data from across the router, and I run lcm-spy on the same machine, neither works. Neither listener-async nor lcm-spy receive any traffic.  When I kill lcm-spy, listener-async starts getting data again. 

I tried another computer on the 192.168.2 subnet to see if it could receive things as I thought it should--no, it couldn't.  tcpdump showed the data showing up, but lcm wouldn't receive it.


Jonathan Howland

unread,
Oct 10, 2019, 3:00:07 PM10/10/19
to Lightweight Communications and Marshalling
And...the resolution:  as explained in https://stackoverflow.com/questions/13471680/udp-multicast-client-does-not-see-udp-multicast-traffic-generated-by-tcpreplay, there is a kernel "function" called "reverse path filtering" that needs to be turned off in this case.  I did it temporarily with 
sudo sysctl net.ipv4.conf.all.rp_filter=0 and
sudo sysctl net.ipv4.conf.eth0.rp_filter=0

Apparently performing

echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

will ensure that it lasts through rebooting, etc.

Once I did this, everything worked as I hoped
Reply all
Reply to author
Forward
0 new messages