For suffers of dual-network interfaces on embedded Ubuntu processors trying to use Ros2

[Michael Wimble]

If you are not trying to run both an Ethernet and wifi network at the same time, under Ubuntu Linux, for use with Ros2 across multiple computers (e.g., my desktop Mac and my embedded AMD processor), then this article is not for you. Nothing to see here. Go about your business.

If your embedded robot computer is running Ros2 and is using only one network interface, typically the wifi, then launching your robot stack and then going to your desktop to run rviz or teleop_keyboard is a piece of cake. Do what it says in the docs, which is pretty much nothing special.

In my robot, Puck, my main embedded processor, an AMD 3700X is doing the bulk of the work and talks to my house network over wifi. But, and here’s the gotcha, I have two other computers that talk over ethernet to that AMD processor: an Nvidia processor for AI and a custom processor based on the Teensy 4.1 that acts as a sensor processor and system monitor.

For the computer to both talk to wifi and the ethernet, you need to set the “Make available to other users” option in the Details pane of the Wired Network panel.

Then you need to assign a static IP subdomain to devices using the ethernet. Here I say that everything connected to the ethernet bus must have an IP address in the 10.42.0.X subdomain. For example, my custom monitor talks on the IP address of 10.42.0.132 and the AMD processor has an ethernet address of 10.42.0.132 Ignore that the photo below erroneously shows and address of 10.42.0.0 instead of 10.42.0.132. This will set up routing tables automatically so embedded traffic between the 10.0 and 10.42 subdomains can talk to each other.

So far, this is all easy and you can probably find articles that tell you how to get this far. If you run the “ip address” command, you will see that my AMD processor has two network addresses: 10.0.0.48 which is the address when talking over the wlp38s0 wifi interface, and 10.42.0.0 when talking to the enp37s0 ethernet interface (that is wrong here, it should be set to 10.42.0.133, but that’s not important for this article).

As soon as you turn on both the ethernet and wifi interfaces, if you look at the route table, you will see something like this:

The important bit is the ‘link-local’ entry in the routing table. This says that all traffic not specifically directed to 10.0 or 10.42 is to be shuttled off to the enp37s0 device, which is my ethernet device. Now for the meat of this article.

When you start some ros2 node on your embedded Linux on the robot, it advertises itself via multicasting. And, by default, it does so via the link-local route. Which means that, as configured so far, only other ros2 nodes on the embedded AMD processor or devices connected to the ethernet channel of the AMD processor can see each other. The Nvidia and custom processor happily talk with the AMD processor. But my desktop processor, which is on my home's 10.0.0.X wifi subnet, cannot talk to the 10.42.0.X subnet (I won’t go into the magic you can use to slightly get around that limitation). More importantly, the ros2 nodes running on my desktop computer are going to look for multicast traffic on the 10.0.0.X subnet, and that’s not where the AMD processor is sending the multicast network.

So, until you do the right magic, the desktop computer cannot see any ros2 traffic from the AMD processor.

Being at the level of ignorance that I am, but being persistent, I spent many days trying to figure out why the two computers couldn’t talk to each other. If I turned off the ethernet interface on the AMD, everything worked fine. But when I turned the ethernet interface back on, the two computers couldn’t talk to each other via ros2. That clue led me into a maze of twisty little passages, all  alike. I had hints that the DDS software used for ros2 networking was using multicast so nodes could talk to each other, but all my attempts to force multicast traffic to be routed through the wifi interface instead the ethernet interface failed.

If you know of magic that might have solved this, I’d love to hear about it.

Now for the magic that worked. If you come across this situation and this magic solves it for you, please send me artwork of Ben Franklin from the government mint, or at least buy me a donut some day.

You need to set the environment variable for CYCLONEDDS_URI. Well, assuming your using the default DDS for the ros2 Galactic distribution. So, here is the magic I have in a shell script that I run to set up my working environment on my embedded computer before I start using ros2. Note that the “wlp38s0” is the name of my wifi interface.

[Alex Sy]

Hi Michael,

You could instead look at using a bridge.  This will cause all the networks to use the same subnet.  The following is from very old memory since I used it a lot over 15 years ago.  I think it should still work in Linux although the danger is some of the new network managers that run in the background may fight with your changes.  I usually get rid of them if I want to control exactly what my system is doing.

 

1.  You make sure the Ethernet and Wifi do not have an IP assigned, but is up.  You can use "ifconfig <interface> 0.0.0.0 up" 

2.  You use brctl to create a new bridge interface, lets call it brg0.  Use "brctl addbr brg0"

3.  You use brtcl to add the wifi interface to it.  Use "brctl addif brg0 wifi..."  Change wifi... to your wifi interface

4.  You use brctl to add the ethernet interface to it.  Use "brctl addif brg0 enp.."  Change enp... to your ehternet interface

5.  You then let the DHCP get an IP for the brg0 interface, it will ifnd the DHCP server of your wireless network.  Depends on your DHCP client

Any other device you have on the robot, that connects to the ethernet will get it's IP from the wireless network.

ROS would then think everything is on the same subnet and the multicast will be seen by both the WIFI and Ethernet.

You could modify your /etc/init.d scripts or rc.local to do some of this so that is is done at startup.

 

With some tricks like this, together with iwconfig (for wireless) to make the robot work both standalone (acting as AP) and when it is connected to a home network, or when it is outside network or a cellular network.

 

Hope this helps.

Alex 

--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hbrobotics/BB63CDB6-63EE-4F61-90D3-3114E0992917%40gmail.com.

[camp .]

    I assume the primary reasons for going Ethernet are speed, reliability, and security??? Certainly not ease of use.  ;-)))

 

Thanks,

Camp

--

[Michael Ferguson]

I posted some notes on CycloneDDS a while back on my blog: https://www.robotandchisel.com/2020/08/12/cyclonedds/ - specifically see the section about

export CYCLONEDDS_URI='<CycloneDDS><Domain><General><NetworkInterfaceAddress>wlp2s0</NetworkInterfaceAddress></General></Domain></CycloneDDS>'

to setup Cyclone to default to your wifi interface.

-Fergs

To view this discussion on the web visit https://groups.google.com/d/msgid/hbrobotics/696616334.6144558.1655735215873%40mail.yahoo.com.

[Michael Ferguson]

I didn't quite read far enough in your post to see you actually found the same solution - that's not really "magic" - it's just not well documented - but it is the correct solution. There is a ticket against cyclonedds to make it default to all interfaces - but AFAIK, they have not made any progress on that.

I'll also link to my HBRC talk from a while ago: https://github.com/robotandchisel/robotandchisel.github.io/blob/master/assets/papers/HBRC-ros2.pdf - as you may be interested in other things you can put in that CYCLONEDDS_URI - such as the Discovery/Tag - which is a Cyclone specific improvement over just having a DOMAIN_ID (this basically gives us infinite domains, because now the domain is DOMAIN_ID + Tag has to match)

-Fergs

[Marco Walther]

And that might also be the solution for the 'CyclonDDS only wants to
work on one interface'
https://github.com/eclipse-cyclonedds/cyclonedds/blob/master/docs/manual/config.rst#network-and-discovery-configuration
.
I believe, you made your desktop and embedded AMD work together, but do
the Teensy & Nvidia still work as expected with your setup?

-- Marco

>
> ----- Original Message -----
> *From:* Michael Wimble <mailto:mwi...@gmail.com>
> *To:* hbrob...@googlegroups.com <mailto:hbrob...@googlegroups.com>
> *Sent:* Sunday, June 19, 2022 10:20 PM
> *Subject:* [HBRobotics] For suffers of dual-network interfaces on

> embedded Ubuntu processors trying to use Ros2
>
> If you are not trying to run both an Ethernet and wifi network at
> the same time, under Ubuntu Linux, for use with Ros2 across multiple
> computers (e.g., my desktop Mac and my embedded AMD processor), then
> this article is not for you. Nothing to see here. Go about your
> business.
>
> If your embedded robot computer is running Ros2 and is using only
> one network interface, typically the wifi, then launching your robot
> stack and then going to your desktop to run rviz or teleop_keyboard
> is a piece of cake. Do what it says in the docs, which is pretty
> much nothing special.
>
> In my robot, Puck, my main embedded processor, an AMD 3700X is doing
> the bulk of the work and talks to my house network over wifi. But,
> and here’s the gotcha, I have two other computers that talk over
> ethernet to that AMD processor: an Nvidia processor for AI and a
> custom processor based on the Teensy 4.1 that acts as a sensor
> processor and system monitor.
>
> For the computer to both talk to wifi and the ethernet, you need to
> set the “Make available to other users” option in the Details pane
> of the Wired Network panel.
>

> Then you need to assign a static IP subdomain to devices using the
> ethernet. Here I say that everything connected to the ethernet bus
> must have an IP address in the 10.42.0.X subdomain. For example, my
> custom monitor talks on the IP address of 10.42.0.132 and the AMD
> processor has an ethernet address of 10.42.0.132 Ignore that the
> photo below erroneously shows and address of 10.42.0.0 instead of
> 10.42.0.132. This will set up routing tables automatically so
> embedded traffic between the 10.0 and 10.42 subdomains can talk to
> each other.
>

> So far, this is all easy and you can probably find articles that
> tell you how to get this far. If you run the “ip address” command,
> you will see that my AMD processor has two network addresses:
> 10.0.0.48 which is the address when talking over the wlp38s0 wifi
> interface, and 10.42.0.0 when talking to the enp37s0 ethernet
> interface (that is wrong here, it should be set to 10.42.0.133, but
> that’s not important for this article).
>

> As soon as you turn on both the ethernet and wifi interfaces, if you
> look at the route table, you will see something like this:
>

> --
> You received this message because you are subscribed to the Google
> Groups "HomeBrew Robotics Club" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to hbrobotics+...@googlegroups.com

> <mailto:hbrobotics+...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/hbrobotics/BB63CDB6-63EE-4F61-90D3-3114E0992917%40gmail.com

> <https://groups.google.com/d/msgid/hbrobotics/BB63CDB6-63EE-4F61-90D3-3114E0992917%40gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "HomeBrew Robotics Club" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hbrobotics+...@googlegroups.com

> <mailto:hbrobotics+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/hbrobotics/0c0101d8846a%24a4b0f550%241f01a8c0%40NBE
> <https://groups.google.com/d/msgid/hbrobotics/0c0101d8846a%24a4b0f550%241f01a8c0%40NBE?utm_medium=email&utm_source=footer>.

[Alex Sy]

Hi Marco,
I do not have that setup that Michael had. However since it seems that it
is a common problem of trying to get multiple network interfaces to work, so
I gave the suggestion that worked for me in case it is useful.

Alex


----- Original Message -----

email to hbrobotics+...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/hbrobotics/f7d970a9-1592-4824-97a1-d1158f854f4e%40gmail.com.

[Michael Wimble]

Thanks again, Michael Ferguson for all the help you provide to this group.

[Michael Wimble]

I haven’t turned on the Nvidia in months, but the Teensy certainly works.

[Scott Horton]

Hi Michael,

Thanks for posting that info last summer. I'm trying to setup a similar networking architecture.

I just wanted to confirm, can you list and subscribe to topics published by the Nvidia and Teensy devices from your Mac? Or, are you only able to see those topics published by your AMD device?

Thanks,

Scott

[Michael Wimble]

The Nvidia was never an issue. So I believe all topics were visible. 

On Jan 29, 2023, at 1:05 PM, Scott Horton <horton...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hbrobotics/cd2a8739-08d1-43df-8dea-ca3800163abbn%40googlegroups.com.