My fix for cycloneddds and large messages (and slow discovery)

[Michael Wimble]

Subject: CycloneDDS gotcha — large ROS 2 topics (images/pointclouds) failing over WiFi/Ethernet

Hi all,

I just spent a while chasing a nasty CycloneDDS bug on my robot (two machines,

one on WiFi, one on Ethernet, both running Ubuntu + ROS 2 with the default

`rmw_cyclonedds_cpp` middleware) and wanted to share it since it's a generic

DDS/network config issue, not specific to any particular processor. It'll bite

you the same way whether you're running an AMD box, an Intel NUC, or a

Raspberry Pi — the only thing that matters is that CycloneDDS is talking over

Ethernet/WiFi with a normal ~1500-byte MTU.

## The symptom

Large topics — camera images, point clouds, depth images — would fail

intermittently or sometimes completely. Small topics (odom, tf, cmd_vel) were

always fine. Symptoms included:

- `ros2 topic hz` on an image topic freezing after a few messages

- rviz2 showing a handful of frames then nothing

- Node logs showing something like:

`ddsi_udp_conn_write ... failed with retcode -58`

`-58` is `DDS_RETCODE_NOT_ENOUGH_SPACE`, which is CycloneDDS's wrapper around

the POSIX `EMSGSIZE` errno — i.e. "you tried to send a UDP datagram bigger

than the kernel will allow in one `sendmsg()` call."

## The wrong instinct (I fell for this first)

My first assumption was: "images are big, so I should raise

`General/MaxMessageSize` in `cyclonedds.xml` to something generous like 16MB."

**This makes things worse, not better.** `MaxMessageSize` is *not* "the

maximum size of a sample/image CycloneDDS can send." Per CycloneDDS's own

docs, it's *"the maximum size of the UDP payload Cyclone DDS will generate"* —

i.e. how much it will try to stuff into a single `sendmsg()` call. Set it to

16MB and CycloneDDS will try to write single UDP datagrams far larger than any

real UDP/IP stack supports, and the kernel will reject every single one with

`EMSGSIZE`. In my case this caused **100% failure**, even on localhost with no

network involved at all.

The setting that actually controls the max size of a sample (image, point

cloud, etc.) CycloneDDS is willing to reassemble is `Internal/MaxSampleSize`,

which defaults to effectively unlimited (~2GB). You almost never need to touch

this one.

## The actual fix

Keep `General/MaxMessageSize` **at or below your path MTU** (1500 bytes for

plain Ethernet/WiFi) — I used **`1470 B`**. Example `~/.ros/cyclonedds.xml`

snippet:

```xml

<CycloneDDS>

<Domain>

<General>

<!-- Keep below the 1500-byte MTU so CycloneDDS never needs OS-level

IP fragmentation - let its own DDSI FragmentSize below handle

chopping up large samples reliably instead. -->

<MaxMessageSize>1470B</MaxMessageSize>

<FragmentSize>1200B</FragmentSize>

</General>

</Domain>

</CycloneDDS>

```

Why this specific range matters — there are really three zones:

1. **Above ~65507 bytes or so, or just "too big" in general** → guaranteed

`EMSGSIZE` on every large message (my 16MB mistake).

2. **Between the MTU (1500) and the OS UDP max (~65507)** → works fine on

localhost, but forces the *kernel* to fragment each UDP datagram at the IP

level to fit the physical MTU. This works fine over Ethernet but **WiFi

drops IP-fragmented packets much more aggressively** than DDSI-level

fragments, causing the "one lost fragment = whole image lost" intermittent

failures. This is the value CycloneDDS ships as its *own default*

(14720B!), so don't assume the out-of-the-box config is safe for WiFi.

3. **At or below the MTU (1470B is a safe margin under 1500)** → CycloneDDS

never asks the kernel to fragment anything at the IP layer. Instead it

relies entirely on its own DDSI-level fragmentation (`FragmentSize`,

~1200B) which has proper per-fragment reliability/retransmission built

into the RTPS protocol. This is what you want.

## How to check what you're actually running

CycloneDDS reads its config from the file pointed to by the `CYCLONEDDS_URI`

environment variable, which usually defaults to `~/.ros/cyclonedds.xml`. Watch

out for a **stale system-wide `/etc/cyclonedds.xml`** too — if one exists it

can silently take precedence depending on how `CYCLONEDDS_URI` is (or isn't)

set, so it's worth checking both locations and removing/backing up whichever

one isn't your intended active config.

## How to verify the fix worked

```bash

ros2 daemon stop && ros2 daemon start # picks up the edited config

ros2 topic hz /your/large/image/topic

```

Watch the publishing node's log for `retcode -58` / `ddsi_udp_conn_write ...

failed` — you should see zero of these after the fix, both on localhost and

across machines over WiFi.

Hope this saves someone else the couple of hours it cost me. Happy to help if

anyone hits the same thing — feel free to reach out.

So, what to do:

* I removed /etc/cyclonedds.xml from each machine (if you have one).

* I created ~/.ros/cyclonedds.xml on each machine. Here is my particular contents: <?xml version="1.0" encoding="UTF-8"?> <CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaL ocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/c yclonedds.xsd"> <Domain> <General> <!-- Use multicast for discovery and add unicast peers for reliability --> <AllowMulticast>true</AllowMulticast> <!-- IMPORTANT: keep MaxMessageSize BELOW the real network MTU (1500). See cyclonedds.xml in this directory (or CYCLONEDDS_ISSUE_ANALYSIS.md in the Sigyn repo root) for the full explanation. Do NOT raise this to "fix" large image/pointcloud delivery - it makes things worse. --> <MaxMessageSize>1470 B</MaxMessageSize> <FragmentSize>1200 B</FragmentSize> <!-- Ethernet interface on amdc desktop --> <Interfaces> <NetworkInterface name="eno1" priority="default" multicast="true"/> </Interfaces> </General> <Discovery> <!-- Add explicit peer addresses for unicast discovery --> <Peers> <Peer address="192.168.86.28"/> <!-- amdc (self) --> <Peer address="192.168.86.109"/> <!-- sigyn7900a robot --> </Peers> <ParticipantIndex>auto</ParticipantIndex> </Discovery> <Internal> <SocketReceiveBufferSize min="10MB"/> <SocketSendBufferSize min="10MB"/> </Internal> </Domain> </CycloneDDS>

Don't use my <Peer address.../> entries, use your own. This makes cyclonedds use unicast discovery instead of multicast (I believe). It's much faster.

Make it a list of all IP address in your robot network that might want to talk over cyclonedds.

You build this file for every one of your computers (robot and desktop in this example).

Also fix the "<NetworkInterface name=" entry to the name is the name of the network device you're using on that computer. That means that cyclonedds.xml might different for each computer. My desktop uses "eno1", my robot uses "wlp8s0".

* Add a line in your ~/.bashrc or in ~/.bash_aliases (I prefer ~/.bash_aliases but either will work); if you use zsh instead of your are already smart enough to figure out where to put this export line: export CYCLONEDDS_URI=file:///etc/cyclonedds.xml

* If you aren't restarting after this, re-source the bash setup so you get the CYCLONEDDS_URI as an environment variable: source ~/.bashrc

echo $CYCLONEDDS_URI

You should see: file:///home/ros/.ros/cyclonedds.xml

(where ros (not .ros) will be replace with your user name) as a result of the echo command. * If you aren't restarting after this, make sure you kill all ros nodes on all machines (ros2 node list should show no nodes running),

and then kill the ros2 daemon on all machines (ros2 daemon stop). That way all the cyclonedds instances will use the new configuration.

— Wimble Robotics

[Marco Walther]

And as always, keep *all* your messages as small a you can;-) It's very
easy to fill up your throughput 'space' and at that point, no amount of
retry logic will really reliably safe you.

-- Marco

On 8/8/26 12:49, Michael Wimble wrote:
> Subject: CycloneDDS gotcha — large ROS 2 topics (images/pointclouds)
> failing over WiFi/Ethernet
> Hi all,
> I just spent a while chasing a nasty CycloneDDS bug on my robot (two
> machines,
> one on WiFi, one on Ethernet, both running Ubuntu + ROS 2 with the default
> `rmw_cyclonedds_cpp`middleware) and wanted to share it since it's a generic
> DDS/network config issue, not specific to any particular processor.
> It'll bite
> you the same way whether you're running an AMD box, an Intel NUC, or a
> Raspberry Pi — the only thing that matters is that CycloneDDS is talking
> over
> Ethernet/WiFi with a normal ~1500-byte MTU.
> ## The symptom
> Large topics — camera images, point clouds, depth images — would fail
> intermittently or sometimes completely. Small topics (odom, tf, cmd_vel)
> were
> always fine. Symptoms included:
> -`ros2 topic hz`on an image topic freezing after a few messages

> -rviz2 showing a handful of frames then nothing
> -Node logs showing something like:

> Keep `General/MaxMessageSize`**at or below your path MTU**(1500 bytes for

> plain Ethernet/WiFi) — I used **`1470 B`**. Example `~/.ros/cyclonedds.xml`
> snippet:
> ```xml
> <CycloneDDS>
> <Domain>
> <General>
> <!-- Keep below the 1500-byte MTU so CycloneDDS never needs OS-level
> IP fragmentation - let its own DDSI FragmentSize below handle
> chopping up large samples reliably instead. -->
> <MaxMessageSize>1470B</MaxMessageSize>
> <FragmentSize>1200B</FragmentSize>
> </General>
> </Domain>
> </CycloneDDS>
> ```
> Why this specific range matters — there are really three zones:

> 1.**Above ~65507 bytes or so, or just "too big" in general**→ guaranteed

> `EMSGSIZE`on every large message (my 16MB mistake).

> 2.**Between the MTU (1500) and the OS UDP max (~65507)**→ works fine on
> localhost, but forces the *kernel*to fragment each UDP datagram at the IP

> level to fit the physical MTU. This works fine over Ethernet but **WiFi
> drops IP-fragmented packets much more aggressively** than DDSI-level
> fragments, causing the "one lost fragment = whole image lost" intermittent
> failures. This is the value CycloneDDS ships as its *own default*
> (14720B!), so don't assume the out-of-the-box config is safe for WiFi.

> 3.**At or below the MTU (1470B is a safe margin under 1500)**→ CycloneDDS

> never asks the kernel to fragment anything at the IP layer. Instead it
> relies entirely on its own DDSI-level fragmentation (`FragmentSize`,
> ~1200B) which has proper per-fragment reliability/retransmission built
> into the RTPS protocol. This is what you want.
> ## How to check what you're actually running
> CycloneDDS reads its config from the file pointed to by the `CYCLONEDDS_URI`
> environment variable, which usually defaults to `~/.ros/cyclonedds.xml`.
> Watch

> out for a **stale system-wide `/etc/cyclonedds.xml`**too — if one exists it

> can silently take precedence depending on how `CYCLONEDDS_URI`is (or isn't)
> set, so it's worth checking both locations and removing/backing up whichever
> one isn't your intended active config.
> ## How to verify the fix worked
> ```bash

> ros2daemonstop&& ros2daemonstart# picks up the edited config

> ros2topichz/your/large/image/topic
> ```
> Watch the publishing node's log for `retcode -58`/ `ddsi_udp_conn_write ...
> failed` — you should see zero of these after the fix, both on localhost and
> across machines over WiFi.
> Hope this saves someone else the couple of hours it cost me. Happy to
> help if
> anyone hits the same thing — feel free to reach out.

> *So, what to do*:
> * I removed *//etc/cyclonedds.xml/* from each machine (if you have one).
> * I created */~/.ros/cyclonedds.xml/* on each machine. Here is my

> * Add a line in your */~/.bashrc/* or in */~/.bash_aliases/* (I prefer
> */~/.bash_aliases/* but either will work); if you use zsh instead of

> your are already smart enough to figure out where to put this export

> line: */export CYCLONEDDS_URI=file:///etc/cyclonedds.xml/*

> * If you aren't restarting after this, re-source the bash setup so you

> get the CYCLONEDDS_URI as an environment variable: /*source ~/.bashrc*/
> /*echo $CYCLONEDDS_URI*/
> You should see: */file:///home/ros/.ros/cyclonedds.xml/*
> (where *ros*/(not *.ros)* will be replace with your user name) /as a
> result of the */echo/* command. * If you aren't restarting after this,
> make sure you kill all ros nodes on all machines (*/ros2 node list/*

> should show no nodes running),

> and then kill the ros2 daemon on all machines (*/ros2 daemon stop/*).

> That way all the cyclonedds instances will use the new configuration.
> — Wimble Robotics
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "HomeBrew Robotics Club" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hbrobotics+...@googlegroups.com
> <mailto:hbrobotics+...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/
> hbrobotics/d7a6d8bf-754c-4afc-9437-0d295284715d%40gmail.com <https://
> groups.google.com/d/msgid/hbrobotics/
> d7a6d8bf-754c-4afc-9437-0d295284715d%40gmail.com?
> utm_medium=email&utm_source=footer>.

[James H Phelan]

Michael,

I had a problem with sometimes loosing WiFi contact with the robot half way across the house.
The Raspberry Pi's built-in WiFi is buried in the middle of Stormy the Stingray sandwiched between the 52Pi Power Board above and the SSD hat below.

So I got a USB WiFi dongle https://www.amazon.com/dp/B078NSSM7W

Working closely with Claude-Opus-4.7 it was theorized and demonstrated that the internal WiFi and the USB WiFi would compete. The internal WiFi was disabled.

Supposedly "The Raspberry Pi 5 has a well-documented interference problem on 2.4 GHz".  This, too, was disabled.

We conducted an interesting experiment* logging the WiFi strength and the particular router in use as Stormy was driven to the opposite end of the house and back. I expected that it would use my study router and then pick up the other downstairs router in the master closet. But no. It picked up the router in the upstairs sewing room at the other end of the house. It never used the master closet router. I therefore moved that router to an ethernet port in the breakfast room in about the middle of the house.

* using Claude's modification of Sergei's clone of your WiFi logger

Similar to yours, we set up a configuration so that the other computers would only seek the USB WiFi adapter.  I'll be adding your two lines to it. 

What about the

<SocketReceiveBufferSize min="10MB"/> 

<SocketSendBufferSize min="10MB"/>

ubuntu@Stingray:~$ echo $CYCLONEDDS_URI
file:///home/ubuntu/cyclonedds.xml
ubuntu@Stingray:~$ ls /etc/cyclonedds*
ls: cannot access '/etc/cyclonedds*': No such file or directory
ubuntu@Stingray:~$ ls cyclonedds.xml
cyclonedds.xml
ubuntu@Stingray:~$ cat cyclonedds.xml
<CycloneDDS>
  <Domain>
    <General>
      <Interfaces>
        <NetworkInterface name="wlx90de801012a6" priority="10"/>
      </Interfaces>
      <AllowMulticast>false</AllowMulticast>

     <MaxMessageSize>1470B</MaxMessageSize>

     <FragmentSize>1200B</FragmentSize>

    </General>
    <Discovery>
      <ParticipantIndex>auto</ParticipantIndex>
      <Peers>
        <Peer address="192.168.68.66"/>
        <Peer address="192.168.68.99"/>
      </Peers>
    </Discovery>
  </Domain>
</CycloneDDS>

See attached Builders Manual Chapter 9 and WiFi_Field_Notes.txt extracted by Claude from my multi-year, 900pp Stingray Experience log.

James

James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/d7a6d8bf-754c-4afc-9437-0d295284715d%40gmail.com.