Problem with ROS message-size larger than 64K Bytes on Windows version

[EricZ]

Hello, ROS2 experts,

We are having problem with ROS message-size larger than 640000 bytes on Windows platform which we hope to gain some insights.  Below are more details of the problem:

1) Our application overlay needs to publish ROS message size larger than 64000 bytes (94068 bytes ROS message size). We excluded the eProsima from the build by placing AMENT_IGNORE file under the eProsima directory.  The build was successful, but we hit a different run-time error which caused the program to crash in the createPublisher() function without explicit error messages.

2) Then we built for Windows platform with eProsima included in the build.  Our build was successful.  But we hit below run-time errors which caused the program to crash in the createPublisher() function, and seems to complain about the ROS messages larger than 64000 bytes:

[Error][ParticipantImpl::registerType]

[Error][ParticipantImpl::createPublisher]

2) The third thing we tried was to re-define the values for the "PAYLOAD_MAX_SIZE" to 128000 in the two SerializedPayload.h files found in "C:\dev\ros2\src\eProsima\Fast-RTPS\include\fastrtps\rtps\common\" and the "C:\dev\ros2\install\include\fastrtps\rtps\common" directories.  The build was successful, and we no-longer had the run-time errors.  However, we are not able to publish successfully.  From sniffing, we can tell that the RTPS protocol did not send correct message data.

4) Last but not least, we built on the Linux platform by excluding the eProsima from the build.  The build was successful. And we are able to run the application without run-time crashing, and it seems to be publishing fine - although we have not yet verified that the published data is all correct and complete.

So our question is - are ROS message sizes larger than 64000 bytes formally supported on either Linux or Windows platforms?  If so, then how do we fix the above Windows-side run-time problems we have been seeing?

Thanks,

Eric

[Morgan Quigley]

Hi Eric,

Message sizes larger than 64,000 are definitely supported, but as you've discovered, RTPS message fragmentation (that is, when the message payload is larger than a single 64K UDP datagram) is not (yet?) supported by FastRTPS.

However, OpenSplice should be willing to fragment and re-assemble messages just fine. Unfortunately, it sounds like your scenario is doing something different than what's in our test suite. We'd love to know more about it so that we can add this type of application to our CI farm and make sure it's fixed (and stays fixed). Can you provide source code that demonstrates the problem?

Best regards,

Morgan

--
You received this message because you are subscribed to the Google Groups "ROS SIG NG ROS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ros-sig-ng-ro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Baillio]

We are specifically trying to transmit the VelodyneScan message via OpenSplice.

VelodyneScan has lots of VelodynePackets which are essentially byte buffers of size 1208. So 70+ of those is definitely larger than the 64k limit.

We are probably going to put together a simple talker/listener pair that transmit dummy VelodyneScan. If we can recreate the problem, we can send you that code to test with.

Thanks.

[Morgan Quigley]

Hi Brad,

Thanks for the feedback. We don't currently have a Velodyne in our CI farm, so if there was a standalone program that transmits dummy Velodyne-like traffic and can reproduce the problem without the hardware, that would be very helpful. It's such an important sensor nowadays that we really want to be sure that ROS2 can handle it gracefully!

Best regards,

Morgan

[Dirk Thomas]

Hi Brad,

one of the current ROS 2 demos (https://github.com/ros2/ros2/wiki/Quality-Of-Service) already transfers images. The resolution and therefore the size of the messages is configurable with command line options. Even for higher resolutions which exceed 64k the messages can be passed using OpenSplice. So as far as we know this should work for other messages the same way. If you have a reproducible example where it doesn't work for you we would be happy to look into it. Maybe you can try to run the demo to check if that works for you.

Thanks,

- Dirk

[Brad Baillio]

Great idea. I know I've run that demo on windows before. We should compare the two examples and see if we're doing something wrong.

Thanks.

[Eric Zhu]

Hi, Dirk and Morgan,

Today, both Brad and I re-ran the cam2image tests in two different ways.  

- First test was to download the alpha4 binary version for windows, and run the cam2image off the box, and that test succeeded in publishing images.

- Second test was to run the cam2image command from our own ROS2 build from source code, and that test failed in publishing any images, and complaining about the vector exceeding maximum size.

Because our velodyne overlay application was built based on our own ROS2 build, we basically reproduced the same problem by using the cam2image demo app.  So far, we have not figured out the difference between our own ROS2 build versus the downloaded binary ROS2.  We even made sure the OpenSlice versions are the same between the binary and the source-build as advertised on the ROS2 website.

We also tried to take the binary download, copy everything into the ros2/install directory, and then build our Velodyne overlay that way, but unfortunately the build fails.

So the biggest question is: why the binary download and our own ROS2 build behave differently for cam2image?

Thanks,

Eric 

 

[Brad Baillio]

I think we figured it out. We had some weird dependency in our CMakeLists.txt file that was screwing things up.

Yea! Velodyne driver on Windows!

[Eric Zhu]

Right.  And we built the eProsima and introduced dependency to the rtps libraries.

Thanks everyone for the help!

[Morgan Quigley]

Hi Brad and Eric,

I'm glad to hear things are starting to work for you. That's awesome you have now Velodyne on Windows!

I'd be very interested to hear more about what went awry at first, so that we can improve the info/warning/error messages and start to put together something like the "roswtf" utility that exists in ROS1.

Best regards,

Morgan

[William Woodall]

Hi Eric,

It sounds like you might be running into an issue where you are actually running with FastRTPS, when you mean to run with OpenSplice. I could be wrong, but that's my best guess as to why this might happen, because:

I believe the error you are mentioning is only defined in the FastRTPS code base:

https://github.com/eProsima/ROS-RMW-Fast-RTPS-cpp/blob/1cfbe2e9cf626d6f6f5def103fece97d195814f6/rmw_fastrtps_cpp/src/TypeSupport.cpp#L182

Also because the only difference I can think of between the packaged binary and the from source build is that fastrtps is enabled by default in the later and explicitly disabled in the former.

This might be our fault, or a quirk of CMake that you didn't expect. I'd recommend putting the AMENT_IGNORE file in the FastRTPS, FastCDR, and FastRTPS rmw directory and then deleting the entire build and install folders, and rebuild. This shouldn't be strictly necessary, it's sort of the nuclear option, but it's a way to be absolutely sure you're not building with FastRTPS.

You can all print out the rmw implementation you're using from your code by using the API call: `rmw_get_implementation_identifier` (https://github.com/ros2/rmw/blob/master/rmw/include/rmw/rmw.h#L40-L43)

That way you can check that you're using the middleware you're expecting.

William Woodall

ROS Development Team

wil...@osrfoundation.org

http://wjwwood.io/

[Jack O'Quin]

On Tue, Apr 19, 2016 at 10:38 AM, Brad Baillio <brad.b...@gmail.com> wrote:

We are specifically trying to transmit the VelodyneScan message via OpenSplice.

VelodyneScan has lots of VelodynePackets which are essentially byte buffers of size 1208. So 70+ of those is definitely larger than the 64k limit.

As a temporary work-around, you could set the driver's ``npackets`` parameter to a lower value.

--

 joq

[Brad Baillio]

I noticed early on that letting the ament build process compile the FastRTPS stuff forced all code to use it. So all the code I wanted to write needed boost dependencies, etc. Even when I ran the opensplice-specific variant of the executable: myTalkerNode__rmw_opensplice_cpp.

So early on I AMENT_IGNORE'd the whole eProsima directory. I figured if we ever decided to run on bare metal, we could reinvestigate.

Not sure why I had that experience. I assume if I download the binaries off the Ros2 GitHub, that FastRTPS was compiled into that directory, and those examples all work.

This was the main culprit in our CMakeLists.txt file that caused all the issues:

include_directories(include ../../../examples/include ${rosidl_generator_cpp_INCLUDE_DIRS}/rosidl_typesupport_introspection_cpp/impl)

I can't speak for Eric and why he thought he needed to put that ${rosidl_generateor_cpp...} stuff in the include, but it definitely broke things.

Once we removed that, things worked as expected.