wamp router drops connection to cppwamp component due to malformed message

[Rosa Lisin]

Hello,

I've been using cppWamp to create various backend components.  Components have been running without any wamp-related issues for over a year, which gives me great confidence in the library. However, there is an issue that has started happening recently, probably because of the data load increase:  one of the components gets disconnected from the router (crossbar.io wamp router), due to received malformed message from the cppWamp component:

[Router      12676 crossbar.router.protocol.WampRawSocketServerProtocol] WampRawSocketProtocol: WAMP Protocol Error (invalid serialization of WAMP message ('utf8' codec can't decode byte 0xb2 in position 196607: invalid start byte)) - aborting connection

 [Router      12676 crossbar.router.protocol.WampRawSocketServerProtocol] failing RawSocket connection - message length exceeded: message was 826684728 bytes, but current maximum is 1048576 bytes

This problem happens when the cppwamp component simultaneously finishes executing remote procedure that returns pretty big result (around 200K) and it tries to publish an event.

I have turned on tracing on cppwamp session, but there are are no "****WAMP Trace: Tx message:" before the disconnection. 

However, given the ridiculous message length that the router is complaining about, and invalid start byte, I assume that the message that was send from the component is malformed. 

Is this is a known issue? If not, please advise on further troubleshooting.

Thanks,

Rosa

PS I am using 0.6.3 version of cppwamp

[Emile Cormier]

Hi Rosa,

I've been using cppWamp to create various backend components.  Components have been running without any wamp-related issues for over a year, which gives me great confidence in the library. 

That's nice to hear! CppWAMP has been rock-steady for our project as well.

However, there is an issue that has started happening recently, probably because of the data load increase:  one of the components gets disconnected from the router (crossbar.io wamp router), due to received malformed message from the cppWamp component:

No, I am not aware of this issue. Nobody has reported anything similar on the CppWAMP GitHub.

This problem happens when the cppwamp component simultaneously finishes executing remote procedure that returns pretty big result (around 200K) and it tries to publish an event.

The CppWAMP session objects are not thread-safe. You cannot invoke session object methods simultaneously from multiple threads.

Can you clarify what you mean by "simultaneously" and which Session/CoroSession methods are involved? When you say "executing remote procedure", I don't know if you mean calling an RPC, or handling it. It would helpful if you showed the sequence of Session/CoroSession method calls.

 

I have turned on tracing on cppwamp session, but there are are no "****WAMP Trace: Tx message:" before the disconnection. 

However, given the ridiculous message length that the router is complaining about, and invalid start byte, I assume that the message that was send from the component is malformed. 

You can try sniffing the raw TCP data using WireShark. If you're using JSON encoding, then the messages will be easy to read in plain text. The WAMP protocol specification explains how each message is delimited (a length prefix is sent before each message).

Is this is a known issue? If not, please advise on further troubleshooting.

What I suspect might be happening is that the event publishing is not waiting for its turn and is interfering with the RPC message. Before the RPC message is finished being transmitted over the wire, the PUBLISH message cuts in on the TCP stream.

So instead of this pattern over the TCP stream:

|RPC Length Prefix| Long RPC Message | Publish Length Prefix | Publish Message |

We get this:

|RPC Length Prefix| Partial RPC Message | Publish Length Prefix | Publish Message | Remainder of RPC Message

Having the publish message spliced in the middle of the RPC message will surely lead to the router deserializer failing.

If the event publish originates from a different thread than the one running the Session/CoroSession's io_service, then it must be posted to the io_service being used by the Session/CoroSession object:

sessionIoService.post([session, eventInfo]()

{

    session->publish(Event("foo").withArgs(eventInfo));

});

 

Thanks,

Rosa

PS I am using 0.6.3 version of cppwamp

The changes since the 0.6.3 release only involves Variant serialization and should have no bearing on your problem.

Hope this helps,

Emile

[Rosa Lisin]

Hi Emilio, 

Thanks for the prompt response. I apologize for not not being clear enough.  By "simultaneously" I mean the following scenario: Component 1 registers procedure Func (by calling enroll), and it also subscribes to handle event E1. Event handler for E1 results in publishing event E2. At some point Component 2 calls remote procedure Func  and at the same time component 3 publishes event E1. That results in Component 1 executing Func() and publishing E2 (from event handler for E1). At that point Component 1 gets disconnected from the router. 

How would you recommend handling that situation to avoid message corruption?

Thanks,

Rosa

[Emile Cormier]

I had to draw it out on paper to see all the interactions, but I understand it now, thanks.

Just to be clear, there's no multithreading involved in component 1, right?

I'll investigate into this, but I cannot make any promises on the time frame. It would speed things up if you could provide a minimal working example that reproduces the problem (something that I can compile and run myself on a single machine). This is a problem that could potentially affect our our application that uses CppWAMP, but I have other high-priority matters to attend to right now.

In the meantime, you can try making your Component1-E1 handler perform the publish operation within an io_service::post:

sessionIoService.post([&session, &publishInfo]()

{

    session->publish(Event("E2").withArgs(publishInfo));

});

If this works, it's just a band-aid solution that does not really resolve the problem within CppWAMP itself.

[Emile Cormier]

Sorry, that should be

sessionIoService.post([session, publishInfo]()

{

    session->publish(Event("E2").withArgs(publishInfo));

});

You'll definitely want to capture by copy to avoid dangling references when the lambda is eventually executed.

[Rosa Lisin]

Hi Emile,

There is no multithreading involved in Component1.  I will try to come up with the working example , and try using post.

 Thank you for the help.

Rosa

[Emile Cormier]

Hi Rosa,

I don't know if you're still using CppWAMP, but I though I'd let you know that we've recently encountered a similar problem to yours in our parent project that uses CppWAMP.

We need to fix this problem for our own purposes, so hopefully that will help you as well once it's resolved. This problem is now high up in our priority list.

Cheers,

Emile

[john.os...@butterflyenergy.ca]

Hi Rosa,

We figured out what is causing the issue (https://github.com/ecorm/cppwamp/issues/117). You can try the patch from my fork:

https://github.com/postry/cppwamp/commit/1b87823543de16e9b8e27784cb014ff925749e97

Best regards,

John

[Emile Cormier]

The latest commit of CppWAMP should now fix the problem.

[Emile Cormier]

I'd like to give credit to John Ostrowski for his hard work in troubleshooting the problem. He spent much time consulting the WAMP spec, corresponding with the Crossbar team, debugging Crossbar & CppWAMP, and analyzing wire traffic.

Because of the urgency of solving this problem (due to a company deadline), and because I work different time shifts than John, it was not possible for me to walk him through the changes needed in his pull request in order for the CppWAMP unit tests to pass.

I therefore had to proceed with my own solution, which I had independently drafted before I saw his. Because John had isolated the problem to a specific module, it was easy for me to come up with my own independent solution.

The lion's share of the effort was in the troubleshooting the problem, and thank John for his hard work.

[Rosa Lisin]

Hi Emilio,

Yes, we are still using CppWamp, but we were avoiding that problem by using the workaround that you've proposed. I am glad you were able to fix the issue. We will start using the patched version of the library.

Thanks,

Rosa

. 

[Rosa Lisin]

Thanks a lot! We will start using the patched version of the library ASAP.

Rosa