Handling large data via RPC (crossbar + autobahn|python)

[Ciclet Adrien]

Hello,

I am trying to transmit large data through websockets using crossbar/autobahn's RPC. My setup is as follow:

- Python 2.7

- A crossbar router (version 17.8.1.post1)

- A back-end that will try to send a large pandas DataFrame as a json string

- A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will return a large string.

class MyComponent(ApplicationSession):

    @inlineCallbacks
    def onJoin(self, details):
        print("session ready")
        try:
            res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error: ApplicationError(error=<wamp.close.transport_lost>, args=[u'WAMP transport was lost without closing the session before'], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead to him, but I thought that autobahn would chunk my data and that the call would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket treatment; thanks to that I was able to transmit larger amount of data but eventually I would hit a limit (config file largely copied and pasted from sam & max).

                            "options": {
                                "enable_webstatus": false,
                                "max_frame_size": 16777216,
                                "auto_fragment_size": 65536,
                                "fail_by_drop": true,
                                "open_handshake_timeout": 2500,
                                "close_handshake_timeout": 1000,
                                "auto_ping_interval": 10000,
                                "auto_ping_timeout": 5000,
                                "auto_ping_size": 4,
                                "compression": {
                                    "deflate": {
                                        "request_no_context_takeover": false,
                                        "request_max_window_bits": 11,
                                        "no_context_takeover": false,
                                        "max_window_bits": 11,
                                        "memory_level": 4
                                   }
                                }
                            }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

[Adam Jorgensen]

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

[Tobias Oberstein]

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:
> This sounds like a job for progressive results?
>
> See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download via
WAMP".

side note: for the opposite direction, that is "large file upload", is
missing the "progressive call arguments" feature in CB still (and in the
WAMP spec). it is straight forward, but needs work/time.

[Adam Jorgensen]

While progressive call parameters are missing you could probably mimic it by reversing the direction with a workflow like:

caller calls x.y.z.upload and includes a parameter that contains a WAMP URI for an end-point it has registered to handle the upload

callee for x.y.z.upload end-point process initial upload details and then calls the URI it received from the initial caller in order to retrieve the chunks for the upload part

Once callee's call to the custom URI is done it completes and the original caller continues executing past it's call to x.y.z.upload and maybe unregisters it's custom end-point for the "upload"?

I imagine I could probably wrap this functionality up in a decorator :-)

[Tobias Oberstein]

Am 10.08.2017 um 10:33 schrieb Adam Jorgensen:
> While progressive call parameters are missing you could probably mimic it
> by reversing the direction with a workflow like:

yeah, right. that would work;) it is a slight hack still .. but a clever one

[Adam Jorgensen]

I think on the level of hackyness it's not thaaaaaaaaaat bad ;-) 

I think it's more a side-effect of the flexibility of WAMP and the way it decouples your application components from concepts like server-side and client-side :-)

[Tobias Oberstein]

Am 10.08.2017 um 10:42 schrieb Adam Jorgensen:
> I think on the level of hackyness it's not thaaaaaaaaaat bad ;-)
>
> I think it's more a side-effect of the flexibility of WAMP and the way it
> decouples your application components from concepts like server-side and
> client-side :-)

yeah, absolutely, I agree;)

this, lets call it work-around, is only possible because the artificial
"client vs server" distinction is gone with WAMP.

it seems somewhat hard to grasp for developers in general, because we've
got used to this client/server thinking so much.

but it is one of the major strength of WAMP. it's all about components
talking peer-to-peer (at logical level), rather "client/server".

[Adam Jorgensen]

Indeed, the peer-to-peer nature of WAMP and Crossbar is one of the reasons I'm so keen on it. And in practise it's proven to be really powerful to work with :-)

[Ciclet Adrien]

Thank you for the suggestion, it was indeed the solution. I am just sad to have to chunk the result myself.

For other having the same question the link to the example is https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/wamp/rpc/progress

[Tobias Oberstein]

Hi,

the configuration options in Crossbar.io are for automatic fragmentation
of WebSocket messages.

However, a big WebSocket message, even when fragmented, will still
"head-of-line" block other WebSocket messages that come after.

This is how (plain) WebSocket works (nothing Crossbar.io specific).

Since WAMP messages are mapped 1:1 to WebSocket messages, a big WAMP RPC
return message will effectively hinder any other WAMP message (such as
eg an EVENT) to arrive at the client until the formter WAMP message has
been completely received.

And hence the progressive Call Results feature of WAMP.

It does not make sense IMO to do that inside a WAMP client library,
because imagine you want to send a 1GB file.

What a user probably wants to do is NOT read the 1GB into RAM to chunk
it, but read chunks directly from the file.

Or: imagine you have mmap'ed the file and want to get chunks from there.

etc etc

Cheers,
/Tobias