Send large files over bridged eventbus

[Do Bo]

Hey everyone,

as you can tell from the subject, I'm playing around with the vertx eventbus and I'd like to ask you for any advice or hints on my solution.

The given scenario is that I'd like to send large file dumps (1-2GB) over a bridged (TCPEventbusBridge) eventbus. 

So far I've had 2 approaches, which I'm not totally confident about.

• Approach 1: Dedicated Worker verticle reads from file chunks (FileInputStream) and sends them encoded in JSON to the destination verticle
  --> Problem: JVM goes out of memory, because the eventbus is congested due to the high amount of JSON massages
  
  
• Approach 2: Every file chunk is requested and acknowledged sequentially, such that there are never 2 chunks on the bus at the same time
  --> Probem: I haven't implemented this,yet. But it seems to have a high impact on the performance

Topology:

A------> B -----------------------------> C

                 TCPEventbusBridge

The files are stored on A and should be received by C. 

How do you guys implement this? Is there any built-in way to accomplish my filetransfers? Maybe using the pump?

Thanks, Dom

[Paulo Lopes]

Sending such a big payload over the EB will probably be a bad idea since the bridge will keep all the payload content in memory and once it is received and converted to JSON sent over the network to the destination node, so you might be flooding the network/cluster.

I'd say use alternative 2 or maybe create a file upload mechanism say a REST API that just dumps the upload to a shared network drive and notify the node that needs to consume this file once the upload is complete with a simple message pointing to its location on disk...

[Dom B]

Hi Paulo,

thanks for your fast reply. I will go the REST-way and report my results later on.

[David Zaffery]

Dom, 

I would suggest a different approach if you need to transmit large files in a stateless way.  Basically, create a task that specifically is listening for requests from clients to receive the file(s).  Then send a message via the EventBus with the connection information.  Any listeners receiving the message can simply connect to that channel/socket and receive the file(s).  Just make sure that you create your fileserver task that, 1. its on thread that doesn't block the event loop, 2. that you have a way to handle multiple connection/download requests concurrently. 

Its a nice stateless way to transfer large amounts of data in a seemless way. 

I used this approach back in the 90's using C/UNIX platforms and it worked very well. 

Regards, 

Dave