Is it wise to stream file over vertx event bus

[tharindu ranasingha]

We have set of services in a vertx cluster. We serve web front end through a API gateway which is one service within the cluster. Client ask a requirement for download some data as a CSV file. It should be transmitted as bellow.

Service A --(Event bus)---> API gateway ---(Web socket)---> Browser

My question is, is it wise to stream such file over event bus from Service A to API gateway? (File may get as large as 100 MB)

[Julien Viet]

Hi,

this is not recommended

sending 100MB on the event bus will these MB on a multiplexed channel (all event bus events) but without fragmentation.

even with fragmentation that can potentially make a lot of traffic which could be fine if your TCP is well configured and you have a great network, but that can potentially mean a bottleneck.

So that will keep the connection busy, specially if you send a lot of them.

In addition it will not take in account back pressure as you will need to load this as a Buffer in memory, consuming lot of heap.

As far as I can see your API gateway is written with Vert.x, is that correct ?

Julien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a60ad543-762a-478b-9a58-9757d78b929f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tharindu ranasingha]

Yes api gateway is written with vertx.

If it is a bottleneck to event bus we need to use a separate method to transfer file. (Probably via a HTTP response).

We may have to work on write the data onto disk at one request with the same path. Then send that file over http request.

Thanks for your advice

[Julien Viet]

Hi,

Since the gateway split the file in chunks to send them via websocket

I think then one possibility is to send chunks from ServiceA directly and then the gateway only has to forward the chunks to the websocket.

WDYT ?

Julien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a7585e5e-9aa2-4676-9cfc-d42ad47b5766%40googlegroups.com.