Download large files using vertx

[Anil]

HI,

is there any in vertx to support stream response like in jersey which supports download of large files ?

Please advice.

Regards,

Anil

[Julien Viet]

Hi,

you need to create a Pump between an HttpServerResponse (WriteStream) and an AsyncFile (ReadStream), it will do that for you.

http://vertx.io/docs/vertx-core/java/#streams

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/07f45382-89cd-40e6-b14d-69ab3d7810ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anil]

Thanks Julien. I will try that.

[Anil]

Hi Julien,

I tried file download using pump with async file and http response. 1kb file download is taking long time. 

public class Download extends AbstractVerticle{

public static void main(String[] args) {

Vertx vertx = Vertx.vertx();

vertx.deployVerticle(new Download());

}

@Override

public void start() throws Exception {

Router router = Router.router(vertx);

router.route(HttpMethod.GET, "/download").handler(rtx -> {

System.out.println("download started");

HttpServerResponse response = rtx.response();

response.setChunked(true);

// C:/Anil/dev_config.json

vertx.fileSystem().open("C:/Anil/dev_config.json", new OpenOptions(),  file -> {

AsyncFile asyncFile = file.result();

Pump pump = Pump.pump(asyncFile, response);

response.closeHandler(res -> { System.out.println("completed");});

response.exceptionHandler(ex -> 

{

System.err.println("exception"); 

ex.printStackTrace();

});

pump.start();

});

});

vertx.createHttpServer().requestHandler(router::accept).listen(8080, res -> {

if(res.succeeded()){

System.out.println("listenting to port 8080");

}else {

System.out.println("port is busy");

}

});

}

}

Could you please review the above snippet ? and correct me if i am wrong.

I am trying to build rest interface to access files ( > 10 mb). Do you recommend anything ? Thanks.

Regards,

Anil 

[Julien Viet]

why don’t you simply use response.sendFile() ?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/62c3eb43-fae4-43df-9a71-d107487d6ed0%40googlegroups.com.

[Tim Fox]

+1

Also the code below isn't ending the response.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5BAE7512-973A-4FA6-94B8-FF3DE3003791%40julienviet.com.

[Anil]

Hi Tim and Juliet,

Thank you. its working now.

Do you think serving files (using sendFile/Async file)  in standard verticle have impact on throughput ? 

how to transfer the large files which are in some data source (any nosql other than filesystem) as sending large content though event bus is not recommended ? using web-socket handler ? please advice.

Regards,

Anil

[Anil]

Any ideas ? Thanks.

[Micael Pedrosa]

I'm having similar problems (version 3.5.1) using sendFile with sizes > 600MB in my system.

It stops downloading at a certain point with no errors.

[Grey Seal]

I have tried dowloading csv file (40 mb) , fetching the records from database and then sending the output as CSV. When I access the url http://localhost:8181/api/csv for the first time, it's working but after that any subsequent calls to the url are throwing IllegalStateException. I am using vertx 3.5.0.

Code is at: https://github.com/greyseal/vertx-csv-download

Apr 18, 2018 12:16:14 PM io.vertx.core.impl.ContextImpl

SEVERE: Unhandled exception

java.lang.IllegalStateException: Response has already been written

at io.vertx.core.http.impl.HttpServerResponseImpl.checkWritten(HttpServerResponseImpl.java:572)

at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:598)

at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:293)

at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:58)

at io.vertx.core.streams.impl.PumpImpl.lambda$new$1(PumpImpl.java:64)

at io.vertx.core.eventbus.impl.BodyReadStream.lambda$handler$0(BodyReadStream.java:46)

at io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:212)

at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:189)

at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$3(EventBusImpl.java:538)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:344)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

at java.base/java.lang.Thread.run(Thread.java:844)

[Julien Viet]

can you try with latest Vert.x SNAPSHOT ?

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8353f758-ce40-4660-b415-b44745147b2b%40googlegroups.com.

[Grey Seal]

Apologies for the late post. Finally, I was able to resolve it. While doing performance tests, I am getting too many threads blocked warnings.

WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2840 ms, time limit is 2000

Also, the stats for Heap memory (touching 3GB) and Old gen is too high.  (https://github.com/greyseal/vertx-csv-download)