Getting into the endHandler of the event but the pump is still pumping the file

[shell...@gmail.com]

Hi,

I wrote a simple client for downloading files from a web server using vertx client.

I'm doing something like this:

HttpClient httpClient = vertx.createHttpClient(options);

httpClient.get(url, response-> {

    response.pause();

    vertx.fileSystemOpen(localPath, new OpenOptions(), openFileEvent() -> {

         AsyncFile file = OpenFileEvent.result();

         Pump downloadPump =  Pump.pump(response, file);

         downloadPump.start;

         response.resume();

         response.endHandler(finishedPumpEvent -> file.flush.close(closeEvent-> logger.debug("error while trying to close the file")))

    });

});

When I'm checking the hash of the files after the download I see a different hash because the endHendler is called before the end of the pumping and the flushing is done before it should.

Is this a known bug? I've added a 1-second sleep() for a workaround but it's only a temporary solution. (that actually works!).

Any idea what is happening here? 

Thanks,

Shelly Bar-On

[Julien Viet]

Hi,

do you mean that the HttpClientResponse#handler is called after the HttpClientResponse#endHandler ?

what do you call end of pumping ?

Julien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b7514032-e2d8-497f-8de2-10e6cd7e067c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[shell...@gmail.com]

I would expect that the HttpClientResponse#endHandler will be called only after the file that I've copied using pump has fully copied. 

The current behavior - When the HttpClientResponse#endHandler is called, the md5 of the file I've downloaded is different from the md5 of the file in the webApp.

Shelly

[Julien Viet]

can you provide a small reproducer of this ?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4d2dfceb-b1b6-4ec2-bd53-093570ffecf2%40googlegroups.com.

[Thomas SEGISMONT]

When the request ends, the pump should have sent all the data to the WriteStream (asyncFile). But maybe the file write queue is not empty yet. The speed of your disk and the size of your download may influence the result. Could you provide this information? I have not been able to reproduce the issue with your snippet.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4d2dfceb-b1b6-4ec2-bd53-093570ffecf2%40googlegroups.com.

[shell...@gmail.com]

To reproduce:

Put on your server a big file (I saw the problem with 14MB file size)

calculate hash on the file before the pump.start (I used sha256 but I think it doesn't matter)

calculate hash on the downloaded file first thing inside the HttpClientResponse#endHandler 

Print both hashes. In my case the hashed were identical for small files but different for big files.

[shell...@gmail.com]

It happens only with big files (I used 14MB).

It's problematic because I need to reply with a notification when the file is copied to the file system. 

I thought that the endEndler of the request will be called once the stream was ended so it should be related to the file size or the speed of my disk.

[Julien Viet]

it might be a problem on your side, we would need to reproduce it and observe it first.

Thomas said he tried to reproduce, perhaps he has not used a large enough file.

That’s why it would be good if your provide an accurate reproducer of the problem.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c883b1f9-1841-4b27-9d9d-6249f99ed6c5%40googlegroups.com.

[Thomas SEGISMONT]

Please provide you Vert.x version, operating system and type of disk (local, net share?) as well.

I have tried to reproduce again with this code and a 100MB file to download from httpd, without success

public class WVerticle extends AbstractVerticle {
  private static final Logger logger = LoggerFactory.getLogger(WVerticle.class);

  public static void main(String[] args) throws Exception {
    VertxOptions vertxOptions = new VertxOptions();
    Vertx vertx = Vertx.vertx(vertxOptions);
    vertx.deployVerticle(new WVerticle());
  }

  @Override
  public void start() throws Exception {
    logger.info("Started");
    HttpClient httpClient = vertx.createHttpClient(new HttpClientOptions());
    httpClient.get("172.17.0.2", "/file.txt", response -> {
      logger.info("Connected");
      response.pause();
      vertx.fileSystem().open("downloadtest", new OpenOptions().setCreate(true).setTruncateExisting(true), openFileEvent -> {
        if (!openFileEvent.succeeded()) {
          logger.error("Failed", openFileEvent.cause());
          return;
        }
        logger.info("File created");
        AsyncFile file = openFileEvent.result();

        Pump downloadPump = Pump.pump(response, file);

        downloadPump.start();
        logger.info("Pump started");
        response.resume();
        logger.info("Response resumed");

        response.endHandler(v -> {
          logger.info("Response end");
          file.flush().close(cv -> logger.info("closed"));
        });
      });
    }).end();
  }
}

By the way, I had a closer look at the AsyncFile implementation and the "close" callback should be invoked only when there is no more outstanding writes.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c883b1f9-1841-4b27-9d9d-6249f99ed6c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/30A5F53E-C113-46A5-A71A-AF383CFCC366%40julienviet.com.

[shell...@gmail.com]

Thanks Thomas,

I'm using vertx 3.0 deployed on linux server(local disk) and found this post that seems exactly like my issue https://groups.google.com/forum/m/#!topic/vertx/N_wSoQlvMMs

I'm working in a close environment so It's not so easy for me to reproduce it in other environment because it means that I'll need to write all the client-server code from scratch.

So I think that it will be easier for me to upgrade the vertx version and check if it works.

If it won't work I'll work to reproduce it and get back to you guys.

Thanks a lot :)

Shelly

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c883b1f9-1841-4b27-9d9d-6249f99ed6c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.