pattern for balancing IO

[klas eriksson]

Hi

I'm a vertx newbie and trying to figure out some way to balance IO in a simple for loop.
I've reduced my problem to this toy example:

for (int i = 0; i < 1000000; i++) {
   JsonObject body = new JsonObject().put("number", i);
   httpclient.postAbs("http://foobar.com:8080/isPrime", clientResp -> {
      if (clientResp.statusCode() == 200) {
         log.debug("is prime: {}", body);
      } else {
         log.debug("not prime {}", body);
      }
   }).end(body.encode());
}

And, after a while I will ("of course") get:
java.util.concurrent.TimeoutException: The timeout period of 60000ms has been exceeded while executing POST /isPrime for host foobar.com
        at io.vertx.core.http.impl.HttpClientRequestBase.timeout(HttpClientRequestBase.java:183)

What "pattern" shall I use to get "some" concurrency in the httpclient code, but not "too much"?

I can write some explicit code to "take 10 and then wait". It will make the simple code above really ugly:(
I would like some input on some best-practice "how to do it" in vertx...

br
klas

[Thomas SEGISMONT]

2017-09-18 18:17 GMT+02:00 klas eriksson <klas...@fastmail.fm>:

Hi

I'm a vertx newbie and trying to figure out some way to balance IO in a simple for loop.
I've reduced my problem to this toy example:

for (int i = 0; i < 1000000; i++) {
   JsonObject body = new JsonObject().put("number", i);
   httpclient.postAbs("http://foobar.com:8080/isPrime", clientResp -> {
      if (clientResp.statusCode() == 200) {
         log.debug("is prime: {}", body);
      } else {
         log.debug("not prime {}", body);
      }
   }).end(body.encode());
}

And, after a while I will ("of course") get:
java.util.concurrent.TimeoutException: The timeout period of 60000ms has been exceeded while executing POST /isPrime for host foobar.com
        at io.vertx.core.http.impl.HttpClientRequestBase.timeout(HttpClientRequestBase.java:183)

What "pattern" shall I use to get "some" concurrency in the httpclient code, but not "too much"?

Configure the HTTP client pool size?

 

I can write some explicit code to "take 10 and then wait". It will make the simple code above really ugly:(
I would like some input on some best-practice "how to do it" in vertx...

br
klas

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f7fde1d5-d2c1-4d5a-ac11-271cf3e0d1e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[klas eriksson]

Hi

Still not clear to me how it is supposed to behave...
Are you saying that vertx is supposed to "pile up" 1000000 http-posts and then wait for a loong time
until there is a connection? (because the for-loop is quick and non-blocking)
And that the 60000ms timeout is counted from when the io actually starts? Not when the for loop calls httpclient.postAbs?

Den tisdag 19 september 2017 kl. 00:07:20 UTC+2 skrev Thomas Segismont:

2017-09-18 18:17 GMT+02:00 klas eriksson <klas...@fastmail.fm>:

Hi

I'm a vertx newbie and trying to figure out some way to balance IO in a simple for loop.
I've reduced my problem to this toy example:

for (int i = 0; i < 1000000; i++) {
   JsonObject body = new JsonObject().put("number", i);
   httpclient.postAbs("http://foobar.com:8080/isPrime", clientResp -> {
      if (clientResp.statusCode() == 200) {
         log.debug("is prime: {}", body);
      } else {
         log.debug("not prime {}", body);
      }
   }).end(body.encode());
}

And, after a while I will ("of course") get:
java.util.concurrent.TimeoutException: The timeout period of 60000ms has been exceeded while executing POST /isPrime for host foobar.com
        at io.vertx.core.http.impl.HttpClientRequestBase.timeout(HttpClientRequestBase.java:183)

What "pattern" shall I use to get "some" concurrency in the httpclient code, but not "too much"?

Configure the HTTP client pool size?

 

I can write some explicit code to "take 10 and then wait". It will make the simple code above really ugly:(
I would like some input on some best-practice "how to do it" in vertx...

br
klas

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Thomas SEGISMONT]

2017-09-20 21:07 GMT+02:00 klas eriksson <klas...@fastmail.fm>:

Hi

Still not clear to me how it is supposed to behave...
Are you saying that vertx is supposed to "pile up" 1000000 http-posts and then wait for a loong time
until there is a connection? (because the for-loop is quick and non-blocking)

Yes. But to avoid this situation in real life projects you'd apply some "backpressure" to the source of events.

 

And that the 60000ms timeout is counted from when the io actually starts? Not when the for loop calls httpclient.postAbs?

Yes. Julien can confirm but I don't think we apply any kind of timeout of items in the httpClient queue.

 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8e587221-8d05-42c2-a5f7-2afe7b3a8ac2%40googlegroups.com.

[Julien Viet]

On Sep 22, 2017, at 2:13 PM, Thomas SEGISMONT <tsegi...@gmail.com> wrote:

2017-09-20 21:07 GMT+02:00 klas eriksson <klas...@fastmail.fm>:

Hi

Still not clear to me how it is supposed to behave...
Are you saying that vertx is supposed to "pile up" 1000000 http-posts and then wait for a loong time
until there is a connection? (because the for-loop is quick and non-blocking)

Yes. But to avoid this situation in real life projects you'd apply some "backpressure" to the source of events.

yes in real life, you have the source of event that is back-pressured.

 

And that the 60000ms timeout is counted from when the io actually starts? Not when the for loop calls httpclient.postAbs?

Yes. Julien can confirm but I don't think we apply any kind of timeout of items in the httpClient queue.

we apply optionally a per HttpClientRequest timeout.

can you give the full stacktrace you ahve ?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CACiEr_QmoXgfh_5YqnQWhcJ7JWvz7Vcr-3W6FJFhF9S%3DsoowCw%40mail.gmail.com.