HttpClient best practices

[Brad Orego]

Hey all,

I'm working on building a vertx app with a lot of HTTP requests going out to our API and I haven't found any sort of best practices for what to do with requests, clients, etc. We've been running into some performance issues over time (e.g. if the app is idle for awhile, it won't make any further connections), so we're not sure if it's a problem with vertx or a problem with my code.

Are there any sort of guidelines as to whether or not it's better to explicitly close vertxHttpClient objects, how to configure them (re: keepalive, pool size, etc), and how to deal with an app that sometimes has to make several requests in a row and wait for all of them to return before ending the server response, etc.

If I can provide more detail that'll help answer, I can. Let me know. Thanks!

[Nate McCall]

What does "netstat -antp TCP" tell you (client and server) about the
current state of the sockets?

If you can scrape out a gist or similar of your client code, that
would be helpful. Also, please include the vert.x version (netty as
well actually) you are using.

> --
> You received this message because you are subscribed to the Google Groups
> "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vertx+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Brad Orego]

Ah, crap, forgot about that. It's vertx 2.0.0-CR2. Scraping out my client code would be about 2500 lines of code and a lot of proprietary information that isn't really worth the time to pick out.

I'm basically just asking for best practices. Should I be explicitly closing the client after every request? Should I not close any and let vertx/netty handle opening/closing? Should I be making multiple simultaneous requests with the same client? Should I have multiple client objects? What about keepAlive? And the connection pool? How big should it be? What implications do any/all of those decisions make?

I don't have the netstat info at the moment but I could get it for you later if that's still relevant information.

Let me know if there's anything else I can do to help. Thanks!

[Brad Orego]

I've tried both explicitly calling client.close() on all requests and not calling it on any requests, and we still seem to see the same hang-on-idle behavior from vertx. This is a major issue, and we haven't been able to see anything in our logs, in vertx error logs, or using netstat or any other tools. 

I'm at the point where I'm thinking about rewriting everything in RoR :(

[Tim Fox]

Brad, there's not much we can do unless you provide some kind of test
case or example that demonstrates the issue.

Without that, we have nothing to work on.

[Nate McCall]

Brad - can you retry with CR3 real quick and see if the issue
persists? http://repo2.maven.org/maven2/io/vertx/vertx-core/2.0.0-CR3/

[Brad Orego]

Asking for "best practices" is too much to ask for....?

[Tim Fox]

On 15/07/13 21:25, Brad Orego wrote:
> Asking for "best practices" is too much to ask for....?

"Best practices" depend entirely on what you're trying to do. But we
don't know what you're trying to do, and you haven't shown any code! ;)

[Brad Orego]

Will do. It'll take a good 10-15 mins to know if it hangs or not, though.

Really, though, I was asking for best practices so we can debug the code ourselves. I wasn't asking for help with that yet, as I wanted to make sure we weren't doing anything stupid on our end.

[Nate McCall]

Does the stock proxy example exhibit the same behavior in your
environment if you let it run for a bit?
https://github.com/vert-x/vertx-examples/tree/master/src/raw/java/proxy

IME, there just hasnt been much magic to it. Sizing the pool correctly
and toggling keep-alive is really all i've had to do. Other than the
pooling just kind of doing it's thing, i've not had any problems
playing around with it for a load testing harness I wrote (never with
an idle workload though as I've nothing using the client in production
unfortunately).

[Brad Orego]

Best Practices typically refer to what the standard way of doing things within the language/framework are. It shouldn't be application dependent, and should guide the way to write and structure code.

Basically what we're doing right now is we declare one HttpClient object globally ("gnClient = vertx.createHttpClient().host(container.config.api_host).port(container.config.api_port)") and then reuse it throughout the server.js file, sometimes sending multiple requests per RouteMatcher match function. 

What we've found so far is that without explicitly closing the client within the RouteMatcher functions, it's slower and tends to hang more readily, but even with explicit (maybe even overzealous) client closing, we can get the system to hang if it sits idle for a period of time. 

What happens when it hangs is that all of the server-side code is fine (e.g. routes match properly, execution within the RM function is fine), but none of the outgoing HTTP requests come back from the external API (gnClient.get's callback never executes).

As I said, I can show you snippets, but I'd much rather there be some way to already know if what we're doing is wrong and can fix it ourselves.

[Tim Fox]

On 15/07/13 21:42, Brad Orego wrote:
> Best Practices typically refer to what the standard way of doing things
> within the language/framework are. It shouldn't be application dependent,
> and should guide the way to write and structure code.
>
> Basically what we're doing right now is we declare one HttpClient object
> globally ("gnClient =
> vertx.createHttpClient().host(container.config.api_host).port(container.config.api_port)") and
> then reuse it throughout the server.js file, sometimes sending multiple
> requests per RouteMatcher match function.

Do you mean you are creating one instance of the Http Client per
JavaScript verticle? A code example would be useful here...

How many instances of your verticle are you creating?

What pool size are you using?

>
> What we've found so far is that without explicitly closing the client
> within the RouteMatcher functions,

If you have a single instance of the http client per verticle, and you
close it in a routematcher function, then it's going to closed, and not
usable for any more requests. This doesn't make sense to me. But, again,
without seeing any code it's hard to know what you are actually doing.

> it's slower and tends to hang more
> readily, but even with explicit (maybe even overzealous) client closing, we
> can get the system to hang if it sits idle for a period of time.

I don't see how anything would work if you have an instance of the
client per verticle and you close it after every request.

>
> What happens when it hangs is that all of the server-side code is fine
> (e.g. routes match properly, execution within the RM function is fine), but
> none of the outgoing HTTP requests come back from the external API
> (gnClient.get's callback never executes).

If you have a non closed client and you aren't seeing responses from
requests that implies the servers you're talking to aren't sending back
responses.

>
> As I said, I can show you snippets, but I'd much rather there be some way
> to already know if what we're doing is wrong and can fix it ourselves.

Without seeing code, we're just playing 20 questions. It's a fun game
but unlikely to be the best way to resolve the problem ;)

By far the best way to resolve this would be to create the simplest
possible test case that demonstrates the issue and show it to us (you
can send it to me privately if you prefer) :)

[Brad Orego]

Here's a gist with some sample code: https://gist.github.com/borego/6003652 - This version might not hang, but it shows you the dev pattern we're using.

I'm creating one client per verticle, yes, and we only have one instance of the verticle running.

Pool size and keepAlive are left at their default values.

I'm explicitly closing the client right after I call req.response.end(), signaling the end of execution for that route. I'm doing so in order to close any lingering sockets/connections that might be leftover for whatever reason.

When the system is hanging, the request isn't making it out at all. We know this two ways: 1) none of my code within the callback executes (e.g. the first line that's just console.log(), 2) our external API never receives the request (nothing in the logs for it).

I can provide a more verbose code example if what you're seeing doesn't make sense yet. Thanks for your time on this one.

[Tim Fox]

On 15/07/13 22:36, Brad Orego wrote:
> Here's a gist with some sample code: https://gist.github.com/borego/6003652
> - This version might not hang, but it shows you the dev pattern we're using.

Thanks

>
> I'm creating one client per verticle, yes, and we only have one instance of
> the verticle running.
>
> Pool size and keepAlive are left at their default values.

Default pool size is 1, so this means your client will be not be able to
have more than one request in progress at any one time

>
> I'm explicitly closing the client right after I call req.response.end(),
> signaling the end of execution for that route. I'm doing so in order to
> close any lingering sockets/connections that might be leftover for whatever
> reason.

In your code you have one instance of the client which you create when
the verticle is deployed.

When you receive a request in the route matcher you then send two
requests to google.com. When both those requests have responded you then
*close* the client. This means the client can't be used any more, so the
next request that is received in the routematcher after the client is
closed will be attempting to use a closed client.

Is this really what you want? It doesn't seem right to me.

[Tim Fox]

Also using a timer to check both requests have responded seems a bit
hacky and unnecessary. You could do the same by just having two boolean
flags req1Responded and req2Responded.

On 15/07/13 22:36, Brad Orego wrote:

[Brad Orego]

For some reason I thought default pool size was 10, but either way I've updated our code to set the pool size. 

I'm almost positive I've had multiple requests out simultaneously, as I've used this pattern in about 20 different route matches and it's worked.

We began explicitly closing clients when this issue came up the first time. We thought there were issues with resource constraint, or requests not ending, so we decided to explicitly close the client at the end of a route lifecycle. I was under the impression that making a request with a close client would reopen the client, and that the close() function was just a way to make sure all outstanding requests were ended. Obviously making a request with a closed client works/does something, as this pattern works throughout our production code (as well as in this example).

Re: the timer, if all I did was check once if the two requests had come back, it's entirely possible for execution to end before the clients come back (given everything is async). It might not be the best form to put the timer inside the request callback of one of the two, but I figure this way saves some unnecessary spinning by the main thread (e.g. why bother checking if two have come back when you can only check if one has).

This is the type of feedback I was looking for initially, though - what to do with keepAlive, where to set the pool size, how to handle multiple client requests from one server request, etc.

[Tim Fox]

On 16/07/13 12:04, Brad Orego wrote:
> For some reason I thought default pool size was 10, but either way I've
> updated our code to set the pool size.
>
> I'm almost positive I've had multiple requests out simultaneously,

Yes you might have several requests out simultaneously if you manage to
send several before the responses for the first one comes back. You
might even be able to send 1000s if you're quick.

> as I've
> used this pattern in about 20 different route matches and it's worked.
>
> We began explicitly closing clients when this issue came up the first time.
> We thought there were issues with resource constraint, or requests not
> ending, so we decided to explicitly close the client at the end of a route
> lifecycle. I was under the impression that making a request with a close
> client would reopen the client, and that the close() function was just a
> way to make sure all outstanding requests were ended.

No, closing a client while there are still outstanding requests means
the connections are closed and you won't receive responses for those
outstanding requests.

Clients should only be closed when you are finished using it. Close
means "I am done with this client and I will never use it again".

> Obviously making a
> request with a closed client works/does something, as this pattern works
> throughout our production code (as well as in this example).
>
> Re: the timer, if all I did was check once if the two requests had come
> back, it's entirely possible for execution to end before the clients come
> back (given everything is async).

I don't understand what you mean here. All you need is two flags, no
timer is required.

[Brad Orego]

The requests are asynchronous, correct? If I just have two flags and let execution proceed as normal, the function will end before the requests come back from the server. Nothing guarantees that the second request will be done before the first request returns, and without a timer the server will respond without the payload of the 2nd request

[Tim Fox]

On 16/07/13 12:30, Brad Orego wrote:
> The requests are asynchronous, correct? If I just have two flags and let
> execution proceed as normal, the function will end before the requests come
> back from the server.

What function are you referring to?

> Nothing guarantees that the second request will be
> done before the first request returns,

It doesn't matter. You don't need a timer, just use two flags something
like this:

var resp1Complete = false;
var resp2Complete = false;

var bodyToReturn = null;

resp1.bodyHandler(function(body) {
resp1Complete = true;
bodyToReturn = body;
checkFinished();
});

resp2.bodyHandler(function(body) {
resp2Complete = true;
checkFinished();
});

function checkFinished() {
if (resp1Complete && resp2Complete) {
{
req.response.end(bodyToReturn);

[NexttAPI]

Hey Nate & Tim,

I'm the API developer working with Brad on this issue.  First I'd like to thank you for your prompt responses & all of the help you've given us.  We started experiencing issues where the website would become unresponsive and hang on most incoming requests.  We were able to narrow the hanging requests down to requests that were calling our API using the HttpClient connection pool.  Using netstat we were able to determine that incoming sockets were being created but outgoing sockets to our API were not.  We began the process of seeing how the HttpClient connection pool reacts to various API responses.  IE: HTTP 500, socket closed w/o HTTP body \r\n etc.  We were eventually able to reproduce the issue by adding a thread sleep to one of our API responses.  What we're seeing is that with the default vert.x HttpClient settings if we send through 5 requests to our API that sleeps for 5 seconds only the first request is actually being sent to our API and we believe the other 4 are being queued.  However, we are not seeing the remaining 4 requests ever get dequeued and sent to our API.  This ultimately cascaded into a deadlock because of the logic to check for completion was never ran because it's in the response callbacks.

We've been testing this morning with using the .setTimeout method on the HttpRequest object and the HttpRequest errorHandler callback will close the route manager's incoming request.  This along with setting the max pool size to larger than 1 seems to have mitigated the issue but not resolved it.  Why aren't subsequent HttpRequests being fired? 

Best Regards,

Mike

[Tim Fox]

On 16/07/13 19:47, NexttAPI wrote:
> Hey Nate & Tim,
>
> I'm the API developer working with Brad on this issue. First I'd like to
> thank you for your prompt responses & all of the help you've given us. We
> started experiencing issues where the website would become unresponsive and
> hang on most incoming requests. We were able to narrow the hanging
> requests down to requests that were calling our API using the HttpClient
> connection pool. Using netstat we were able to determine that incoming
> sockets were being created but outgoing sockets to our API were not. We
> began the process of seeing how the HttpClient connection pool reacts to
> various API responses. IE: HTTP 500, socket closed w/o HTTP body \r\n etc.
> We were eventually able to reproduce the issue by adding a thread sleep to
> one of our API responses.

You cannot use Thread.sleep in a verticle as it blocks the event loop -
the verticle will grind to a halt!

[Tim Fox]

On 16/07/13 19:47, NexttAPI wrote:

> Hey Nate & Tim,
>
> I'm the API developer working with Brad on this issue. First I'd like to
> thank you for your prompt responses & all of the help you've given us. We
> started experiencing issues where the website would become unresponsive and
> hang on most incoming requests. We were able to narrow the hanging
> requests down to requests that were calling our API using the HttpClient
> connection pool. Using netstat we were able to determine that incoming
> sockets were being created but outgoing sockets to our API were not. We
> began the process of seeing how the HttpClient connection pool reacts to
> various API responses. IE: HTTP 500, socket closed w/o HTTP body \r\n etc.
> We were eventually able to reproduce the issue by adding a thread sleep to
> one of our API responses. What we're seeing is that with the default
> vert.x HttpClient settings if we send through 5 requests to our API that
> sleeps for 5 seconds only the first request is actually being sent to our
> API and we believe the other 4 are being queued. However, we are not
> seeing the remaining 4 requests ever get dequeued and sent to our API.
> This ultimately cascaded into a deadlock because of the logic to check for
> completion was never ran because it's in the response callbacks.
>
> We've been testing this morning with using the .setTimeout method on the
> HttpRequest object and the HttpRequest errorHandler callback will close the
> route manager's incoming request. This along with setting the max pool
> size to larger than 1 seems to have mitigated the issue but not resolved
> it. Why aren't subsequent HttpRequests being fired?

If you are closing the client as described by Brad after the first
request, that would explain that.

[Brad Orego]

The thread.sleep was introduced in C#.NET on the API side. It wasn't in our verticle at all.

All of the client.close() calls have been removed from our codebase at your recommendation.

[Mike]

The thread sleep is in the server side API code.  I was just using that to simulate a Http request taking a long time.  We were seeing the hanging before adding the close calls though.  In fact we added them because of the hanging.  The exceptionHandler on the HttpRequest wasn't firing from the close either it was firing because of the timeouts.  Agreed, though that we need to go back and double check all close calls are removed and see if we still see the requests not being executed.

Mike

[Tim Fox]

If you can knock up a test program that demonstrates the issue I can
debug it, and figure out what is going on.

Should be a fairly simple matter to knock together a server that
pretends to be your API and sleeps for 5 seconds...

[Mike]

Ok it looks like the version I was using still had the explicit calls to .close().  Let me remove them and see what we get before we take up your time debugging.

[Brad Orego]

I just commented out all of the client.close() statements in my app and now it's hanging after a few minutes of idle time on my local machine (which it doesn't do with the client being explicitly closed). I have .keepAlive(true), .maxPoolSize(25), and .timeout(60000) and .exceptionHandler() on all of my requests.

The only behavior I'm seeing when it hangs is for the TimeoutException to fire. It seems as if the requests aren't making it out at all after sitting idle for a few minutes.

[Tim Fox]

Ok, next step would be to create a fake api server then we have a self
contained unit that replicates the issue and I can play with.

[Brad Orego]

One thing worth mentioning that might be relevant - both the server and client are using SSL in production. Not sure if this matters.

[Mike]

Tim,

We're working on getting an example to exhibit the same behavior as our website code.  Which ultimately will probably lead us to the root cause of the issue anyway.  Are there any commercial support options available yet that we could use to help get to the bottom of this?

Thanks,

Mike

[Brad Orego]

Hi all,

Here's a brief update with what we've found in testing (using this code):

• does not hang when pointed at google.com
• does not hang when using a dummy vertx server as API
• hangs when pointed at a static HTML file served by our Azure IIS server
• hangs when pointed at any of our routes (again, Azure IIS server)
• hangs against Mike's local dev API (IIS, not Azure)
• hangs against msn.com and microsoft.com
• works on our API if you change routes to /system/diagnostics (returns JSON with one property - PingTime)

Our test is generally: 1) start server, 2) hit it with a few requests, 3) wait ~5 minutes, 4) try to hit it again

This quite obviously seems to be a problem with vertx and IIS, maybe with some SSL thrown in. Any insights would be much appreciated, as we need to keep moving forward with production and either need this figured out or we'll be switching over to Ruby on Rails.

Thanks for your time and help thus far.

- Brad

[Tim Fox]

If you can provide me with an example I can run and can point at your
IIS Server, then I can debug it.

On 17/07/13 12:48, Brad Orego wrote:
> Hi all,
>

> Here's a brief update with what we've found in testing (using this code<https://gist.github.com/borego/6019764>
> ):
>
> - does not hang when pointed at google.com
> - does not hang when using a dummy vertx server as API
> - hangs when pointed at a static HTML file served by our Azure IIS server
> - hangs when pointed at any of our routes (again, Azure IIS server)
> - hangs against Mike's local dev API (IIS, not Azure)
> - hangs against msn.com and microsoft.com
> - works on our API if you change routes to /system/diagnostics (returns

[Tim Yates]

Just to confirm it seems to hang on my machine as well

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

[Brad Orego]

You can use the code from the pastebin and try it against alphaapi.getnextt.com. There's some auth and other stuff you'd need to get actual responses, but we were seeing it hang without using any auth credentials and just getting 403s from the server.

[Tim Fox]

Ok, I will take a look tomorrow, once I've got the 2.0 release out

[Tim Fox]

Tim - what did you actually run?

The snippet is just a snippet, not runnable. And I've no idea what urls
to use.

Can you send me what you ran?

On 17/07/13 13:16, Tim Yates wrote:
> Just to confirm it seems to hang on my machine as well
>
>
> On 17 July 2013 13:16, Tim Fox <timv...@gmail.com> wrote:
>
>> If you can provide me with an example I can run and can point at your IIS
>> Server, then I can debug it.
>>
>>
>> On 17/07/13 12:48, Brad Orego wrote:
>>
>>> Hi all,
>>>
>>> Here's a brief update with what we've found in testing (using this code<

>>> https://gist.github.com/**borego/6019764<https://gist.github.com/borego/6019764>

>> email to vertx+unsubscribe@**googlegroups.com<vertx%2Bunsu...@googlegroups.com>
>> .
>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>> .
>>
>>
>>

[Tim Yates]

I saved https://gist.github.com/borego/6019764 as file.js

And ran

   vertx run file.js

Load localhost, wait 3 mins, refresh and it hangs...

I did a quick rewrite to groovy, and got the same result...

Increasing the httpclient pool size *seemed* to remove the locking, but I suspect it might have just delayed it

email to vertx+unsubscribe@**googlegroups.com<vertx%2Bunsubscribe@googlegroups.com>

.
For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Tim Fox]

I don't follow - surely you need to fire some requests at it too....
otherwise it will do nothing.

If so, what requests?

On 17/07/13 15:16, Tim Yates wrote:
> I saved https://gist.github.com/borego/6019764 as file.js
>
> And ran
>
> vertx run file.js
>
> Load localhost, wait 3 mins, refresh and it hangs...
>
> I did a quick rewrite to groovy, and got the same result...
>
> Increasing the httpclient pool size *seemed* to remove the locking, but I
> suspect it might have just delayed it
>
>
> On 17 July 2013 15:04, Tim Fox <timv...@gmail.com> wrote:
>
>> Tim - what did you actually run?
>>
>> The snippet is just a snippet, not runnable. And I've no idea what urls to
>> use.
>>
>> Can you send me what you ran?
>>
>>
>> On 17/07/13 13:16, Tim Yates wrote:
>>
>>> Just to confirm it seems to hang on my machine as well
>>>
>>>
>>> On 17 July 2013 13:16, Tim Fox <timv...@gmail.com> wrote:
>>>
>>> If you can provide me with an example I can run and can point at your IIS
>>>> Server, then I can debug it.
>>>>
>>>>
>>>> On 17/07/13 12:48, Brad Orego wrote:
>>>>
>>>> Hi all,
>>>>> Here's a brief update with what we've found in testing (using this code<

>>>>> https://gist.github.com/****borego/6019764<https://gist.github.com/**borego/6019764>
>>>>> <https://gist.**github.com/borego/6019764<https://gist.github.com/borego/6019764>

>>>> email to vertx+unsubscribe@**googlegrou**ps.com<http://googlegroups.com>
>>>> <vertx%2Bunsubscribe@**googlegroups.com<vertx%252Buns...@googlegroups.com>
>>>> .
>>>> For more options, visit https://groups.google.com/****groups/opt_out<https://groups.google.com/**groups/opt_out>
>>>> <https://groups.**google.com/groups/opt_out<https://groups.google.com/groups/opt_out>

>>>> .
>>>>
>>>>
>>>>
>>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "vert.x" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to vertx+unsubscribe@**googlegroups.com<vertx%2Bunsu...@googlegroups.com>

[Brad Orego]

The snippet is runnable. Note that you should be getting 403s on both requests (or, if for some reason our auth isn't working, you should be getting 404s). Regardless, though, it's hanging. Again, something to do with vertx and IIS, but no idea what.

email to vertx+unsubscribe@**googlegroups.com<vertx%2Bu...@googlegroups.com>

.
For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Tim Yates]

Sorry, by "Load localhost" I meant "Go to http://localhost:8080/ in a browser" ;-)

[Tim Fox]

On 17/07/13 15:20, Tim Yates wrote:

Sorry, by "Load localhost" I meant "Go to http://localhost:8080/ 

Root path "/" is the correct url for the api? That seems strange to me.

When I type: http://livingapi.getnextt.com/ into a browser I just get a 403 anyway

[Tim Fox]

On 17/07/13 15:16, Tim Yates wrote:

> I saved https://gist.github.com/borego/6019764 as file.js
>
> And ran
>
> vertx run file.js
>
> Load localhost, wait 3 mins, refresh and it hangs...

Can you be a bit more specific about what you mean by "it hangs"?

>
> I did a quick rewrite to groovy, and got the same result...
>
> Increasing the httpclient pool size *seemed* to remove the locking, but I
> suspect it might have just delayed it
>
>
> On 17 July 2013 15:04, Tim Fox <timv...@gmail.com> wrote:
>
>> Tim - what did you actually run?
>>
>> The snippet is just a snippet, not runnable. And I've no idea what urls to
>> use.
>>
>> Can you send me what you ran?
>>
>>
>> On 17/07/13 13:16, Tim Yates wrote:
>>
>>> Just to confirm it seems to hang on my machine as well
>>>
>>>
>>> On 17 July 2013 13:16, Tim Fox <timv...@gmail.com> wrote:
>>>
>>> If you can provide me with an example I can run and can point at your IIS
>>>> Server, then I can debug it.
>>>>
>>>>
>>>> On 17/07/13 12:48, Brad Orego wrote:
>>>>
>>>> Hi all,
>>>>> Here's a brief update with what we've found in testing (using this code<

>>>>> https://gist.github.com/****borego/6019764<https://gist.github.com/**borego/6019764>
>>>>> <https://gist.**github.com/borego/6019764<https://gist.github.com/borego/6019764>

>>>> email to vertx+unsubscribe@**googlegrou**ps.com<http://googlegroups.com>
>>>> <vertx%2Bunsubscribe@**googlegroups.com<vertx%252Buns...@googlegroups.com>

>>>> .

>>>> For more options, visit https://groups.google.com/****groups/opt_out<https://groups.google.com/**groups/opt_out>

>>>> <https://groups.**google.com/groups/opt_out<https://groups.google.com/groups/opt_out>

>>>> .
>>>>
>>>>
>>>>
>>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "vert.x" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to vertx+unsubscribe@**googlegroups.com<vertx%2Bunsu...@googlegroups.com>

[Brad Orego]

You'll get a 403 if you try to hit it from a browser because there's an auth layer that we haven't even touched yet (because it's hanging just on the 403s/404s). Technically the API is using SSL, as well, so you'd need .port(433).ssl(true) onto the client in the snippet.

By "it hangs" I mean exactly what we said to begin with: the outgoing requests never leave/never reach the API server. The console.logs within the request callback and within the bodyHandler callback are never executed. The system just waits on the request.

If you put a .timeout and an .exceptionHandler, the requests just timeout. Otherwise they wait indefinitely.

>>>> <vertx%2Bunsubscribe@**googlegroups.com<vertx%252Bunsubscribe@googlegroups.com>

>>>> .
>>>> For more options, visit https://groups.google.com/****groups/opt_out<https://groups.google.com/**groups/opt_out>
>>>> <https://groups.**google.com/groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .
>>>>
>>>>
>>>>
>>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "vert.x" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to vertx+unsubscribe@**googlegroups.com<vertx%2Bu...@googlegroups.com>

[Tim Fox]

On 17/07/13 15:27, Brad Orego wrote:
> You'll get a 403 if you try to hit it from a browser

And it will also return a 403 using your example as that doesn't provide
any auth credentials either.

> because there's an
> auth layer that we haven't even touched yet (because it's hanging just on
> the 403s/404s). Technically the API is using SSL, as well, so you'd need
> .port(433).ssl(true) onto the client in the snippet.
>
> By "it hangs" I mean exactly what we said to begin with: the outgoing
> requests never leave/never reach the API server.

Ok, I wouldn't call that "hanging", more request is not sent. Hanging
generally applies the process is locked up in some way.

> The console.logs within
> the request callback and within the bodyHandler callback are never
> executed. The system just waits on the request.
>
> If you put a .timeout and an .exceptionHandler, the requests just timeout.
> Otherwise they wait indefinitely.
>
> On Wednesday, July 17, 2013 9:24:17 AM UTC-5, Tim Fox wrote:
>> On 17/07/13 15:16, Tim Yates wrote:
>>> I saved https://gist.github.com/borego/6019764 as file.js
>>>
>>> And ran
>>>
>>> vertx run file.js
>>>
>>> Load localhost, wait 3 mins, refresh and it hangs...
>> Can you be a bit more specific about what you mean by "it hangs"?
>>> I did a quick rewrite to groovy, and got the same result...
>>>
>>> Increasing the httpclient pool size *seemed* to remove the locking, but
>> I
>>> suspect it might have just delayed it
>>>
>>>
>>> On 17 July 2013 15:04, Tim Fox <timv...@gmail.com <javascript:>> wrote:
>>>
>>>> Tim - what did you actually run?
>>>>
>>>> The snippet is just a snippet, not runnable. And I've no idea what urls
>> to
>>>> use.
>>>>
>>>> Can you send me what you ran?
>>>>
>>>>
>>>> On 17/07/13 13:16, Tim Yates wrote:
>>>>
>>>>> Just to confirm it seems to hang on my machine as well
>>>>>
>>>>>

>>>>> On 17 July 2013 13:16, Tim Fox <timv...@gmail.com <javascript:>>

>> vertx%252Buns...@googlegroups.com <javascript:>>

>>>>>> .
>>>>>> For more options, visit https://groups.google.com/****groups/opt_out<
>> https://groups.google.com/**groups/opt_out>
>>>>>> <https://groups.**google.com/groups/opt_out<
>> https://groups.google.com/groups/opt_out>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>> Groups
>>>> "vert.x" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>> an
>>>> email to vertx+unsubscribe@**googlegroups.com<

>> vertx%2Bu...@googlegroups.com <javascript:>>

[Brad Orego]

The process isn't locked, but the request never returns, so to a browser (or any other client), it's more or less the same as being locked.

Yes, it does return a 403, but that's irrelevant to the fact that the behavior is the same regardless of the status code or the return. There's something about either how IIS returns responses or how vertx sends requests that doesn't jive with the other, and it's abstracted way below the level of the JS code I wrote (possibly even below the Java code that runs vertx, but I haven't been able to dig that deeply yet)

>> vertx%252B...@googlegroups.com <javascript:>>

[Tim Fox]

Ok, now I have the facts, I will take look later. Thanks.

>>>> vertx%252B...@googlegroups.com <javascript:> <javascript:>>

[Mike]

I'm also going to try and get a packet sniffer on our API server and see if I can't see anything. I'll keep you guys posted.

[Norman Maurer]

Yeah a wireshark dump would be really helpful.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Yates]

Ok, here I ran the posted script

loaded localhost:8080

loaded it again after 19 seconds

loaded it again after 105 seconds (still works)

Then, at 237 seconds from the initial load, we get a packet sent to us that wireshark colours red (my wireshark experience is limited)

After that, requests no longer do anything and nothing shows on wireshark

https://gist.github.com/timyates/f7fe4db75ca6aba319de

[Norman Maurer]

Can you send me the raw wireshark dump file? 

[Tim Yates]

Sent

[Mike]

I got up a reverse proxy on my dev machine and captured the HTTP requests coming through from vertx.  The requests before waiting 5 minutes look like valid HTTP.  I also got a wireshark capture on our alpha API server.  I'm not seeing any traffic on server port 443 after the timeout.  I'll send the capture file.

[Tim Fox]

https://github.com/vert-x/vert.x/issues/696

The IIS server was forcibly terminating the connection (not doing a nice clean close like other servers) which caused a connection exception on the client. Unfortunately this wasn't getting through to closed handler which meant the pool didn't know the connection was freed up. The connection pool is size 1 so subsequent requests just queued for ever waiting for the connection to become free which never happened.

Fixed in master.

[Tim Yates]

Cool, just to confirm it works in my test :-)

[Brad Orego]

Leave it to Microsoft to make everyone's life more difficult.

I'll pull from master and if we're still seeing trouble, I'll get back to you guys.

Excellent work, and thanks for taking the time to take care of this.

[Mike]

We'll also look into why IIS is forcibly closing the connections and update this thread with any findings.  My guess is that it's something they added to mitigate some security vulnerability.  

[Tim Fox]

On 18/07/13 15:35, Mike wrote:
> We'll also look into why IIS is forcibly closing the connections and update
> this thread with any findings. My guess is that it's something they added
> to mitigate some security vulnerability.

Perhaps. For some reason Microsoft likes doing it this way. Internet
Explorer does something similar - when you close tabs/shut down browser
it seems to hard kill the http connections, which seems rather rude, we
had issues with this in the past.

But, in any case, Vert.x should be able to deal with it whether
connections are closed cleanly or not.

>>>>>>> https://gist.github.com/**borego/6019764<https://gist.github.com/borego/6019764>>

[Kaushal Panjwani]

Hi All, 

I am writing my http client in  Java and using vertx-maven-plugin for development. I am facing a similar issue with vert.x version : 2.0.1-final.

here is a gist to analyse the issue.https://gist.github.com/kaushalpanjwani/708658

 If I have my timer running at interval of 1 or 2 minutes I get the response every time, however if I increase it to 5 min. I just get response firstime only.

And, ofcourse if I do keepalive : false, it works fine for any interval.

Let me know if I am doing anything wrong here.

[Tim Fox]

On 21/10/13 19:59, Kaushal Panjwani wrote:

Hi All, 

I am writing my http client in  Java and using vertx-maven-plugin for development. I am facing a similar issue with vert.x version : 2.0.1-final.

here is a gist to analyse the issue.https://gist.github.com/kaushalpanjwani/708658

I get a 404

[Kaushal Panjwani]

Sorry about the broken URL : https://gist.github.com/kaushalpanjwani/7086586

Also, My webservices are on : Windows 2008 running IIS 7.