[erlang-questions] confusing returns from httpc:request/4

220 views
Skip to first unread message

Dan Kelley

unread,
Oct 7, 2010, 11:51:48 AM10/7/10
to erlang-q...@erlang.org
Hi,

I've got a simple router that's talking to a remote HTTP load balancer via
httpc:request/4. For every request that I get into the router, I spawn a
new process then attempt to connect to the HTTP LB.

That works well most of the time, but occasionally I get
{error, socket_closed_remotely} instead of results. I had assumed that
these corresponded to the HTTP LB prematurely closing the connection, but
when I look at the logs for that process, it always thinks that it's
returning clean responses (HTTP 200 code, appropriately-formed bodies, etc).

Anyone seen anything like this before? What else might I do to debug this?

Thanks,

Dan

Anthony Molinaro

unread,
Oct 7, 2010, 2:00:57 PM10/7/10
to Dan Kelley, erlang-q...@erlang.org
Hi Dan,

A quick websearch turned up

http://stackoverflow.com/questions/2126630/erlangs-maximum-number-of-simultaneous-open-ports

which seems to suggest you might be at the maximum allowable open file
descriptors and need to update it.

I've had less than stellar results with httpc (see old posts on this group
http://groups.google.com/group/erlang-programming/browse_thread/thread/1d94595c83b21ef6/1b2416a346b65596?lnk=gst&q=httpc#1b2416a346b65596
http://groups.google.com/group/erlang-programming/browse_thread/thread/2f3684d1433c465d/2866bae1935138a2?lnk=gst&q=httpc#2866bae1935138a2).

I would highly recommend ibrowse though, it's worked really well.

-Anthony

--
------------------------------------------------------------------------
Anthony Molinaro <anth...@alumni.caltech.edu>

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org

Dan Kelley

unread,
Oct 9, 2010, 11:25:38 AM10/9/10
to Dan Kelley, erlang-q...@erlang.org
On Thu, Oct 7, 2010 at 2:00 PM, Anthony Molinaro <
anth...@alumni.caltech.edu> wrote:

> Hi Dan,
>
> A quick websearch turned up
>
>
> http://stackoverflow.com/questions/2126630/erlangs-maximum-number-of-simultaneous-open-ports
>
> which seems to suggest you might be at the maximum allowable open file
> descriptors and need to update it.
>

Yeah, I had found that same thing. I did some testing with lsof (I'm on
linux), and could never see the number of open filehandles go over ~200, and
my ulimit is set to 1024.

.. and this turned out to be the answer for me too. Based on your advice, I
switched to ibrowse, which has worked great. Thanks!

Dan

Ingela Andin

unread,
Oct 13, 2010, 4:53:20 AM10/13/10
to djk...@gmail.com, erlang-q...@erlang.org
Hi!

The only reason that the httpc-client will return the value {error,
socket_closed_remotely}
is that the connection is prematurely closed by the server.
Unfortunately that something
is sent on a tcp socket does not mean it has been sent on the wire
yet, and depending on
the circumstances it is possible for the socket to be closed before
all data has been sent
so occasionally a tcp_close can come before all data is delivered.

When you use ibrowse do you verify that you always get a correct body?
(compleate body) from what I can tell from the code ibrowse will on a
tcp_closed message always return whatever happens to be in the buffer
compleate or not.

Regards Ingela Erlang/OTP team - Ericsson AB


2010/10/7 Dan Kelley <djk...@gmail.com>:

________________________________________________________________

Chandru

unread,
Oct 13, 2010, 5:05:04 AM10/13/10
to Ingela Andin, djk...@gmail.com, erlang-q...@erlang.org
On 13 October 2010 09:53, Ingela Andin <ingela...@gmail.com> wrote:
> Hi!
>
> The only reason that the httpc-client will return the value {error,
> socket_closed_remotely}
> is that the connection is prematurely  closed by the server.
> Unfortunately  that something
> is sent on a tcp socket does not mean it has been sent on the wire
> yet, and depending on
> the circumstances it is possible for the socket to be closed before
> all data has been sent
> so occasionally a tcp_close can come before all data is delivered.
>
> When you use ibrowse do you verify that you always get a correct body?
> (compleate body) from what I can tell from the code ibrowse will on a
> tcp_closed  message always return whatever happens to be in the buffer
> compleate or not.
>

ibrowse does this only in the following cases:
* server returned a "Connection: Close"
* Server supports HTTP/0.9 or HTTP/1.0

In all other cases, it returns an error to the caller.

regards,
Chandru

Ingela Andin

unread,
Oct 13, 2010, 5:58:09 AM10/13/10
to Chandru, djk...@gmail.com, erlang-q...@erlang.org
2010/10/13 Chandru <chandrashekha...@gmail.com>:

> On 13 October 2010 09:53, Ingela Andin <ingela...@gmail.com> wrote:
>> Hi!
>>
>> The only reason that the httpc-client will return the value {error,
>> socket_closed_remotely}
>> is that the connection is prematurely  closed by the server.
>> Unfortunately  that something
>> is sent on a tcp socket does not mean it has been sent on the wire
>> yet, and depending on
>> the circumstances it is possible for the socket to be closed before
>> all data has been sent
>> so occasionally a tcp_close can come before all data is delivered.
>>
>> When you use ibrowse do you verify that you always get a correct body?
>> (compleate body) from what I can tell from the code ibrowse will on a
>> tcp_closed  message always return whatever happens to be in the buffer
>> compleate or not.
>>
>
> ibrowse does this only in the following cases:
>  * server returned a "Connection: Close"
>  * Server supports HTTP/0.9 or HTTP/1.0
>
> In all other cases, it returns an error to the caller.
>
> regards,
> Chandru
>

Well I do not know if any of these things is true in Dans senario. But
I do know that
getting the problem with a tcp close message before all data is
deliverd is a highly timing dependant problem
and if it happens in the server there is nothing inets-client or
ibrows can do about it except return an error.

Regards Ingela Erlang/OTP team - Ericsson AB

________________________________________________________________

amitm

unread,
Oct 20, 2010, 2:00:54 AM10/20/10
to erlang-q...@erlang.org
FWIW, I had a similar problem. It went away when I disabled pipelining
completely by executing

ok = httpc:set_options([{max_keep_alive_length, 0},
{max_pipeline_length, 0}, {max_sessions, 0} ]),

before my http calls. I used to usually see it on the 4th consecutive
call to the same host when pipelining was enabled. The URL in question
in my case was an Amazon EC2 url.

The set of URLs in my case were

"http://169.254.169.254/2009-04-04/meta-data/instance-id",
"http://169.254.169.254/2009-04-04/meta-data/local-hostname",
"http://169.254.169.254/2009-04-04/meta-data/local-ipv4",
"http://169.254.169.254/2009-04-04/meta-data/public-hostname",
"http://169.254.169.254/2009-04-04/meta-data/public-ipv4",
"http://169.254.169.254/2009-04-04/meta-data/ami-id",

While executing this set, with pipelining enabled, it used to always
bomb on the 4th URL, i.e. for 'public-hostname' in 25% of the cases

with pipelining disabled, the problem disappears.

Amit

On Oct 13, 2:58 pm, Ingela Andin <ingela.an...@gmail.com> wrote:
> 2010/10/13 Chandru <chandrashekhar.mullapar...@gmail.com>:

> To unsubscribe; mailto:erlang-questions-unsubscr...@erlang.org

Ingela Andin

unread,
Oct 21, 2010, 9:32:22 AM10/21/10
to amitm, erlang-q...@erlang.org
Hi!

Humm ... what settings did you have when you had the problem?
Did you set the pipeline timout ? What webserver did you connect to
(apache, yaws, ...)? What
version of inets? If there is some bug in the pipeling or
persistenconnection code
of course we would like to know so that we can fix it.

Regards Ingela Erlang/OTP team - Ericsson AB


2010/10/20 amitm <amit....@gmail.com>:

amitm

unread,
Oct 21, 2010, 10:06:46 AM10/21/10
to erlang-q...@erlang.org

> Humm ... what settings did you have when you had the problem?

The function making the call in the situation when the problem
manifests itself is:

mkGetReq2(GetUrl) ->
GetRequest = {GetUrl, []},
GetHttpOptions = [{autoredirect, true}],
Options = [ {sync,true}, {headers_as_is,true}, {body_format,
binary} ],
httpc:request(get, GetRequest, GetHttpOptions, Options).

No other options were set, which means they run with their default
values.

> Did you set the pipeline timout ?

No pipeline timeout was set.

> What webserver did you connect to (apache, yaws, ...)?

The webserver is one used by Amazon internally on EC2. Curl in verbose
mode, as executed by the command
"curl -v http://169.254.169.254/2009-04-04/meta-data/instance-id "
prints out

* About to connect() to 169.254.169.254 port 80 (#0)
* Trying 169.254.169.254... connected
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /2009-04-04/meta-data/instance-id HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/
0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 169.254.169.254
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Accept-Ranges: bytes
< ETag: "-1124024050"
< Last-Modified: Tue, 19 Oct 2010 13:58:54 GMT
< Content-Length: 10
< Connection: close
< Date: Thu, 21 Oct 2010 13:48:19 GMT
< Server: EC2ws
<
* Closing connection #0

> What version of inets?

erl -v prints
"Erlang R14B (erts-5.8.1) [source] [rq:1] [async-threads:0] [kernel-
poll:false]"

Hope it helps,

Amit


On Oct 21, 6:32 pm, Ingela Andin <ingela.an...@gmail.com> wrote:
> Hi!
>
> Humm ... what settings did you have when you had the problem?
> Did you set the pipeline timout ? What webserver did you connect to
> (apache, yaws, ...)? What
> version of inets? If there is some bug in the pipeling or
> persistenconnection code
> of course we would like to know so that we can fix it.
>
> Regards Ingela Erlang/OTP team - Ericsson AB
>

> 2010/10/20 amitm <amit.mur...@gmail.com>:

Ingela Andin

unread,
Oct 21, 2010, 11:54:02 AM10/21/10
to amitm, erlang-q...@erlang.org
Hi!

Yes thank you for the information. I think I have located the bug. It seems
that to avoid crash reports the return value when receiving a tcp_closed
was changed and this altered the terminate clause that is run, alas breaking
the automatic retries that should be done if the server terminates a connection
before serving all pipelined requests. This will be fixed in an
upcoming release.

Regards Ingela Erlang/OTP-team, Ericsson AB

2010/10/21 amitm <amit....@gmail.com>:

amitm

unread,
Oct 27, 2010, 1:25:56 AM10/27/10
to erlang-q...@erlang.org
Sorry, I missed out on stating that to make the problem go away, I
also set the HTTP version as "HTTP/1.0"

So you may also want to look at the code for handling persistent
connections in general, especially when the server only supports HTTP
1.0 .


Amit


On Oct 21, 8:54 pm, Ingela Andin <ingela.an...@gmail.com> wrote:
> Hi!
>

> Yes thank you for the information. I think I have located the bug. It seems
> that to avoid crash reports the return value when receiving a tcp_closed
> was changed and this altered the terminate clause that is run,  alas breaking
> the automatic retries that should be done if the server terminates a connection
> before serving all pipelined requests. This will be fixed in an
> upcoming release.
>
> Regards Ingela Erlang/OTP-team,  Ericsson AB
>

> 2010/10/21amitm<amit.mur...@gmail.com>:


>
>
>
>
>
>
>
> >> Humm ... what settings did you have when you had the problem?
>
> > The function making the call in the situation when the problem
> > manifests itself is:
>
> > mkGetReq2(GetUrl) ->
> >    GetRequest = {GetUrl, []},
> >    GetHttpOptions = [{autoredirect, true}],
> >    Options = [ {sync,true}, {headers_as_is,true}, {body_format,
> > binary} ],
> >    httpc:request(get, GetRequest, GetHttpOptions, Options).
>
> > No other options were set, which means they run with their default
> > values.
>
> >> Did you set the pipeline timout ?
>
> > No pipeline timeout was set.
>
> >> What webserver did you connect to (apache, yaws, ...)?
>
> > The webserver is one used by Amazon internally on EC2. Curl in verbose
> > mode, as executed by the command

> > "curl -vhttp://169.254.169.254/2009-04-04/meta-data/instance-id"

> >> 2010/10/20amitm<amit.mur...@gmail.com>:

Ingela Andin

unread,
Oct 28, 2010, 5:51:51 AM10/28/10
to amitm, erlang-q...@erlang.org
Hi!

Thank you for pointing that out, and yes the piepline code and the
persistent connection code has a lot of common code
so it is quite natural that the bug affects them both.

Regards Ingela Erlang/OTP team - Ericsson AB


2010/10/27 amitm <amit....@gmail.com>:

Reply all
Reply to author
Forward
0 new messages