Program hanging on recv()

[Ramon]

Hi,

I'm writing an application (basic client) that communicates with an HTTP
server (Yahoo weather server, to be exact).

I've got a problem when receiving data from the server: i.e. program
hangs while reading the last chunk of data. Also the server is normally
sending 2, 3 or 4 responses, each response in not of the specified size
in recv() but smaller.

So my questions are:
1) Why servers' responses size are not the same as the specified size
of recv()?
2) The program in hanging on the last recv() for a short period of time
(say 10 seconds). How is it possible to remove such hang?

Thank you.

Code:

/*
* SERVER_RESPONSE_SIZE = 8192
* Code in C++.
*/

ssize_t resp = recv(serverSocket, response, SERVER_RESPONSE_SIZE, 0);
string outputPage;

if (resp <= 0)
/* error */

do
{
printf("%d>>\n", resp); // debugging info

outputPage += response;
memset(response, '\0', SERVER_RESPONSE_SIZE);
} while ((resp = recv(serverSocket, response, SERVER_RESPONSE_SIZE, 0))
> 0);

// last recv returns -1

[Richard Kettlewell]

Ramon <ramif_47...@yahoo.co.uk> writes:
> I'm writing an application (basic client) that communicates with an
> HTTP server (Yahoo weather server, to be exact).
>
> I've got a problem when receiving data from the server: i.e. program
> hangs while reading the last chunk of data. Also the server is
> normally sending 2, 3 or 4 responses, each response in not of the
> specified size in recv() but smaller.
>
> So my questions are:
> 1) Why servers' responses size are not the same as the specified size
> of recv()?

That’s the expected behaviour. “The receive calls normally return any
data available, up to the requested amount, rather than waiting for
receipt of the full amount requested.” The actual lengths you see will
reflect the division of the response stream into TCP segments.

> 2) The program in hanging on the last recv() for a short period of
> time (say 10 seconds). How is it possible to remove such hang?

It blocks if there's nothing to read. It’s hard to say more without
more detail but I imagine the peer is waiting for another request and
closing the connection only when one doesn’t arrive within a reasonable
timeframe.

--
http://www.greenend.org.uk/rjk/

[Ramon]

What I'm actually doing is just downloading an XML page to memory using
the HTTP protocol. (e.g. http://weather.yahooapis.com/forecastrss?w=2502265)

The actual problem is that I do not know the size of that XML file
beforehand.

Originally, what I've did was to call recv() multiple times: the
download operation is finished once recv() reads less than the buffer size.

Such approach is similar to when we use the read() system call to read a
file, after all the basic concept of sockets is just an extension of
file so to speak.

But the problem with that approach is that recv() returns with different
bytes read sizes.

So if I had a buffer size of 8192 bytes, it first reads/returns 1440,
then 1359 and finally 5 (bytes). It then hangs and after say 10 seconds
it returns -1.

*Question*: Is it possible to know the size of the page to download
beforehand? Or is then any other way how I can solve the recv() hanging
issue?

Thanks.

[Lew Pitcher]

On Sunday 03 February 2013 17:05, in comp.os.linux.development.apps,

ramif_47...@yahoo.co.uk wrote:

> What I'm actually doing is just downloading an XML page to memory using
> the HTTP protocol. (e.g.
> http://weather.yahooapis.com/forecastrss?w=2502265)

[snip]

>
> *Question*: Is it possible to know the size of the page to download
> beforehand?

Yes, and no.

If you first converse with the HTTP server using a HEAD method, you will
get, in the response, a "Content-Length" tag which will tell you how big
the content is. The problem is that this is still a TCP conversation, and
will have the same "response size" issue that you currently have, but for
it's own response. In other words, you will *not* know how big the server's
response to your HEAD request will be beforehand, and will only be able to
detect that you've received it all by encountering the termination of the
conversation.

If you send a GET method instead, you will get both the response headers
(including the "Content-Length" header) /and/ the data contents.

In either case, you *must* interpret the data you receive from the server to
determine how big the response is going to be.

> Or is then any other way how I can solve the recv() hanging
> issue?

Is there any reason you can't use already existing utilities to fetch the
page /as a file/, and then perform post-processing on that file? If not,
then why not just use wget(1) (or similar) to retrieve the XML into a file,
and then process the file without worrying about the HTTP protocol?

HTH
--
Lew Pitcher
"In Skills, We Trust"

[Richard Kettlewell]

Please do not CC usenet postings to me.

Ramon <ramif_47...@yahoo.co.uk> writes:
> What I'm actually doing is just downloading an XML page to memory
> using the HTTP
> protocol. (e.g. http://weather.yahooapis.com/forecastrss?w=2502265)
>
> The actual problem is that I do not know the size of that XML file
> beforehand.
>
> Originally, what I've did was to call recv() multiple times: the
> download operation is finished once recv() reads less than the buffer
> size.
>
> Such approach is similar to when we use the read() system call to read
> a file, after all the basic concept of sockets is just an extension of
> file so to speak.
>
> But the problem with that approach is that recv() returns with
> different bytes read sizes.
>
> So if I had a buffer size of 8192 bytes, it first reads/returns 1440,
> then 1359 and finally 5 (bytes). It then hangs and after say 10
> seconds it returns -1.
>
> *Question*: Is it possible to know the size of the page to download
> beforehand? Or is then any other way how I can solve the recv()
> hanging issue?

Yes, this information is provided in one of several possible ways to
HTTP clients. Start with http://www.ietf.org/rfc/rfc2616.txt for
further information.

Have you considered using a pre-existing HTTP library such as cURL
instead?

--
http://www.greenend.org.uk/rjk/

[Jasen Betts]

On 2013-02-03, Ramon <ramif_47...@yahoo.co.uk> wrote:
> What I'm actually doing is just downloading an XML page to memory using
> the HTTP protocol. (e.g. http://weather.yahooapis.com/forecastrss?w=2502265)

HTTP version 1.1 should always indoicate the size, either explicitly
or by using chunked transmission.

> The actual problem is that I do not know the size of that XML file
> beforehand.

You're not using libxml2 by any chance? Did you know it contained a
http client?

> *Question*: Is it possible to know the size of the page to download
> beforehand?

HTTP1.1

> Or is then any other way how I can solve the recv() hanging issue?

EOF should be signalled somehow, I'd cheat and use wireshark to look at
what hppens when firefox does the same request.

--
⚂⚃ 100% natural

--- news://freenews.netfront.net/ - complaints: ne...@netfront.net ---

[Rainer Weikusat]

Ramon <ramif_47...@yahoo.co.uk> writes:
> What I'm actually doing is just downloading an XML page to memory
> using the HTTP
> protocol. (e.g. http://weather.yahooapis.com/forecastrss?w=2502265)

[...]

> *Question*: Is it possible to know the size of the page to download
> beforehand?

If you're only interested in using the TCP connection for a single
request-reply exchange, send a HTTP1.0 request without a Connection:
header or with Connection: close as connection header and call
shutdown(sockfd, SHUT_WR) on you socket after sending the complete
request. The server is then required/ supposed to mark the end of the
reply by closing the connection so you can just read until EOF.

Otherwise, there are (AFAIK) two additional cases to deal with:

- length of the content is provided in form a a Content-Length
header

- so-called 'chunked transfer coding' (reply is send in chunks
each of which is preceded by a chunk length)

[Ramon]

Thank you for the chunked transmission hint. That did the trick :)