Http Client, response is not complete

2,711 views
Skip to first unread message

ec.developer

unread,
Sep 10, 2010, 5:57:45 AM9/10/10
to nodejs
I have a http client on my nodejs server: httpClient =
http.createClient(port, host);
When some requests arrives to my node server, it sends a request to
another server, gets the response, and returns it to the client.

var request = httpClient.request("POST", '/
some_script_on_my_server.php',
{
'host':'localhost',
"User-Agent": "NodeJS HTTP Client",
'Content-Length': toServ.length,
'Content-Type': 'application/x-www-form-urlencoded'
});

request.on('response', function (response) {

response.on('data', function (chunk) {

sys.puts(chunk);
}
});

The problem is - my script returns to this request an array, encoded
to json:
{"array":[["29","39"],[28,"39"],[28,38],[29,37],[30,38]],"array2":
[{"x":29,"y":38,"data":"76742235197_on_opp"}],"result":"ok"}
But when I put it to the screen (sys.puts(chunk)), sometimes it
outputs just a part of that response:
{"array":[["29

So, I think that the problem is that my httpClient receives just a
part of response.
How to prevent such kinds of problems?

Thanks

mscdex

unread,
Sep 10, 2010, 9:41:47 AM9/10/10
to nodejs
On Sep 10, 5:57 am, "ec.developer" <ec.develo...@gmail.com> wrote:
> So, I think that the problem is that my httpClient receives just a
> part of response.
> How to prevent such kinds of problems?

Does the rest of the JSON display on separate lines? If not, it sounds
like maybe a server issue.

Tane Piper

unread,
Sep 10, 2010, 7:56:22 AM9/10/10
to nod...@googlegroups.com
You need to wait until all the data has been received, so attach a
callback to the end event, something like this:


request.on('response', function(response){
var data = [];
response.on('data', function(chunk) {
data.push(chunk);
});
response.on('end', function() {
var result = JSON.parse(data.join(''))
return result
});
});

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

mscdex

unread,
Sep 10, 2010, 12:32:50 PM9/10/10
to nodejs
On Sep 10, 7:56 am, Tane Piper <piper.t...@gmail.com> wrote:
> You need to wait until all the data has been received, so attach a
> callback to the end event, something like this:
>
> request.on('response', function(response){
>       var data = [];
>       response.on('data', function(chunk) {
>         data.push(chunk);
>       });
>       response.on('end', function() {
>         var result = JSON.parse(data.join(''))
>         return result
>       });
>     });

IIRC 'chunk' is instanceof Buffer, so you'd probably want to do
chunk.toString() instead. Or you could just use a string variable in
the first place and just append to that. However, if the OP doesn't
see the entire response (possibly separated by multiple lines), then
there is an issue with the server not flushing out the entire JSON or
something else.

ec.developer

unread,
Sep 11, 2010, 5:26:32 PM9/11/10
to nodejs
I'm not sure that the problem is caused by php script. It's just
echo's the data in JSON format, and node server should receive this
reply. Before echoing the result, my php script creates a file and
dump there the entire response, and when the trouble appears - I check
this file and there is a complete response. So I think that the
problem is on node's side.

But what about 'end' event? Is it possible that when 'response' event
fires - the 'chunk' var contains incomplete response? Maybe I should
use 'response' var (argument) instead of chunk?

fuzzy spoon

unread,
Sep 11, 2010, 5:29:21 PM9/11/10
to nod...@googlegroups.com
I noticed this recently in a game i was using php to list my top10
players. What happened was it was sporadic,
And on PHP side i sent a newline at the last step...

echo('\n');

This cut down my response time alot, AND fixed the sporadic cut offs.
I wasn't sure even why, but its what mscdex mentioned too so i figured
id throw it out there.
Iv seen PHP not respond fully with a flush or newline.

2010/9/11 ec.developer <ec.dev...@gmail.com>:

mscdex

unread,
Sep 11, 2010, 7:12:14 PM9/11/10
to nodejs
On Sep 11, 5:26 pm, "ec.developer" <ec.develo...@gmail.com> wrote:
> I'm not sure that the problem is caused by php script. It's just
> echo's the data in JSON format, and node server should receive this
> reply. Before echoing the result, my php script creates a file and
> dump there the entire response, and when the trouble appears - I check
> this file and there is a complete response. So I think that the
> problem is on node's side.
>
> But what about 'end' event? Is it possible that when 'response' event
> fires - the 'chunk' var contains incomplete response? Maybe I should
> use 'response' var (argument) instead of chunk?

Well, as the variable name suggests, it's at least a chunk of the
incoming data from the server. Depending on the size of the server's
output, it may be the entire response or it might not be. This is why
you generally have a local variable that you append to every time the
'data' event on the response fires, then you know you are finished
receiving chunks when the 'end' event is emitted.

So, similar to what Tane said earlier, you should be doing something
like this to ensure you get the entire response:

var request = httpClient.request("POST", '/
some_script_on_my_server.php', {
'Host': 'localhost',
'User-Agent': 'NodeJS HTTP Client',
'Content-Length': toServ.length,
'Content-Type': 'application/x-www-form-urlencoded'
});

request.on('response', function(response) {
var data = "";
response.on('data', function(chunk) {
data += chunk.toString();
});
response.on('end', function() {
var jsondata = JSON.parse(data);
// do something with the parsed JSON here
});
});

request.end(toServ); // sends the content you've described in the
headers and completes the request in one step

ec.developer

unread,
Sep 13, 2010, 4:59:08 PM9/13/10
to nodejs
Thank you all, guys! :)
I'm sure that Tane's and mscdex's suggestions will solve my problem. I
will try it right now. Hope the problem will never appear again.

Akzhan Abdulin

unread,
Sep 13, 2010, 8:00:46 PM9/13/10
to nod...@googlegroups.com
Here is my own implementation: http://gist.github.com/578286

2010/9/14 ec.developer <ec.dev...@gmail.com>
Reply all
Reply to author
Forward
0 new messages