How to close a file writable stream when the request pipe is done?

[Gabriel Petrovay]

Hi,

I have a POST request carrying an image (image content type). And I pipe this into a file writable stream:

fileStream = fs.createWriteStream(fileName);

req.pipe(fileStream);

fileStream.on('end', function() {

    console.log('Done with ' + fileName);

});

req.on('end', function() {

    console.log('Ending ' + fileName);

    res.send(200);

});

But there are two problems:

1. The file description is not closed and I cannot open the file with another GET request.

2. "Done with ..." never get called.

How can I close the file descriptor after writing the file? I also tried to call fileStream.close() right before res.send(200) but with no success.

Thanks!

Gabriel

[Gabriel Petrovay]

One side node if this helps: I had one pending request in the browser and I saw that the GET request for the file that was written goes through after 2 minutes. That means that the file descriptor is closed somehow after two minutes automatically.

--
MSc Gabriel Petrovay
Mobile: +41(0)787978034

[Dave Clements]

Hi Gabriel, what is it your trying to achieve here?

btw to output a status code and end response you'd do

res.writeHead(200);
res.end();

[Gabriel Petrovay]

I am uploading an image with the XMLHttpRequest. That issues a POST
request sending the image in the body. The request has the content
type for example 'image/jpeg' and the body is the image itself. This
is not a mime request.

So now I am on the server and want to save the image in a file. So I
pipe the request stream into a writable file stream:
req.pipe(fileStream);

Everything works just fine except that the request is not ending and
therefore, I cannot open the written file because the file descriptor
is still open/hanging around. A GET request to the written file hangs
until that writable file stream is closed. This happens automatically
after 2 minutes.

But if I manually restart the server after the file was written (that
closes the file writable stream and the underlying open file
descriptor) the files can be requested through GET requests.

So, how can I force the request to close when it has been "consumes"?

The XMLHttpRequest POST requests have no "Connection: keep-alive"
headers. Is this considered default on the server is missing? (If this
plays any role)

Any other ideas that I should try?

Thanks!

[Liam]

Does your POST have a correct Content-Length header?

[Gabriel Petrovay]

No it is not, and if I want to set it:

xhr.setRequestHeader('Content-Length', file.fileSize);

Chrome thrown an error:

Refused to set unsafe header "Content-Length"

and no Content-Length is sent to the server.

Chrome shows me in Dev tools in the Network tab for my request:

Name Method Status Type
... POST (pending) image/jpeg

[Gabriel Petrovay]

But as I see, content-length should be handled automatically:
http://stackoverflow.com/questions/2623963/webkit-refused-to-set-unsafe-header-content-length

[Liam]

Check the Content-Length header in the http request, and compare it
with your file size.

Make sure you're closing the XHR cleanly, too.

On Dec 3, 10:21 am, Gabriel Petrovay <gabipetro...@gmail.com> wrote:
> But as I see, content-length should be handled automatically:http://stackoverflow.com/questions/2623963/webkit-refused-to-set-unsa...

[Gabriel Petrovay]

Same (bytes):

File: 92270
ReqL: 92270

[Martin Cooper]

It doesn't look like you're registering any error handlers. Are you
sure there are no errors?

--
Martin Cooper

> Thanks!
>
> Gabriel
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

[Liam]

If you're on Node 0.6.x you might try 0.4...

[Gabriel Petrovay]

I also have this in my code and nothing gets written in the console.
And the file is always completely uploaded/written. It just hangs (in
browser pending until Chrome decides to cancel it; on the server 'end'
event never gets called)

req.on('error', function(err) { console.log(err);});
On Dec 3, 7:49 pm, Martin Cooper <mfncoo...@gmail.com> wrote:

[mscdex]

On Dec 3, 7:33 am, Gabriel Petrovay <gabipetro...@gmail.com> wrote:
> fileStream.on('end', function() {
>     console.log('Done with ' + fileName);});

I don't think a writable *file* stream will ever emit 'end', did you
try the 'close' event?

[Liam]

Could you post your client code?

[Gabriel Petrovay]

I have this jQuery plugin that does the POST:

http://codepad.org/hPv38KxZ

[Gabriel Petrovay]

It is the request stream that I expect to emit the 'end' event when
the request has been consumed (if I understand this properly).

Having such an event I could call close() on the file stream. I put
the end event there to see if this gets somehow called. But it
doesn't. So that was just a debug trial.

[Liam]

Have you tried a form-based POST to your app? Does that 'end'
properly?

[Dave Clements]

think I'd need to see your code..

[Gabriel Petrovay]

@Dave:
http://codepad.org/DEbxFWyD

@Liam:
I did not try it through the form. First because I am doing it do it
from the XMLHttpRequest where I want to keep it light. And on the
server, if I post it through a form that it will go through another
path (bodyParser and request parameters). Or is there a streaming
scenario when using forms?

[Liam]

Just see if a form POST to your current logic works. If it does, you
have a client side issue.

[Gabriel Petrovay]

Getting closer... I have half of the problem solved.

Minimizing the problem and writing it back I discovered one leak: the
request was not terminating because the uploadImage (http://
codepad.org/DEbxFWyD ) was wrapped in one more function (adding some
entries in the DB). That was the problem because this wrapper was not
properly calling it's callback function and therefore uploadImage's
callback was not bring called.

But that still leaves me with this weird behavior:
1. Now the image is uploaded and saved and the POST requests finished
with 200
2. But the 1st request to node after this POST hangs (pending...). It
doesn't matter what the GET request is (a static file, a route, or a
non-exitent 404 URL)

If I reload this 1st request it loads properly. Where could the
problem be here?

PS: thanks for having me solve the first issue by minimizing and
rewriting that piece of code.

[Gabriel Petrovay]

For this request nothing gets printed with the handler below. It
simply hangs, regardless if I make it immediately after my post or
after some time, in the same tab or a new window.

app.on('request', function(request) {
    console.log(request.url);
});
app.on('connection', function() {
    console.log('connection');
});

How can I go deeper to see why nodejs is not receiving this request?

[Gabriel Petrovay]

Yes indeed, I can confirm that even though if I don't know what the
problem is.

I loaded the 1st URL after the POST with curl and with a 2nd browser
and it works OK. Also the same hanging behavior in the 2nd browser, so
its most probably my bug. Something in the client code (http://
codepad.org/hPv38KxZ) is not properly handling the XMLHttpRequest.

Thanks again! (I'll be back once I find the problem)

[Gabriel Petrovay]

The remaining problem I think is a bug and I posted it here:

https://github.com/joyent/node/issues/2263