LibMongoose: File upload (hugh files)

[Marcel Wirtz]

Hello,

we try to implement file uploading for our application using LibMongoose, but we have some problems with files larger than 530 MB.

1) Server returns "400 Bad request".

2) The used RAM grows with the upload up to more than 530 MB.

3) If you start a second upload it could be more than 1060 MB in RAM

It would be practical not to store the whole fileupload in RAM and to allow to write the file part by part and not only at once when complete.

[Sergey Lyubka]

Hi Marcel,

when uploaded file is spooled to the filesystem, user callback should still be called, and POST content be exposed to the caller. I see two options:

   1. memory-map spooled file. conn->content, conn->content_len will point to the memory mapped file in the same way as it was a usual small POST request

   2. add conn->filep attribute which will be a "FILE *" stream pointer of uploaded file. Either conn->content or conn->filep might hold uploaded data.

Could you describe your use case please, what uploads are used for, how frequently, how large?

--
You received this message because you are subscribed to the Google Groups "mongoose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-user...@googlegroups.com.
To post to this group, send email to mongoos...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongoose-users.
For more options, visit https://groups.google.com/d/optout.

[Marcel Wirtz]

Hello Sergey,

I used the upload.c example (https://github.com/cesanta/mongoose/blob/master/examples/upload.c) and a normal Browser (FireFox, Chrome, IE)  to test it and for files smaller than 530 MB this works fine. With larger files, it uploads the file and response with a "400 Bad request" error.

I debuged this yesterday and found that after the upload the conn->content_len is set to -1 in mongoose.c line 2086 because re->request_method is filled with something like "------WebKitFormBoundarybZokRVDBKdAGemWa\r\nContent-Disposition:" or "------------------------bZokRVDBKdAGemWa\r\nContent-Disposition:" so there is no valid http method given and not "HTTP/".

The case we need this is: 

We have a client application which uses libCurl to upload files to a server which runs with LibMongoose. That can be files with any size.

[Marcel Wirtz]

Hello Sergey,

i have something to attend:

You wrote that we can use the conn->filep attribute but there is no filep in conn and we can not save the files part by part because the whole file is uploaded complete before Mongoose calls the handler callback so the complete file has been stored in RAM.

Am Donnerstag, 8. Mai 2014 17:58:56 UTC+2 schrieb Sergey Lyubka:

[Marcel Wirtz]

Still need help.