question concerning the storage of big uploads in a temporary file

[william opensource4you]

Hi,

I've just looking for the best solution to avoid in-memory big uploads.

By looking at Lighhttp code, they are using mkstemp to store the
uploaded files in a temporary folder: /var/tmp.

Thus my question is: does any body knows a shred library that will
manage such "buffer" and store them into a temporary file ?

This sounds a quite "generic" feature :-) So instead of re-inventing
the wheel ...

Thanks

[Jonas H.]

On 03/04/2012 10:28 AM, william opensource4you wrote:
> I've just looking for the best solution to avoid in-memory big uploads.
>
> By looking at Lighhttp code, they are using mkstemp to store the
> uploaded files in a temporary folder: /var/tmp.

Why not just rely on the operating system's ability to swap?

[Andrey Grygoryev]

It's not a good solution, because swapping can slow down other processes. Also there are a lot of servers without swap or with small swap.

04.03.2012 12:18 пользователь "Jonas H." <jo...@lophus.org> написал:

[william opensource4you]

AFAIK Cherokee does not store, but lighttp is well storing in temporary files.

Fapws is designed to run on small hw. It could be that the posted file is bigger than the available memory (swap included).

;-(

W

[Leonardo Gonzalez]

http://docs.python.org/library/tempfile.html ?

2012/3/4 william opensource4you <willia...@gmail.com>

[william opensource4you]

Indeed, but I would have a flexible solution :-)
Storing every requests to a temporary file will drastically reduce the
overall performance.
Thus, here my goal is to store big requests only.

On the other hand the impacted line of code is the line 700 in
mainloop.c when doing the memcpy.
Thus tempfile.py must be translated into a C peace of code.

As stated by Jonas, we could rely on the swap capabilities of the OS.
This sounds to be a very valid remarks.
Unfortunately I running several instances of Fapws on a small virtual
server having 256MB of ram (+512mb of swap).
Thus, what will happen if a file (HTTP POST) of 1GB is send ?

I cannot imagine that a "smart" buffer library does not exist in C :-).

I'm currently reading mmap implementations.

2012/3/10 Leonardo Gonzalez <medi...@gmail.com>: