streaming patches reloaded

[arthur debert]

Hi folks.
Following this thread (
http://groups.google.com/group/django-developers/browse_frm/thread/a0ae32019bb02f05/2c05c5363516ea76?q=streaming&rnum=2#2c05c5363516ea76)
, since I can't reply there ( over 30 days old, google tells me).

I have an app that uploads a 4mb file and is taking a ton of memory.
>From tickets 1484/1569 they are set as fixed, but I cannot find them on
django's source code. Are these patches active in trunk? If not,
anyones knows if they're working after the merge? Anyone using them?

Thanks a lot,
arthur

[James Bennett]

On 5/12/06, arthur debert <deb...@gmail.com> wrote:
> I have an app that uploads a 4mb file and is taking a ton of memory.
> >From tickets 1484/1569 they are set as fixed, but I cannot find them on
> django's source code. Are these patches active in trunk? If not,
> anyones knows if they're working after the merge? Anyone using them?

#1569 is marked as fixed, but it's for streaming *download*, not
streaming upload. #1484 is for upload, and is still marked as 'new'.

--
"May the forces of evil become confused on the way to your house."
-- George Carlin

[Ivan Sagalaev]

arthur debert wrote:

>I have an app that uploads a 4mb file and is taking a ton of memory.
>>From tickets 1484/1569 they are set as fixed, but I cannot find them on
>django's source code. Are these patches active in trunk? If not,
>anyones knows if they're working after the merge? Anyone using them?
>
>

1569 is applied to the current trunk.

As for 1484 I use Django with it (which is not surprising since I wrote
it :-) ) and I don't encounter any problems with form uploads, be it
with files or not.

My guess is that the main thing that holds this patch from checking in
is lack of documentation and tests. But I dont' feel myself confident to
do either thing being not native English speaker and not very
experienced test writer. Anyone?

Two things need to be documented there:
- new boolean setting "STORE_UPLOAD_ON_DISK"
- new key for request.FILES items: item['file'] which is a file-like
object for accessing uploaded file. The old way item['content'] still
works loading file in memory on first access.

[arthur debert]

Hi Ivan.

Just to make sure I understand:
- after patching django I 've set:
STORE_UPLOAD_ON_DISK = True

I am trying to use this on the admin. Any more steps I must take?

thanks a lot
(for the patch and help)
arthur

[Istvan Albert]

yeah, this is really a huge wart ,

what Django does here - holds everything in memory - works only for the
simplest cases thus makes django unsuited for a whole slew of web
applications

too bad that in a news environment uploading larger files is not a
typical use case, that's why this fix takes so long to be applied,

folks, just please take care of this already, what you have in place is
fundamentally wrong

[Ivan Sagalaev]

arthur debert wrote:

>Hi Ivan.
>
>Just to make sure I understand:
>- after patching django I 've set:
>STORE_UPLOAD_ON_DISK = True
>
>I am trying to use this on the admin. Any more steps I must take?
>
>

Hm... I don't think it would help very much in admin. The patch provides
an interface to an open file-like object stored on disk but admin still
accesses it the old way so the file will be immediately loaded in
memory. However I think that there _will_ be a performance gain even in
this case since the patch will stream the file on disk when it is
received by the web server. Without the patch this file is stored in
memory like 2 or 3 times (including temporary storages when copying).