TemporaryFileUploadHandler leaving tempfiles behind on disk

[msoulier]

Hi,

I'm using a ModelForm to upload a large file that ends up being
written to a tempfile on disk. When I call form.save(), the file gets
copied to the final filename, but the tempfile is left behind on disk.

I am using a custom handler, which is a TemporaryFileUploadHandler
subclass, but each method I override calls the parent class' method
first.

Is the tempfile supposed to be left behind?

Thanks

[Emiliano Dalla Verde Marcozzi]

2013/3/4 msoulier <msou...@digitaltorque.ca>

Hello,

I am not an experienced django developer so maybe this solution doesnt

follow 'the django way', but as i read at:

http://docs.nullpobug.com/django/trunk/django.core.files.uploadhandler-pysrc.html#FileUploadHandler.new_file

TemporaryFileUploadHandler subclass FileUploadHandler. The 'new_file' method at FileUploadHandler saves

the file name at the class instance as 'file_name' (self.file_name). So, i would try to overrite the 'file_complete'

method from my own class that subclass TemporaryFileUploadHandler and do something like:

class MyTemporaryFileUploadHandler(TemporaryFileUploadHandler):

    def file_complete(self, file_size):

        if os.path.isfile(self.file_name):

            os.unlink(self.file_name)

        super(TemporaryFileUploadHandler, self).file_complete(file_size)

I'm not sure if self.file_name points to the temporary file, so maybe this is not util.

--
"Code without tests is broken by design." - Jacob Kaplan-Moss
Show me the mone ... code!: https://bitbucket.org/edvm

[Michael P. Soulier]

On 04/03/13 Emiliano Dalla Verde Marcozzi said:

> method from my own class that subclass TemporaryFileUploadHandler and do
> something like:

Yeah, I'm doing that, but it seems odd to me that the handler leaves the file
around, when it's supposed to be temporary, no?

Mike

[Scott White]

After experimenting the only way I could reliably detect and delete the stuck temp files is a bit of a hack to implement a post_save signal to delete the file, but I have to attach an attribute to the model object being saved to determine the name of the file in /tmp.

def save(self, *args, **kwargs):

    if hasattr(self.sample_file.file, 'temporary_file_path'):

        # A hack to avoid accumulating temp files in /tmp

        # Done before parent save() or else the FileField file is no longer a TemporaryUploadFile

        self.temp_file_path = self.sample_file.file.temporary_file_path()

    super(Sample, self).save(*args, **kwargs)

# Crazy hack ahead...needed for some weird behaviour where temp upload files are getting stuck in /tmp

def remove_temp_sample_file(sender, **kwargs):

    obj = kwargs['instance']

    if hasattr(obj, 'temp_file_path'):

        os.unlink(obj.temp_file_path)

# connect crazy hack to post_save

post_save.connect(remove_temp_sample_file, sender=Sample)

This is working for now, but I'd really like to know why sometimes the temp file gets stuck and sometimes it gets removed.

Thanks,

Scott

On Wednesday, March 6, 2013 1:53:09 PM UTC-5, Scott White wrote:

I'm seeing this same issue with a slight twist...

When uploading a files >2.5MB via a regular Django view or the Django Admin, the file uploads and the temp copy in /tmp is removed. When I upload using django-rest-framework via POST, the file uploads but the temporary copy remains in /tmp. I posted on the django-rest-framework but I'm not sure the issue is in the framework. Here's that post with some more info:

https://groups.google.com/forum/?fromgroups=#!topic/django-rest-framework/AdfJuZ6ay-0

Hopefully someone can clarify how the Django code is working, because I've seen some odd behaviour. Quoting the Django relevant info from the above post:

Also, there's something in the Django code I do not understand. The only place I've found where the temporary file is moved to it's final location is within django/core/files/storage.py. This is in FileSystemStorage._save():
                # This file has a file path that we can move.
                if hasattr(content, 'temporary_file_path'):
                    file_move_safe(content.temporary_file_path(), full_path)
                    content.close()
When I set a breakdance point on the file_move_safe method call, it never gets triggered even when Django removes the temp file successfully using the ModelForm regular view. Eeks! If I set the break point on the if statement, code execution stops. Then I can see the 'temporary_file_path' attribute is not present, explaining why the method wasn't called, but not why the file was removed. Double-eeks! Even stranger is that just by setting the break point on the if statement, the temp file gets stuck in /tmp using the ModelForm based method...have I found a true Heisenbug? I digress, this part is more a question for django-users, and I plan on posting there so I have a better understanding of what is happening.

I'm overriding clean() in my model to do some extra validation, and I can see the TemporaryUploadedFile within clean and it is open. However, if I try to close the file at the end of clean I get a Value Error stating "I/O operation on closed file". So I tried overriding the save method to call the parent save and then hopefully close or delete the temp file, but the FileField.file is no longer the temp version there.

Any ideas?

It was mentioned to write a custom file upload handler, but I'm a little unclear on how to do this or whether I could safely delete the temp file from there. 

Thanks,

Scott