InMemoryUploadedFile object retrieved from request.FILES got closed automatically on django 1.7

[Hong Yi]

Hello,

I am new to Django and have implemented two views and their corresponding template pages and they are working well in Django 1.6. However, when migrating to Django 1.7, I got an error "I/O operation on closed file" when getting to the second view function. I found out the root cause for this error is that on Django 1.7, InMemoryUploadedFile object retrieved from request.FILES got automatically closed at the end of the request by Django. I am trying to find a solution to work around this migration issue. Specifically, the first view and its template page ask users to browse files, and the files can be retrieved from request.FILES and saved to a global variable for use by the second view/request. Since the file got closed automatically at the end of the first request, when the second view tries to operate on this file, that "I/O operation on closed file" error results.

This is my first post, and I am trying to get some recommendations/suggestions on how to handle this use case to work around this new security feature implemented in Django 1.7 (i.e., automatically close file at the end of each request). Any help and suggestions are greatly appreciated.

Thanks,
Hong

[Russell Keith-Magee]

Hi Hong,

In this case, the solution isn't to work around Django - it's to work out why Django is getting in your way. The cause is a fundamental misunderstanding about how you should be looking at the web. 

Each request on a web site should be completely independent - you can't rely on shared state between one request and the "next" request. Two examples for why this is important:

1) The second request might be served by an entirely different server. Once you get into any sort of non-trivial deployment, you will have more than one web server to ensure availability; if a file has been uploaded on one server, it won't be available on the other server unless you're providing some sort of independent storage.

2) There's no guarantees that a single user will be responsible for two requests in a row. It's easy to think of a situation where there are two users using your website at the same time; if your website code makes any assumptions about request ordering, it makes a big difference whether the requests are handled as AABB or ABAB. If A and B are normal users, this might just be an annoying bug; if B is an attacker, then A's data could be stolen or compromised. 

You might claim that your website will never be big enough to require (1), or have enough users that (2) will be a problem - and that might be true - but it doesn't change the fact that web frameworks are built on the assumption that both (1) and (2) are going to happen, and the infrastructure they put in place will ensure that those two cases don't cause problems.

So - you shouldn't be using a global variable to store *anything*. (That's generally good advice for programming anyway, but it's doubly important for web programming). You also shouldn't be loading data into memory on the assumption that it will be used in the "next" request. If I were an attacker, and I found out that you were doing this, I would make a whole stack of the first requests, and then never make the second request - and I've just starved your server of memory.

If you need to have a 2 step process where one view selects a file, and the second view "handles" the file, then what you should be passing around is a filename, or some other reference that lets you retrieve a file. Then, the two views should be built so that they are completely independent. View 1 provides a way for a user to select a file. View 2 opens, reads, processes and closes the file.

Memory usage shouldn't be a concern here. When you open a file, you don't have to load the whole thing into memory to pass it around. Operating systems are really good at handling file pointers, which are just an index into an open file. You don't have to store any more data than the character you're currently pointing at. Most file formats are developed so that you don't need to read the *entire* file into memory in one pass - they're optimised to allow you to read them "on demand". 

In short - don't work around this problem; fix the underlying cause of the problem, and restructure your app.

Hope this has been helpful!

Yours

Russ Magee %-)

[Hong Yi]

Many thanks for the detailed explanation, Russ! Really appreciate it. Now I understand much better why django 1.7 enforces this. I will follow your advice to re-structure my code to make it work on django 1.7.

Best regards,
Hong

[Lin Cai]

Russ,

I have a view which create a new thread to process files user uploaded. I found either InMemoryUploadedFile or TempUploaedFile is closed in the new thread. I think that might be caused by the exit of the view thread. Do you have any idea how to re-open the closed UploadedFile or clone one?

It is related to this bug I found: https://bugs.launchpad.net/horizon/+bug/1451429