Django 1.5 using a cached HttpResponse with WSGI has an empty body

[SteveB]

With the change to HttpResponse made in Django 1.5, I'm finding that in my code, which caches a generated response, results in an empty body when that page is requested a second time. The first time the page is requested, it is not in the cache, and the page is generated normally and added to the cache. A subsequent request for the same page finds the response in the cache and that response is returned, but with a content length of zero.

The reason is that the HttpResponse in Django 1.5 does not reset the content iterator when the content is requested to be iterated over again (the next time the response content is required).

I note the comments made about the way an iterator should behave when requested to iterate again:

https://code.djangoproject.com/ticket/13222

and the code which was added to explicitly prevent a reiteration from resetting the iterator. However, that causes a problem when using cached responses.

The HttpResponse in my case was not created by passing an iterator to HttpResponse. It is just using a string.

The problem is that the __iter__ method of HttpResponse contains the line:

    if not hasattr(self, '_iterator'):
      self._iterator = iter(self._container)

This prevents the iterator being reset the next time it is required to iterate over the content.

_container still has the original content, but __iter__ does not reset the iterator as _iterator exists as an attribute since the first time that response was returned. The cached response is returning a used iterator, which returns no content.

I suspect this is a bug. Any thoughts?

What about a work-around in the meantime?

When I retrieve the response from the cache, I could do:

    response._iterator = iter(response._container)

or:

del response._iterator

This works, but makes my code dependent on the internals of the HttpResponse class, which isn't great. Any better ideas?

Kind regards,

Steve

P.S. I posted a message about this on Django users group about a week ago, but got no reply, hence posting here to get the views of the Django developers.

[Aymeric Augustin]

On 25 mars 2013, at 18:02, SteveB <smbr...@gmail.com> wrote:

> I suspect this is a bug. Any thoughts?

Yes, it's annoying, all the more since Django exposes response.content as an attribute.

> This works, but makes my code dependent on the internals of the HttpResponse class, which isn't great. Any better ideas?

I assume you're pickling the response to cache it. We could define a __getstate__ method to remove _iterator from what's pickled, like this:

def __getstate__(self):
state = self.__dict__.copy()
state.pop('_iterator', None)
return state

Could you file a ticket so we don't lose track of this problem?

--
Aymeric.

[SteveB]

Hi Aymeric,

Thanks for your reply.

Actually I'm just using a memory cache for the response, so I'm not pickling it.

My thoughts are:

1. The __iter__ method of a HttpResponse should create an instance of a separate iterator class to iterate over the container. It should not return self. The iterator class instance can return self in it's own __iter__ method. This, I think, would get around the issue raised in https://code.djangoproject.com/ticket/13222.
2. I accept that StreamingHttpResponse instances can only be iterated once, and are therefore not suitable for caching. That's something that my application can handle.

Do you think this would work OK?

I'll create a ticket to track this.

Thanks,

Stephen

[SteveB]

I created a ticket for this problem: https://code.djangoproject.com/ticket/20187