PicklingError with Queryset Refactor
Julien
I'm back again with an issue that I got since upgrading to queryset-
refactor. That issue appears at run time. Every page on my site caches
the sidebar, which contains some expensive randomly generated content.
Since I upgraded to qs-rf I get the following error for every page of
the site:
Exception Type: PicklingError at /
Exception Value: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed
Here's the traceback:
File "E:\Software\workspace\django\django\core\handlers\base.py" in
get_response
82. response = callback(request, *callback_args,
**callback_kwargs)
File "E:\Software\workspace\django\django\utils\decorators.py" in
_wrapped_view
48. response = view_func(request, *args, **kwargs)
File "E:\Software\workspace\MYSITE\trunk\news\views.py" in frontpage
26. sidebar = populateSidebar(request)
File "E:\Software\workspace\MYSITE\trunk\news\views.py" in
populateSidebar
189. cache.set('sidebar', sidebar, 60 * 2)
File "E:\Software\workspace\django\django\core\cache\backends
\filebased.py" in set
68. pickle.dump(value, f, pickle.HIGHEST_PROTOCOL)
Now here's the function that seems to pose probem:
def populateSidebar(request=None):
cached_sidebar = cache.get('sidebar')
if cached_sidebar:
return cached_sidebar
else:
sidebar = {}
sidebar["countries"] =
Country.objects.exclude(id=1).order_by('name')
sidebar["photo"] =
FlickrPhoto.objects.filter(aspect_ratio__gte=1).order_by('?')[0]
cache.set('sidebar', sidebar, 60 * 2)
return sidebar
I apologise for posting a piece of my code rather than a standalone
working example, but the error seems to come from the cache.set()
method so I guess the issue should "easily" be reproduced.
Any idea what's wrong here?
Thanks heaps!
Julien
Malcolm Tredinnick
On Sun, 2008-04-27 at 05:16 -0700, Julien wrote:
> Hi,
>
> I'm back again with an issue that I got since upgrading to queryset-
> refactor. That issue appears at run time. Every page on my site caches
> the sidebar, which contains some expensive randomly generated content.
> Since I upgraded to qs-rf I get the following error for every page of
> the site:
>
> Exception Type: PicklingError at /
> Exception Value: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
Querysets aren't picklable. It's really hard! I'll work on it at some
point, but... it's really hard!
Turn them into a list instead and cache that.
Malcolm
--
Tolkien is hobbit-forming.
http://www.pointy-stick.com/blog/
Malcolm Tredinnick
On Sun, 2008-04-27 at 22:27 +1000, Malcolm Tredinnick wrote:
[...]
> Querysets aren't picklable. It's really hard! I'll work on it at some
> point, but... it's really hard!
To be more specific: I know what has to be done and have something
half-implemented. It's fairly deep internals stuff, though, so it
requires a bit more thinking and testing and is a little lower priority
than some other missing features.
Caching a list version of the queryset is almost certainly more
efficient (it requires you to turn the queryset into a list, which is a
small hassle): if we pickle a queryset, we're going to have to load all
the results from the database, so now you're pickling the results plus
all the queryset + query overhead. If you cache the list, you get rid of
that second chunk.
Regards,
Malcolm
--
Many are called, few volunteer.
http://www.pointy-stick.com/blog/
Julien
sidebar["countries"] = [country for country in
Country.objects.exclude(id=1).order_by('name')]
Now, just out of curiosity, why didn't I get an error before using
queryset refactor. I can't be sure that the caching actually happened,
but there was no exception raised.
Cheers,
Julien
On Apr 27, 10:37 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Malcolm Tredinnick
On Sun, 2008-04-27 at 05:45 -0700, Julien wrote:
> Thanks Malcolm, I turned the queryset into a list, and it worked:
> sidebar["countries"] = [country for country in
> Country.objects.exclude(id=1).order_by('name')]
list(Country.objects.exclude(id=1).order_by('name') will be faster and
less code to write.
>
> Now, just out of curiosity, why didn't I get an error before using
> queryset refactor. I can't be sure that the caching actually happened,
> but there was no exception raised.
It's actually the Query class inside the QuerySet that is causing the
problem (partly because of the connection to the database backend, which
we have to not pickle); that contains a lot of stuff that didn't exist
before. There are also subtleties in the current implementation because
there's a lot more internal state to manage and once you unpickle a
queryset it's reasonable to expect you can still use it to further
filter the results, which means getting the internal state right on
unpickling.
Malcolm
--
Save the whales. Collect the whole set.
http://www.pointy-stick.com/blog/