UnicodeDecodeError with memcache

[patrickk]

With using the cache_page decorator, I´m getting a Unicode Error on my
page:

UnicodeDecodeError at /spezialprogramme/
'ascii' codec can't decode byte 0x80 in position 0: ordinal not in
range(128)

Any ideas?

Thanks,
Patrick

[Iapain]

> UnicodeDecodeError at /spezialprogramme/
> 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in
> range(128)

try using smart_str, it it doesnt work then use less restrictive decode

[patrickk]

so you say I should use __str__ instead of __unicode__, right?
according to the django-docs, that´s not recommended though ...

[patrickk]

I´m sorry for bothering again, but our site has to be online soon and I
´m currently not able to use the cache_page decorator (which makes me
a bit nervous).
I did some research on smart_str, but I don´t know how and where to
use it and I also don´t see why this should solve the problem.

any feedback is highly appreciated.
btw: I´m able to use the low-level cache.

thanks,
patrick

[patrickk]

additional information:
I just checked the default encoding

>>> import sys
>>> sys.getdefaultencoding()
'ascii'

should that be something different (like utf-8)?
btw: all our data (database) and scripts are encoded with utf-8.

thanks,
patrick

[Iapain]

> >>> import sys
> >>> sys.getdefaultencoding()
>
> 'ascii'

Try using utf-8, btw it shouldnt be a problem. Lastly try using
changing charset to latin1 in your settings.py .. I guess your website
is not in english .. or it has some non-ascii char.

[patrickk]

why prefer latin1 over utf-8 (in settings.py)?

[Iapain]

What middleware you are using .. any external middleware?

[Jeremy Dunck]

On 8/27/07, patrickk <pat...@vonautomatisch.at> wrote:
>
> With using the cache_page decorator, I´m getting a Unicode Error on my
> page:
>
> UnicodeDecodeError at /spezialprogramme/
> 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in
> range(128)
>

patrickk, please give your whole stack trace. If it's not clear from
the stack trace, please also say which CACHE_BACKEND you're using.

Additionally, please give the value of the URL being requested
(request.path) and the value of settings.CACHE_MIDDLEWARE_KEY_PREFIX

[Jeremy Dunck]

On 8/28/07, patrickk <pat...@vonautomatisch.at> wrote:
>
> additional information:
> I just checked the default encoding
>
> >>> import sys
> >>> sys.getdefaultencoding()
> 'ascii'
>
> should that be something different (like utf-8)?
> btw: all our data (database) and scripts are encoded with utf-8.

patrickk, I'm not sure what lapain's experience with unicode is, but
please disregard the advice to switch defaultencoding to utf-8.
Despite appearances, it's *good* that python refuses to guess what
encoding you might wish to emit. You've found a bug in Django, and
it's better to fix Django than tell everybody to go hack their python
installs or change their processes with unusual configurations.

[patrickk]

thanks jeremy.
here´s the information:

CACHE_BACKEND = 'memcached://dedhost-sil-076.sil.at:11211/'

I´m not using the cache_middleware, because I´m using the per-view-
cache. Just re-checked the django-docs and hope I don´t misunderstand
the whole caching-thing - but according to the docs, the
cache_middleware is not necessary for the cache_page decorator.

instead of sending the whole stack, it´s probably easier to just check
it yourself:
http://skip.dedhost-sil-076.sil.at/trailer/

thanks,
patrick

On Aug 29, 4:49 am, "Jeremy Dunck" <jdu...@gmail.com> wrote:

[Michael Radziej]

On Wed, Aug 29, patrickk wrote:

> instead of sending the whole stack, it´s probably easier to just check
> it yourself:
> http://skip.dedhost-sil-076.sil.at/trailer/

File "/usr/lib/python2.4/site-packages/memcache.py" in _set
328. fullcmd = "%s %s %d %d %d\r\n%s" % (cmd, key, flags, time, len(val),
val)

It seems that memcached is not really prepared to take unicode cache keys.
Here, it constructs a cache value by pickle.dumps, so `val` is a bytestring.
key is still a unicode string. Combining both with the % operator won't
always work.

So, somewhere the cache backend need to use smart_str(). I'm not sure about
the right place, but django/core/cache/backends/memcached.py in set() would
be one place.

Michael

--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company

Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689

[patrickk]

I just changed
self._cache.set(key, value, timeout or self.default_timeout)
to
self._cache.set(smart_str(key), value, timeout or
self.default_timeout)
in memcached.py
and it seems to work.

Don´t know if that´s a proper solution though.

thanks,
patrick

On Aug 29, 10:32 am, Michael Radziej <m...@noris.de> wrote:
> On Wed, Aug 29, patrickk wrote:
> > instead of sending the whole stack, it´s probably easier to just check
> > it yourself:
> >http://skip.dedhost-sil-076.sil.at/trailer/
>
> File "/usr/lib/python2.4/site-packages/memcache.py" in _set
> 328. fullcmd = "%s %s %d %d %d\r\n%s" % (cmd, key, flags, time, len(val),
> val)
>
> It seems that memcached is not really prepared to take unicode cache keys.
> Here, it constructs a cache value by pickle.dumps, so `val` is a bytestring.
> key is still a unicode string. Combining both with the % operator won't
> always work.
>
> So, somewhere the cache backend need to use smart_str(). I'm not sure about
> the right place, but django/core/cache/backends/memcached.py in set() would
> be one place.
>
> Michael
>
> --
> noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -

> Tel +49-911-9352-0 - Fax +49-911-9352-100http://www.noris.de- The IT-Outsourcing Company

[Michael Radziej]

On Wed, Aug 29, patrickk wrote:

>
> I just changed
> self._cache.set(key, value, timeout or self.default_timeout)
> to
> self._cache.set(smart_str(key), value, timeout or
> self.default_timeout)
> in memcached.py
> and it seems to work.
>
> Don´t know if that´s a proper solution though.

Me either ;-) It's probably not the full solution.

But please file a ticket so this doesn't get lost (with reference to this
thread).

Michael

--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100

http://www.noris.de - The IT-Outsourcing Company

[Jeremy Dunck]

On 8/29/07, Michael Radziej <m...@noris.de> wrote:
>
> On Wed, Aug 29, patrickk wrote:
>
> >
> > I just changed
> > self._cache.set(key, value, timeout or self.default_timeout)
> > to
> > self._cache.set(smart_str(key), value, timeout or
> > self.default_timeout)
> > in memcached.py
> > and it seems to work.
> >
> > Don´t know if that´s a proper solution though.
>
> Me either ;-) It's probably not the full solution.
>
> But please file a ticket so this doesn't get lost (with reference to this
> thread).

Well, this is the error I suspected, but it now occurs to me that I'm
not sure what version or rev of Django is being used here.

Trunk already had this fixed:
http://code.djangoproject.com/changeset/5718

patrickk, if you're trying to use unicode (especially urls) without
using a version after rev 5609 (0.96 was something like rev 4800),
then you're likely to run into other problems.

[patrickk]

I´m on revision 5646

[patrickk]

I´ll do a django-update on the weekend.
If the problem still exists, I´ll post here again.

Thanks for the answers,
Patrick

[Iapain]

> patrickk, I'm not sure what lapain's experience with unicode is, but
> please disregard the advice to switch defaultencoding to utf-8.

I didnt want this either, but some time you have to toil a bit for
debugging. I suggested him to use smart_str in post1, because i am
aware of recently merger of unicode branch but probably he wasnt or
using old version.