UnicodeDecodeError after update of trunk

[frank h.]

Hi,
I get the following stacktrace after updating trunk (to post-unicode-
branch-merge)
I am posting this here, because I don't see any of my views involved
in the stacktrace and I have a hard time figuring out what is wrong
(having followed all the steps here
http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist)

basically, my views.py file is written in UTF-8 and has #-*- coding:
utf-8 -*- on the top
then I am trying to do something like

channel = u'kortsändningar'
JobStatus.objects.filter(channel__name__istartswith=channel)

note the 'ä' character in the channel. calling the view then causes
this stacktrace

Traceback (most recent call last):

File "/Users/frank/Development/python-apps/site-packages/django/core/
servers/basehttp.py", line 272, in run
self.result = application(self.environ, self.start_response)

File "/Users/frank/Development/python-apps/site-packages/django/core/
servers/basehttp.py", line 614, in __call__
return self.application(environ, start_response)

File "/Users/frank/Development/python-apps/site-packages/django/core/
handlers/wsgi.py", line 190, in __call__
response = self.get_response(request)

File "/Users/frank/Development/python-apps/site-packages/django/core/
handlers/base.py", line 111, in get_response
return debug.technical_500_response(request, *sys.exc_info())

File "/Users/frank/Development/python-apps/site-packages/django/
views/debug.py", line 141, in technical_500_response
return HttpResponseServerError(t.render(c), mimetype='text/html')

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 181, in render
return self.nodelist.render(context)

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 736, in render
bits.append(self.render_node(node, context))

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 749, in render_node
return node.render(context)

File "/Users/frank/Development/python-apps/site-packages/django/
template/defaulttags.py", line 134, in render
nodelist.append(node.render(context))

File "/Users/frank/Development/python-apps/site-packages/django/
template/defaulttags.py", line 228, in render
return self.nodelist_true.render(context)

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 736, in render
bits.append(self.render_node(node, context))

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 749, in render_node
return node.render(context)

File "/Users/frank/Development/python-apps/site-packages/django/
template/defaulttags.py", line 134, in render
nodelist.append(node.render(context))

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 785, in render
return self.filter_expression.resolve(context)

File "/Users/frank/Development/python-apps/site-packages/django/
template/__init__.py", line 603, in resolve
obj = func(obj, *arg_vals)

File "/Users/frank/Development/python-apps/site-packages/django/
template/defaultfilters.py", line 25, in _dec
args[0] = force_unicode(args[0])

File "/Users/frank/Development/python-apps/site-packages/django/
utils/encoding.py", line 42, in force_unicode
s = unicode(s, encoding, errors)

UnicodeDecodeError: 'utf8' codec can't decode bytes in position
155-156: invalid data

bug? my mistake?
any help is appreciated!
thanks,
-frank

[Malcolm Tredinnick]

On Fri, 2007-07-13 at 00:48 -0700, frank h. wrote:
> Hi,
> I get the following stacktrace after updating trunk (to post-unicode-
> branch-merge)
> I am posting this here, because I don't see any of my views involved
> in the stacktrace and I have a hard time figuring out what is wrong
> (having followed all the steps here
> http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist)
>
> basically, my views.py file is written in UTF-8 and has #-*- coding:
> utf-8 -*- on the top
> then I am trying to do something like
>
> channel = u'kortsändningar'
> JobStatus.objects.filter(channel__name__istartswith=channel)
>
> note the 'ä' character in the channel. calling the view then causes
> this stacktrace

The error is during template rendering, not inside your view function.
So although it might well be caused by something you're doing in the
view, we have to dive a little deeper.

Trimming the leading part of the traceback for brevity, we have...

>
> Traceback (most recent call last):
>

[...]

> File "/Users/frank/Development/python-apps/site-packages/django/
> template/defaultfilters.py", line 25, in _dec
> args[0] = force_unicode(args[0])

What is args[0] here?

>
> File "/Users/frank/Development/python-apps/site-packages/django/
> utils/encoding.py", line 42, in force_unicode
> s = unicode(s, encoding, errors)

And what is 's' here?

In both cases you can see these values in the debugging view page by
selecting "local vars".

For bonus points, use one of the earlier lines in the traceback to work
out which filter is being executed (possible the line that says "obj =
func(obj, *arg_vals)" will have the information in the local variables.
That will help you narrow down where in your template the problem is
being raised and so you can verify what information is trying to be
displayed.

The error is saying it is trying to convert something that is a
bytestring (and a real string object, not just something with a __str__
method; something where type(s) is "str") and the data isn't UTF-8 for
some reason. One (apparently) common way to trigger this error is to be
trying to return Unicode data from your __str__ method (forgetting to
encode it to UTF-8 first). However, there may be other ways to trigger
this, too.

Regards,
Malcolm

--
Two wrongs are only the beginning.
http://www.pointy-stick.com/blog/

[Malcolm Tredinnick]

On Fri, 2007-07-13 at 00:48 -0700, frank h. wrote:

> Hi,
> I get the following stacktrace after updating trunk (to post-unicode-
> branch-merge)
> I am posting this here, because I don't see any of my views involved
> in the stacktrace and I have a hard time figuring out what is wrong
> (having followed all the steps here
> http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist)

By the way, this is probably a more appropriate thread for django-users,
since it's not really involved with the development of Django itself. I
wasn't concentrating when I hit reply.

Malcolm

--
The early bird may get the worm, but the second mouse gets the cheese.
http://www.pointy-stick.com/blog/

[frank h.]

> By the way, this is probably a more appropriate thread for django-users,
> since it's not really involved with the development of Django itself. I
> wasn't concentrating when I hit reply.

ups, my bad, but thanks for the answer anyway! I will look at the
things you pointed out