Problem with letters like ÄÖÕÜ in django admin

56 views
Skip to first unread message

zayatzz

unread,
Apr 15, 2010, 11:54:51 AM4/15/10
to Django users
Hello

I recieve this kind of error message when in admin:

Caught an exception while rendering: ('ascii', 'Madalal temperatuuril k
\xc3\xbcpsetatud stressivaba sea parim t\xc3\xbckk', 23, 24, 'ordinal
not in range(128)')

The text should actually be "Madalal remperatuuril küpsetatud
stressivaba sea parim tükk".

The model, that saves this information is:
class Retsept(models.Model):
name = models.CharField(max_length=100, verbose_name=_("Name"),
blank=False, null=False)
and other fields....


And the form for it is:
class RetseptForm(ModelForm):
about = forms.CharField(widget=TinyMCE(attrs={'style': 'width:370px',
'rows': 15, 'theme':"simple"}), label = _("text here"),
required=False)
tags = TagField(label = _("tags"),
widget=TagAutocomplete(attrs={'class':'required'}))
name = forms.CharField(label = _("Name"),
widget=forms.TextInput(attrs={'class':'required'}))
class Meta:
model = Retsept
fields = ('name', 'about', 'image', 'tags',)
widgets = {
'name': TextInput(attrs={'class': 'required' })
}

Can anyone tell me whats wrong with it?

Alan

Peter Herndon

unread,
Apr 15, 2010, 12:24:41 PM4/15/10
to django...@googlegroups.com
My immediate thought, looking at your error, is that your database is storing text data as ASCII rather than a unicode encoding (utf-8 being the most common). Perhaps you should check your database character set. IIRC, MySQL uses an ASCII character set by default, unless you choose otherwise during installation.

---Peter Herndon

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

Bill Freeman

unread,
Apr 15, 2010, 12:31:41 PM4/15/10
to django...@googlegroups.com
We probably need to see the template, and/or the full stack trace.

Or maybe the default characterset is declared incorrectly.

I think that something is trying to convert a unicode object to an
ascii object (because perhaps the default character set is ascii),
and characters like these don't have ascii representations. Or
something is trying to use a utf-8 string under the presumption
that it is ascii. Rendering to the web should probably be rendering
utf-8.

Are there custom template tags involved?

Bill

Karen Tracey

unread,
Apr 15, 2010, 12:33:11 PM4/15/10
to django...@googlegroups.com
On Thu, Apr 15, 2010 at 11:54 AM, zayatzz <alan.ke...@gmail.com> wrote:
Hello

I recieve this kind of error message when in admin:

When doing what, exactly, in admin?

 
Caught an exception while rendering: ('ascii', 'Madalal temperatuuril k
\xc3\xbcpsetatud stressivaba sea parim t\xc3\xbckk', 23, 24, 'ordinal
not in range(128)')

The text should actually be "Madalal remperatuuril küpsetatud
stressivaba sea parim tükk".

The model, that saves this information is:

The traceback from the error would be more helpful in pointing towards where the problem lies.

Karen

Xavier Ordoquy

unread,
Apr 15, 2010, 1:56:10 PM4/15/10
to django...@googlegroups.com
Hello,

I sometime faced this issue when the data were not properly formatted in the database.
I mean, there was stored in the database through another system that did not handle correctly charset conversions.
Sometime one stores unicodes or iso-8859-xx and says it is utf-8 or whatever.

What could possibly help us is to get the part of code where you extract the data from the form and store them in the model.

Regards,
Xavier.

zayatzz

unread,
Apr 15, 2010, 1:59:10 PM4/15/10
to Django users
Header is like that:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://
www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

and database is:
MySQL charset: UTF-8 Unicode (utf8)
and collation is utf8_unicode_ci

Whole traceback is:
TemplateSyntaxError at /admin/retsept/vote/
Caught an exception while rendering: ('ascii', 'Sea\xc5\xa1a\xc5\xa1l
\xc3\xb5kk ', 3, 4, 'ordinal not in range(128)')
Request Method: GET
Request URL: http://DOMAIN/admin/retsept/vote/
Exception Type: TemplateSyntaxError
Exception Value:
Caught an exception while rendering: ('ascii', 'Sea\xc5\xa1a\xc5\xa1l
\xc3\xb5kk ', 3, 4, 'ordinal not in range(128)')
Exception Location: /www/apache/domains/HOST/htdocs/django/
Django-1.1.1/django/template/debug.py in render_node, line 81
Python Executable: /usr/local/bin/python
Python Version: 2.6.4
Python Path: ['/www/apache/domains/HOST/htdocs/grill', '/www/apache/
domains/HOST/htdocs', '/www/apache/domains/HOST/htdocs/django/django-
tagging-autocomplete-0.3.1', '/www/apache/domains/HOST/htdocs/django/
tagging-trunk', '/www/apache/domains/HOST/htdocs/django/django-
tinymce-1.5', '/www/apache/domains/HOST/htdocs/django/sorl-
thumbnail-3.2.5', '/www/apache/domains/HOST/htdocs/django/
Django-1.1.1', '/www/apache/domains/HOST/htdocs/cgi-bin', '/usr/local/
lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg', '/usr/local/
lib/python2.6/site-packages/zope.interface-3.5.2-py2.6-freebsd-7.2-
STABLE-amd64.egg', '/usr/local/lib/python2.6/site-packages/flup-1.0.2-
py2.6.egg', '/usr/local/lib/python2.6/site-packages/SQLAlchemy-0.5.5-
py2.6.egg', '/usr/local/lib/python2.6/site-packages/nose-0.10.4-
py2.6.egg', '/usr/local/lib/python2.6/site-packages/
python_dateutil-1.4.1-py2.6.egg', '/usr/local/lib/python2.6/site-
packages/pytz-2009r-py2.6.egg', '/usr/local/lib/python2.6/site-
packages/M2Crypto-0.20-py2.6-freebsd-7.2-STABLE-amd64.egg', '/usr/
local/lib/python26.zip', '/usr/local/lib/python2.6', '/usr/local/lib/
python2.6/plat-freebsd7', '/usr/local/lib/python2.6/lib-tk', '/usr/
local/lib/python2.6/lib-old', '/usr/local/lib/python2.6/lib-dynload',
'/usr/local/lib/python2.6/site-packages', '/usr/local/lib/python2.6/
site-packages/Numeric', '/usr/local/lib/python2.6/site-packages/PIL',
'/usr/local/lib/python2.6/site-packages/gtk-2.0']
Server time: neljap., 15 Apr 2010 20:53:03 +0300
Template error

In template /www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/
contrib/admin/templates/admin/change_list.html, error at line 78
Caught an exception while rendering: ('ascii', 'Sea\xc5\xa1a\xc5\xa1l
\xc3\xb5kk ', 3, 4, 'ordinal not in range(128)')
68 {% endif %}
69 {% endblock %}
70
71 <form action="" method="post"{% if cl.formset.is_multipart %}
enctype="multipart/form-data"{% endif %}>
72 {% if cl.formset %}
73 {{ cl.formset.management_form }}
74 {% endif %}
75
76 {% block result_list %}
77 {% if action_form and actions_on_top and
cl.full_result_count %}{% admin_actions %}{% endif %}
78 {% result_list cl %}
79 {% if action_form and actions_on_bottom and
cl.full_result_count %}{% admin_actions %}{% endif %}
80 {% endblock %}
81 {% block pagination %}{% pagination cl %}{% endblock %}
82 </form>
83 </div>
84 </div>
85 {% endblock %}
86
Traceback Switch to copy-and-paste view

/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/core/
handlers/base.py in get_response
response = callback(request, *callback_args,
**callback_kwargs) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/contrib/
admin/options.py in wrapper
return self.admin_site.admin_view(view)(*args,
**kwargs) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/views/
decorators/cache.py in _wrapped_view_func
response = view_func(request, *args, **kwargs) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/contrib/
admin/sites.py in inner
return view(request, *args, **kwargs) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/contrib/
admin/options.py in changelist_view
], context, context_instance=context_instance) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/shortcuts/
__init__.py in render_to_response
return HttpResponse(loader.render_to_string(*args, **kwargs),
**httpresponse_kwargs) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
loader.py in render_to_string
return t.render(context_instance) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
return self.nodelist.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
bits.append(self.render_node(node, context)) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
debug.py in render_node
result = node.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
loader_tags.py in render
return compiled_parent.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
return self.nodelist.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
bits.append(self.render_node(node, context)) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
debug.py in render_node
result = node.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
loader_tags.py in render
return compiled_parent.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
return self.nodelist.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
bits.append(self.render_node(node, context)) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
debug.py in render_node
result = node.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
loader_tags.py in render
result = self.nodelist.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
bits.append(self.render_node(node, context)) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
debug.py in render_node
result = node.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
loader_tags.py in render
result = self.nodelist.render(context) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
__init__.py in render
bits.append(self.render_node(node, context)) ...
▶ Local vars
/www/apache/domains/HOST/htdocs/django/Django-1.1.1/django/template/
debug.py in render_node
raise wrapped ...
▶ Local vars
Request information

GET
No GET data
POST
No POST data
FILES
No FILES data
COOKIES
Variable Value
__utma
'149913952.95436242.1271238658.1271342835.1271353741.3'
__utmb
'149913952.2.10.1271353741'
__utmc
'149913952'
sessionid
'468b34de7eea0d6e9a4694ef5d10bc90'
__utmz
'149913952.1271238658.1.1.utmcsr=(direct)|utmccn=(direct)|
utmcmd=(none)'
META
Variable Value
DOCUMENT_ROOT
'/www/apache/domains/HOST/htdocs'
GATEWAY_INTERFACE
'CGI/1.1'
HTTP_ACCEPT
'application/xml,application/xhtml+xml,text/html;q=0.9,text/
plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET
'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING
'gzip,deflate,sdch'
HTTP_ACCEPT_LANGUAGE
'en-US,en;q=0.8'
HTTP_CONNECTION
'keep-alive'
HTTP_COOKIE
'__utmz=149913952.1271238658.1.1.utmcsr=(direct)|utmccn=(direct)|
utmcmd=(none); sessionid=468b34de7eea0d6e9a4694ef5d10bc90;
__utma=149913952.95436242.1271238658.1271342835.1271353741.3;
__utmc=149913952; __utmb=149913952.2.10.1271353741'
HTTP_HOST
'DOMAIN'
HTTP_REFERER
'http://DOMAIN/admin/'
HTTP_USER_AGENT
'Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.2 (KHTML,
like Gecko) Chrome/5.0.342.9 Safari/533.2'
PATH
'/bin:/usr/local/bin:/usr/bin'
PATH_INFO
u'/admin/retsept/vote/'
PATH_TRANSLATED
'/www/apache/domains/HOST/htdocs/admin/retsept/vote/'
QUERY_STRING
''
REDIRECT_STATUS
'200'
REDIRECT_UNIQUE_ID
'S8dSf9W0H7IAAWs2zgA'
REDIRECT_URL
'/admin/retsept/vote/'
REMOTE_ADDR
'84.50.47.52'
REMOTE_PORT
'58274'
REQUEST_METHOD
'GET'
REQUEST_URI
'/admin/retsept/vote/'
SCRIPT_FILENAME
'/www/apache/domains/HOST/htdocs/cgi-bin/django.fcgi'
SCRIPT_NAME
u''
SERVER_ADDR
'213.180.31.178'
SERVER_ADMIN
'a...@elkdata.com'
SERVER_NAME
'HOST'
SERVER_PORT
'80'
SERVER_PROTOCOL
'HTTP/1.1'
SERVER_SIGNATURE
'<ADDRESS>Apache/1.3.41 Server at HOST Port 80</ADDRESS>\n'
SERVER_SOFTWARE
'Apache/1.3.41 (Unix) mod_fastcgi/2.4.6 PHP/5.2.11 mod_ssl/2.8.31
OpenSSL/0.9.8e'
UNIQUE_ID
'S8dSf9W0H7IAAWs2zgA'
wsgi.errors
<flup.server.fcgi_base.OutputStream object at 0x804453190>
wsgi.input
<flup.server.fcgi_base.InputStream object at 0x80443fe90>
wsgi.multiprocess
False
wsgi.multithread
True
wsgi.run_once
False
wsgi.url_scheme
'http'
wsgi.version
(1, 0)
Settings
Using settings module grill.settings
Setting Value
ABSOLUTE_URL_OVERRIDES
{}
ADMINS
()
ADMIN_FOR
()
ADMIN_MEDIA_PREFIX
'/grill/meedia/media/'
ALLOWED_INCLUDE_ROOTS
()
APPEND_SLASH
True
AUTHENTICATION_BACKENDS
('django.contrib.auth.backends.ModelBackend',)
BANNED_IPS
()
CACHE_BACKEND
'locmem://'
CACHE_MIDDLEWARE_KEY_PREFIX
''
CACHE_MIDDLEWARE_SECONDS
600
COMMENTS_ALLOW_PROFANITIES
False
COMMENTS_BANNED_USERS_GROUP
None
COMMENTS_FIRST_FEW
0
COMMENTS_MODERATORS_GROUP
None
COMMENTS_SKETCHY_USERS_GROUP
None
DATABASE_ENGINE
'mysql'
DATABASE_HOST
''
DATABASE_NAME
'vhost8684s1'
DATABASE_OPTIONS
{}
DATABASE_PASSWORD
'********************'
DATABASE_PORT
''
DATABASE_USER
'vhost8684s1'
DATETIME_FORMAT
'N j, Y, P'
DATE_FORMAT
'N j, Y'
DEBUG
True
DEBUG_PROPAGATE_EXCEPTIONS
False
DEFAULT_CHARSET
'utf-8'
DEFAULT_CONTENT_TYPE
'text/html'
DEFAULT_FILE_STORAGE
'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL
'gr...@selver.eu'
DEFAULT_INDEX_TABLESPACE
''
DEFAULT_TABLESPACE
''
DISALLOWED_USER_AGENTS
()
EMAIL_HOST
'localhost'
EMAIL_HOST_PASSWORD
'********************'
EMAIL_HOST_USER
''
EMAIL_PORT
25
EMAIL_SUBJECT_PREFIX
'[Django] '
EMAIL_USE_TLS
False
FILE_CHARSET
'utf-8'
FILE_UPLOAD_HANDLERS
('django.core.files.uploadhandler.MemoryFileUploadHandler',
'django.core.files.uploadhandler.TemporaryFileUploadHandler')
FILE_UPLOAD_MAX_MEMORY_SIZE
2621440
FILE_UPLOAD_PERMISSIONS
None
FILE_UPLOAD_TEMP_DIR
None
FIXTURE_DIRS
()
FORCE_SCRIPT_NAME
None
IGNORABLE_404_ENDS
('mail.pl', 'mailform.pl', 'mail.cgi', 'mailform.cgi', 'favicon.ico',
'.php')
IGNORABLE_404_STARTS
('/cgi-bin/', '/_vti_bin', '/_vti_inf')
INSTALLED_APPS
['django.contrib.auth', 'django.contrib.contenttypes',
'django.contrib.sessions', 'django.contrib.sites',
'django.contrib.admin', 'tinymce', 'sorl.thumbnail', 'tagging',
'tagging_autocomplete', 'retsept', 'grill.profile']
INTERNAL_IPS
()
LANGUAGES
(('et', 'Estonian'),)
LANGUAGES_BIDI
('he', 'ar', 'fa')
LANGUAGE_CODE
'et'
LANGUAGE_COOKIE_NAME
'django_language'
LOCALE_PATHS
()
LOGIN_REDIRECT_URL
'/profile/'
LOGIN_URL
'/login/'
LOGOUT_REDIRECT_URL
'/'
LOGOUT_URL
'/accounts/logout/'
MANAGERS
()
MEDIA_ROOT
'/www/apache/domains/HOST/htdocs/grill/meedia/'
MEDIA_URL
'http://mekaia.com/grill/meedia/'
MIDDLEWARE_CLASSES
('django.middleware.common.CommonMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')
MONTH_DAY_FORMAT
'F j'
PASSWORD_RESET_TIMEOUT_DAYS
'********************'
PREPEND_WWW
False
PROFANITIES_LIST
'********************'
ROOT_URLCONF
'grill.urls'
SECRET_KEY
'********************'
SEND_BROKEN_LINK_EMAILS
False
SERVER_EMAIL
'root@localhost'
SESSION_COOKIE_AGE
1209600
SESSION_COOKIE_DOMAIN
None
SESSION_COOKIE_NAME
'sessionid'
SESSION_COOKIE_PATH
'/'
SESSION_COOKIE_SECURE
False
SESSION_ENGINE
'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE
False
SESSION_FILE_PATH
None
SESSION_SAVE_EVERY_REQUEST
False
SETTINGS_MODULE
'grill.settings'
SITE_ID
1
TAGGING_AUTOCOMPLETE_JS_BASE_URL
'/var/www/media/grill/js/jquery-autocomplete'
TEMPLATE_CONTEXT_PROCESSORS
('django.core.context_processors.auth',
'django.core.context_processors.i18n',
'django.core.context_processors.request',
'django.core.context_processors.media', 'grill.context.loginform')
TEMPLATE_DEBUG
True
TEMPLATE_DIRS
('/www/apache/domains/HOST/htdocs/grill/templates/',)
TEMPLATE_LOADERS
('django.template.loaders.filesystem.load_template_source',
'django.template.loaders.app_directories.load_template_source')
TEMPLATE_STRING_IF_INVALID
''
TEST_DATABASE_CHARSET
None
TEST_DATABASE_COLLATION
None
TEST_DATABASE_NAME
None
TEST_RUNNER
'django.test.simple.run_tests'
TIME_FORMAT
'P'
TIME_ZONE
'Europe/Tallinn'
TRANSACTIONS_MANAGED
False
URL_VALIDATOR_USER_AGENT
'Django/1.1.1 (http://www.djangoproject.com)'
USE_ETAGS
False
USE_I18N
True
YEAR_MONTH_FORMAT
'F Y'

What i am doing in admin is opening a model administrating view. As
you can see from the beginning of traceback the url is /admin/retsept/
vote. Same happens when i try to change tags created by django-tagging.

Bill Freeman

unread,
Apr 15, 2010, 2:07:52 PM4/15/10
to django...@googlegroups.com
The problem is reported at line 78 of the template, which uses a tag
named "result_list". That isn't one of the standard tags. Where does
it come from? Can you provide the source code?

Bill

zayatzz

unread,
Apr 15, 2010, 2:27:24 PM4/15/10
to Django users
I think its standard tag. I have not added any tags in this project
and this string (result_list) does not exist in any of the pluggable
stuff (tagging, tinymce, sorl thumbnail). This string does exist in
django/contrib/admin/templatetags/admin_list.py though.

Alan

> ...
>
> read more »

Daniel Roseman

unread,
Apr 15, 2010, 2:32:30 PM4/15/10
to Django users

Almost every time I have seen this, it is an issue with the model's
__unicode__ method. Can you post that?
--
DR.

Bill Freeman

unread,
Apr 15, 2010, 2:52:17 PM4/15/10
to django...@googlegroups.com
I don't find it on
http://docs.djangoproject.com/en/1.1/ref/templates/builtins/ .

It may be provided by one of your installed apps. What {% load ... %} tags
are there in the template?

zayatzz

unread,
Apr 15, 2010, 2:53:45 PM4/15/10
to Django users
Well i already posted the model and i posted the form. The form is
used in a view like this:
rform = RetseptForm(request.POST, request.FILES, instance=retsept)
if request.method == 'POST':
if rform.is_valid():
rform.save()

And thats it

Alan

Daniel Roseman

unread,
Apr 15, 2010, 2:54:13 PM4/15/10
to Django users
On Apr 15, 7:52 pm, Bill Freeman <ke1g...@gmail.com> wrote:
> I don't find it on http://docs.djangoproject.com/en/1.1/ref/templates/builtins/.

No, it's not a builtin, it's part of the default admin - it renders
the changelist.
--
DR.

zayatzz

unread,
Apr 15, 2010, 2:56:09 PM4/15/10
to Django users
class Retsept(models.Model):
name = models.CharField(max_length=100, verbose_name=_("Name"),
blank=False, null=False)
user = models.ForeignKey(User)
image = RemovableImageField(upload_to=content_file_name, blank=True,
null=True, verbose_name=_("Image"))
about = tinymce_models.HTMLField(verbose_name=_("Enter the
description of the food"),blank=False, null=False)
rating = models.DecimalField(max_digits=4, decimal_places=2,
default=0)
views = models.PositiveIntegerField(default=0)
tags = TagField()
published = models.BooleanField(default=False)
visible = models.BooleanField(default=True)

def __unicode__(self):
return self.name

Bill Freeman

unread,
Apr 15, 2010, 3:13:01 PM4/15/10
to django...@googlegroups.com
Got it. It's an inclusion tag with templage change_list_results.html
. I don't see
anything suspicious in it. The only things that get rendered are
rendered simply,
as just a template variable reference, except for "header.txt" which is filtered
through capfirst. I'm unaware of any of those needing to convert anything to
ascii.

Alan, can you expand the local vars in a few interesting stack frames, like the
bottom several, and maybe the one in options.py further up, and send just the
stack trace part again?

Bill

zayatzz

unread,
Apr 15, 2010, 3:22:27 PM4/15/10
to Django users

On Apr 15, 10:13 pm, Bill Freeman <ke1g...@gmail.com> wrote:
> Got it.  It's an inclusion tag with templage change_list_results.html
> .  I don't see
> anything suspicious in it.  The only things that get rendered are
> rendered simply,
> as just a template variable reference, except for "header.txt" which is filtered
> through capfirst.  I'm unaware of any of those needing to convert anything to
> ascii.
>
> Alan, can you expand the local vars in a few interesting stack frames, like the
> bottom several, and maybe the one in options.py further up, and send just the
> stack trace part again?

Sure. Thats alot of stuff though:
http://pastebin.org/152583

Thanks!

alan.


>
> Bill
>
>
>
> On Thu, Apr 15, 2010 at 2:54 PM, Daniel Roseman <dan...@roseman.org.uk> wrote:
> > On Apr 15, 7:52 pm, Bill Freeman <ke1g...@gmail.com> wrote:

> >> I don't find it onhttp://docs.djangoproject.com/en/1.1/ref/templates/builtins/.

Bill Freeman

unread,
Apr 15, 2010, 3:48:51 PM4/15/10
to django...@googlegroups.com
Sadly, the problem string doesn't occur at the top level of any of those local
vars. It was worth a shot. It's probably in the context.

If I were you, I'd go find the raise wrapped... in debug.py down at the bottom
of the stack trace, and stick a pdb.set_trace() there. Then (assuming you
do this under manage.py runserver) you can poke around to see what was
being rendered and what's in the context. The exception being wrapped may
have useful information that isn't showing in the stacktrace. If you can find
the actuall line of code that is getting the exception, you may be
able to figure
out why it thinks it has to convert something to ascii (which is probably the
default string encoding, use sys.getdefaultencoding() to find out). Probably
something is applying str() to a unicode object.

I'm sorry that I can't provide a shortcut. Perhaps someone else will.

Bill

Karen Tracey

unread,
Apr 15, 2010, 3:52:56 PM4/15/10
to django...@googlegroups.com
On Thu, Apr 15, 2010 at 3:48 PM, Bill Freeman <ke1...@gmail.com> wrote:
Sadly, the problem string doesn't occur at the top level of any of those local
vars.  It was worth a shot.  It's probably in the context.

If I were you, I'd go find the raise wrapped... in debug.py down at the bottom
of the stack trace, and stick a pdb.set_trace() there.  Then (assuming you
do this under manage.py runserver) you can poke around to see what was
being rendered and what's in the context.  The exception being wrapped may
have useful information that isn't showing in the stacktrace.  If you can find
the actuall line of code that is getting the exception, you may be
able to figure
out why it thinks it has to convert something to ascii (which is probably the
default string encoding, use sys.getdefaultencoding() to find out).  Probably
something is applying str() to a unicode object.

I'm sorry that I can't provide a shortcut.  Perhaps someone else will.

If there is a way to run the app using Python 2.5 instead of 2.6 then the debug information would include the original traceback. Alternatively if it could be run with a recent checkout of either the 1.1.X branch or trunk then the full traceback would appearing, instead of it getting cut off at "raise wrapped".

Karen

zayatzz

unread,
Apr 16, 2010, 2:29:08 AM4/16/10
to Django users
Okay. I will try to test it in the way suggested by Bill and Karen.

This will take me some time though. Hopefully you will be able to keep
track of this thread even though it will get pushed to later pages
when new threads are created.

Alan

On Apr 15, 10:52 pm, Karen Tracey <kmtra...@gmail.com> wrote:

Bill Freeman

unread,
Apr 16, 2010, 9:37:18 AM4/16/10
to django...@googlegroups.com
I like the idea of getting a full stack trace using python 2.5

I don't know what you platform is, but at least on linux (and similar systems)
you can easily install multiple versions of python without causing trouble*.
You grab the sources, do a make, and then, instead of doing make install,
you do make altinstall. Then "python" still refers to your original
version, but
your additional version is available as, say "python2.5". Then use virtualenv
to make an environment for django tied to python2.5, and do a checkout/clone/
whatever (assuming that your project is under revision control) of your project
into the virtual environment (saves trouble with wrong version .pyc
files). Activate
the virutal environment, pip install django and your requirements, and fire up
manage.py runserver. The whole process shouldn't take more than an hour
if you know what you're doing, or a couple of hours if you need to read a bunch
of documentation.

[ * I've got 2.4 (for zope/plone), 2.5, 2.6, 2.7, 3.0, and 3.1 on my laptop. ]

Bill

zayatzz

unread,
Apr 29, 2010, 3:28:43 PM4/29/10
to Django users
Buh... took me a while to get around to do it. After first installing
the django trunk it revealed another problem first though. Not sure if
this can be the cause or not?

The problem was django tinymce import error. In django tinymce widgets
line 14 there was
from django.forms.util import smart_unicode
But that did not work. I changed it to:
from django.utils.encoding import smart_unicode

After that i got this error:

http://pastebin.com/QaF6HQbg

Alan
> > For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Bill Freeman

unread,
Apr 29, 2010, 6:22:47 PM4/29/10
to django...@googlegroups.com
I don't have time to look in detail right now, but if you're using
pages cms, try
putting:

PAGE_SANITIZE_USER_INPUT = False

in your settings.py (after or in place of anywhere you set it True).
The last time
I got this it was due to a bug in the sanitize stuff in that it
converted perfectly
valid unicode into a UTF-8 bytestring and left that as the value to render, so
when you call unicode on it, the default codec being 'ascii', it blows
up if there
was anything requiring extended coding in UTF-8, such as a nbsp (in my case,
the entity was being converted by sanitize to the unicode, but in UTF-8.

If that turns out to fix your issue, we should do a real fix, which is simply to
decode the return value from sanitize using the 'utf-8' codec, giving us a
unicode string, which the safe_unicode function will be happy with. One of
us should produce a patch to send to the pages cms folks.

Bill

zayatzz

unread,
Apr 30, 2010, 2:50:31 AM4/30/10
to Django users
I dont use any cms. I use few third party plugins like django-tinymce,
sorl-thumbnail, django-tagging and django-tagging-autocomplete and
thats it.

I checked how the stuff is stored in the database and its exactly the
same html it prints out on the pages. Letters like äöõü are all &auml;
&ouml; &otilde; &uuml; in the database. No idea if this is relevant
info to you... :P

Alan

Bill Freeman

unread,
Apr 30, 2010, 11:54:13 AM4/30/10
to django...@googlegroups.com
Just to be clear, the entity coding exists in the database? It's a unicode
or byte string containing sequences beginning with ampersand, ending
with semi colon, having just lowercase English letters between them?

Because, if so, by the time it gets to items_for_result() (I think) in
/usr/local/lib/python2.6/dist-packages/django/contrib/admin/templatetags/admin_list.py
those entities have been converted to UTF-8 encodings of Unicode code points.
If my decoding is correct, that sequence "\xc3\xb5" translates to 0x00F5, which
is "latin small letter o with tilde".

If you're comfortable driving pdb, I'd like to know what you find if you place a
conditional breakpoint in that function and poke around. I presume that you
know the attribute name of the field containing that data. I'll use
'foo' below,
replace it with the actual attribute name. Between these two lines:

if field_val is not None:
result_repr = escape(getattr(result, f.name))

(lines 180 and 181 in the version of the file I have handy) put:

if f.attname == 'foo':
import pdb; pdb.set_trace()

You are, I'm assuming, using manage.py runserver. It will restart when you
save the file, or if not, kill it and restart it. Then refresh the page in the
browser. That should make pdb come up on the terminal where you started
the server. Then try the following command:

p getattr(result, f.name)

Do you get the entities? If so, we need to look down in escape. If you get
the UTF-8 encodings, then we need to look higher up the stack.

result should be one of your Vote object instances. You might do a by hand
query for this on in manage.py shell (using id__exact as a selector, perhaps)
and see if the value is bad there. If bad, the problem looks to be in the ORM.

I can't tell more from the stacktrace, since the relevant variable values are
objects, and the strings are inside them and not displayed in the
frame varables.

And, of course, I can't easily reproduce this, so you may be on your own to
debug this.

Bill

zayatzz

unread,
May 1, 2010, 2:51:37 AM5/1/10
to Django users
Hi

On Apr 30, 6:54 pm, Bill Freeman <ke1g...@gmail.com> wrote:
> Just to be clear, the entity coding exists in the database?  It's a unicode
> or byte string containing sequences beginning with ampersand, ending
> with semi colon, having just lowercase English letters between them?

Actually my mistake here. The stuff that went into database via
tinymce was with &otilde;'s and so on.

The Grill P\xc3\xb5rsas Meremeeste moodi from pastebin page is name of
one Retsept and is in database like this:Grill Põrsas Meremeeste
moodi.

Retsept model was this:
lass Retsept(models.Model):
name = models.CharField(max_length=100,
verbose_name=_("Name"), blank=False, null=False)
user = models.ForeignKey(User)
image = RemovableImageField(upload_to=content_file_name,
blank=True, null=True, verbose_name=_("Image"))
about = tinymce_models.HTMLField(verbose_name=_("Enter the
description of the food"),blank=False, null=False)
rating = models.DecimalField(max_digits=4, decimal_places=2,
default=0)
views = models.PositiveIntegerField(default=0)
tags = TagField()
published = models.BooleanField(default=False)
visible = models.BooleanField(default=True)
def __unicode__(self):
return self.name

and form, that saves it was this:
class RetseptForm(ModelForm):
about = forms.CharField(widget=TinyMCE(attrs={'style': 'width:
370px', 'rows': 15, 'theme':"simple"}), label = _("text here"),
required=False)
tags = TagField(label = _("tags"),
widget=TagAutocomplete(attrs={'class':'required'}))
name = forms.CharField(label = _("Name"),
widget=forms.TextInput(attrs={'class':'required'}))
class Meta:
model = Retsept
fields = ('name', 'about', 'image', 'tags',)
widgets = {
'name': TextInput(attrs={'class':
'required' })
}
>
> Because, if so, by the time it gets to items_for_result() (I think) in
> /usr/local/lib/python2.6/dist-packages/django/contrib/admin/templatetags/ad min_list.py
> those entities have been converted to UTF-8 encodings of Unicode code points.
> If my decoding is correct, that sequence "\xc3\xb5" translates to 0x00F5, which
> is "latin small letter o with tilde".

Yep, you were correct.

>
> If you're comfortable driving pdb,

Nope, i guess, since i know what pdb stands for :P I ran a google
search and now i know what you were talking about

> I'd like to know what you find if you place a
> conditional breakpoint in that function and poke around.  I presume that you
> know the attribute name of the field containing that data.  I'll use
> 'foo' below,
> replace it with the actual attribute name.   Between these two lines:
>
>                 if field_val is not None:
>                     result_repr = escape(getattr(result, f.name))
>
> (lines 180 and 181 in the version of the file I have handy) put:
>
>                     if f.attname == 'foo':
>                         import pdb; pdb.set_trace()

Im having trouble figuring out where i should put this pdb stuff. I
downloaded last trunk of django 1.2beta and my admin_list.py is
somewhat different than the one you have. lines 180, 181 there are :

if form:
yield mark_safe(u'<td>%s</td>' %
force_unicode(form[cl.model._meta.pk.name]))

I can find the lines you mention in the old django, but stuff in new
admin_list.py is :
else:
if value is None:
result_repr = EMPTY_CHANGELIST_VALUE
if isinstance(f.rel, models.ManyToOneRel):
result_repr = escape(getattr(result, f.name))
else:
result_repr = display_for_field(value, f)
if isinstance(f, models.DateField) or isinstance(f,
models.TimeField):
row_class = ' class="nowrap"'

So i guess i should enter your lines after if isinstance(f.rel,
models.ManyToOneRel): ?

>
> You are, I'm assuming, using manage.py runserver.  

Nope, im using mod_wsgi, but in order to test what you suggest i could
use it.

> It will restart when you
> save the file, or if not, kill it and restart it.  Then refresh the page in the
> browser.  That should make pdb come up on the terminal where you started
> the server.  Then try the following command:
>
> p getattr(result, f.name)
>
> Do you get the entities?  If so, we need to look down in escape.  If you get
> the UTF-8 encodings, then we need to look higher up the stack.
>
> result should be one of your Vote object instances.  You might do a by hand
> query for this on in manage.py shell (using id__exact as a selector, perhaps)
> and see if the value is bad there.  If bad, the problem looks to be in the ORM.
>
> I can't tell more from the stacktrace, since the relevant variable values are
> objects, and the strings are inside them and not displayed in the
> frame varables.
>
> And, of course, I can't easily reproduce this, so you may be on your own to
> debug this.
>
> Bill

zayatzz

unread,
May 1, 2010, 3:21:01 PM5/1/10
to Django users
Well i put this if and following import & trace to several places, but
all i got was the 500 error and pdb prompt did not show up.

If you could provide me with exact info of where to put this trace in
djang 1.2. beta 1 admin_list.py... then it would be really nice :P

Alan

Bill Freeman

unread,
May 2, 2010, 5:31:57 PM5/2/10
to django...@googlegroups.com
As far as breakpoints go, realize that you can only use them from
the development server ("python manage.py runserver"), and that
pdb shows up in the terminal where you ran that command, and
it must still be showing.

But since the data isn't in the expected format in the database, the
particular breakpoints I was talking about before aren't relevant.

Instead, in a terminal, cd'ed to your project directory (directory
containing settings.py and manage.py), and fixing my guess at
the name of the module containing Retsept, and using a query
field lookup that will get you one of these objects instead of my
"id__exact=1", type:

python manage.py shell
from retsepts.models import Retsept
rs = Retsept.objects.get(id__exact=1)
repr(rs.about)

Does the last command print a unicode string (u before the
opening quote) or a byte string with those '\x' excapes in it?
If unicode, then something is broken in the render code and
will require further nasty debugging. If, on the other hand,
you get a byte string, then the problem is in the HTMLField
code, which I could believe depends on behavior that has
chaned in 1.2 (tiny not being a core product) and a work
around could be to subclass the field, making it decode
the UTF-8 into unicode, something like:

class MyHTMLField(tinymce_models.HTMLField):
def to_python(self, value):
v = super(MyHTMLField, self).to_python(value)
return v.decode('utf8')

By the way, what database are you using, and how is it
configured to store strings?

Bill
Reply all
Reply to author
Forward
0 new messages