[Bug 299950] Re: UnicodeDecodeError on sending content and visualizing "Statistics" page

[Andrea]

** Attachment added: "Error log"
http://launchpadlibrarian.net/19789044/error.log

--
UnicodeDecodeError on sending content and visualizing "Statistics" page
https://bugs.launchpad.net/bugs/299950
You received this bug notification because you are a member of Singing &
Dancing developers, which is the registrant for Singing & Dancing.

Status in Singing & Dancing: New

Bug description:
Hi,

I tried to send a page content from my site and got an error. After that error, whenever I go to the Statistics page I get the error attached here. Maybe the problem is caused by accented characters?

Thanks for your time,
Andrea.

[Andrea]

Public bug reported:

Hi,

I tried to send a page content from my site and got an error. After that
error, whenever I go to the Statistics page I get the error attached
here. Maybe the problem is caused by accented characters?

Thanks for your time,
Andrea.

** Affects: singing-dancing
Importance: Undecided
Status: New

[Daniel Nouri]

Andrea,

can you try and change your user interface language? The error log
looks like it's failing while it tries to translate parts of the user
interface. Is there a buggy PO file involved, maybe?

[Andrea]

Dear Daniel,

thank you for your reply.

I tried to change laguage but the problem seems to be linked to the
content of a particular page in my site. You can see the page contents
below.

I attached also the error full trace.

If I can do other tests to help debugging this problem, please let me
know.

Thank you very much for your help.

Page Contents:

title: Attività del Focal Point Nazionale IPCC
description: Le principali attività del Focal Point Nazionale IPCC

<p>Le principali attività del Focal Point Nazionale IPCC includono:</p>
<ul><li>la raccolta di informazioni e la documentazione delle attività tecnico-scientifiche nazionali inerenti le tematiche trattate dall’IPCC: scienza del clima e cambiamenti climatici (osservazioni, modelli, studi di vulnerabilità, stima di impatti, misure di adattamento e mitigazione);&nbsp;</li><li>la divulgazione dell’attività e dei risultati dell’IPCC a tutti i livelli sul territorio nazionale tramite sito-web, riunioni, convegni e workshop dedicati;</li><li>l’individuazione e designazione degli esperti o i rappresentanti nazionali alle riunioni, workshop ed alle sessioni plenarie dell' IPCC e dei relativi gruppi di lavoro;</li><li>il riscontro all’IPCC sulle attività di rilievo realizzate sul territorio nazionale.</li></ul>
<p>L’attività di Focal Point IPCC per l’Italia è svolta dal <a title="Contatti" class="internal-link" href="contatti">Dr. Sergio Castellari</a> presso il Centro Euro-Mediterraneo per i Cambiamenti Climatici (CMCC) a partire dal 1 Dicembre 2006. <br />La documentazione precedente a tale data è reperibile sul sito a cura del precedente responsabile Focal Point IPCC Italia, Dr. Vincenzo Ferrara presso <a class="external-link" href="http://clima.casaccia.enea.it/ipcc/focalpoint/">ENEA-Casaccia</a>.</p>
<p>&nbsp;</p>

** Attachment added: "singing-dancing.log"
http://launchpadlibrarian.net/19831743/singing-dancing.log

[Andrea]

Hi,

I've got a little time to investigate further and now I can be more
precise about the bug I'm experiencing.

It seems that the problem is in the accented letter in the title of
the Plone document I pasted in the previous message.

In fact, just for try, I commented this line in page template
`collective.dancing-0.8.2-py2.4.egg/collective/dancing/browser/jobs.pt':

<td tal:content="job/title">Send "My news item"</td>

and the exception when I visit the statistics page isn't raised
anymore.

I'm not able to find the right place in the code to fix this, but I
hope this information is of some help.

Thank you and best regards,
Andrea.

[Sivan Greenberg]

I can confirm commenting <td tal:content="job/title">Send "My news
item"</td> fixes the problem

[Sivan Greenberg]

I'm experiencing the same bug, this time after queuing a hebrew page to
be sent from the statistics page, this is the traceback:

2008-12-02 15:20:11 ERROR Zope.SiteErrorLog http://zeta:8080/cancer/portal_newsletters/channels/newsletter-stats.html
Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module plone.z3cform.layout, line 49, in __call__
Module Shared.DC.Scripts.Bindings, line 313, in __call__
Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
Module Products.PageTemplates.PageTemplate, line 89, in pt_render
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 891, in do_useMacro
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 891, in do_useMacro
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 949, in do_defineSlot
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 957, in do_defineSlot
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 891, in do_useMacro
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 957, in do_defineSlot
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 949, in do_defineSlot
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 949, in do_defineSlot
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 586, in do_setLocal_tal
Module zope.tales.tales, line 696, in evaluate
- URL: controlpanel
- Line 10, Column 4
- Expression: <PathExpr standard:'view/contents'>
- Names:
{'container': <ChannelContainer at /cancer/portal_newsletters/channels>,
'context': <ChannelContainer at /cancer/portal_newsletters/channels>,
'default': <object object at 0xb7c30580>,
'here': <ChannelContainer at /cancer/portal_newsletters/channels>,
'loop': {},
'nothing': None,
'options': {'args': ()},
'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0xe52968c>,
'request': <HTTPRequest, URL=http://zeta:8080/cancer/portal_newsletters/channels/newsletter-stats.html>,
'root': <Application at >,
'template': <ImplicitAcquirerWrapper object at 0xe52922c>,
'traverse_subpath': [],
'user': <PropertiedUser 'admin'>,
'view': <Products.Five.metaclass.MyFormWrapper object at 0xe5299cc>,
'views': <zope.app.pagetemplate.viewpagetemplatefile.ViewMapper object at 0xe52984c>}
Module zope.tales.expressions, line 217, in __call__
Module Products.PageTemplates.Expressions, line 161, in _eval
Module Products.PageTemplates.Expressions, line 123, in render
Module plone.z3cform.layout, line 58, in contents
Module plone.z3cform.layout, line 66, in render_form
Module z3c.form.form, line 190, in __call__
Module z3c.form.form, line 143, in render
Module zope.app.pagetemplate.viewpagetemplatefile, line 83, in __call__
Module zope.app.pagetemplate.viewpagetemplatefile, line 51, in __call__
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 824, in do_loop_tal
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 745, in do_insertStructure_tal
Module zope.tales.tales, line 696, in evaluate
- URL: /media/sdb1/change-test/trunk/eggs/plone.z3cform-0.5.2-py2.4.egg/plone/z3cform/crud/crud-master.pt
- Line 17, Column 2
- Expression: <PathExpr standard:u'form/render'>
- Names:
{'args': (),
'context': <ChannelContainer at /cancer/portal_newsletters/channels>,
'default': <object object at 0xb7c30580>,
'loop': {},
'nothing': None,
'options': {},
'repeat': {},
'request': <HTTPRequest, URL=http://zeta:8080/cancer/portal_newsletters/channels/newsletter-stats.html>,
'template': <zope.app.pagetemplate.viewpagetemplatefile.ViewPageTemplateFile object at 0xaed86cc>,
'usage': <zope.pagetemplate.pagetemplate.TemplateUsage object at 0xe4f1a4c>,
'view': <collective.dancing.browser.stats.StatsForm object at 0xe529fec>,
'views': <zope.app.pagetemplate.viewpagetemplatefile.ViewMapper object at 0xe4f1ecc>}
Module zope.tales.expressions, line 217, in __call__
Module zope.tales.expressions, line 211, in _eval
Module collective.dancing.browser.stats, line 84, in render
Module zope.app.pagetemplate.viewpagetemplatefile, line 83, in __call__
Module zope.app.pagetemplate.viewpagetemplatefile, line 51, in __call__
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 855, in do_condition
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 536, in do_optTag_tal
Module zope.tal.talinterpreter, line 521, in do_optTag
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 824, in do_loop_tal
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 631, in do_insertText_tal
Module zope.tal.talinterpreter, line 841, in translate
Module zope.app.pagetemplate.engine, line 109, in translate
Module zope.i18n, line 128, in translate
Module zope.i18n.translationdomain, line 79, in translate
Module zope.i18n.translationdomain, line 142, in _recursive_translate
Module zope.i18n, line 166, in interpolate
Module zope.i18n, line 161, in replace
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

[Sivan Greenberg]

I've done some debugging of my own, which reveals that the exception for me occurs in:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

> /media/sdb1/debug-singing/collective.dancing/trunk-buildout/eggs/zope.i18n-3.6.0-py2.4.egg/zope/i18n/__init__.py(161)replace()
-> return unicode(mapping.get(param1 or param2, whole))

the piece of code is:

(Pdb) 156 u'This is ${name}'
157 """
158
159 def replace(match):
160 whole, param1, param2 = match.groups()
161 -> return unicode(mapping.get(param1 or param2, whole))
162
163 if not text or not mapping:
164 return text
165
166 return _interp_regex.sub(replace, text)

and I can confirm this happens only when the title is set with hebrew
text

[tmog]

The UnicodeDecodeError on "Statistics" is indeed the result of a bad
message in job.title when the content being sent has a non-ascii title.

I've committed a fix in r76770.

[Andrea Russo]

Hi,

I can confirm that the fix is working here.

Thank you very much for your help.

Regards,
Andrea.

[Daniel Widerin]

thanks tmog,
this is part of collective.dancing 0.8.3 and collective.singing 0.6.7

** Changed in: singing-dancing
Status: New => Fix Released

--
UnicodeDecodeError on sending content and visualizing "Statistics" page
https://bugs.launchpad.net/bugs/299950
You received this bug notification because you are a member of Singing &
Dancing developers, which is the registrant for Singing & Dancing.

Status in Singing & Dancing: Fix Released