Crazy error with utf-8 & apache (please help!)

[elm]

I have bee trying to deal with an erratic error for the last 3 day. I
hope you can rescue me!!!

After experiencing erratic errors that appeared from time to time on a
larger project (i.e. ViewDoesNotExist) I have reduced my project to a
very simple application that runs on apache with mod_python and
connects to Oracle via cx_oracle (4.3).

after reloading apache I get the following error:

----
TemplateSyntaxError at /test/

Caught an exception while rendering: 'utf8' codec can't decode bytes
in position 3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)
...
----
After refreshing the browser page a few times the error disappears the
the template displays correctly (a list of words, some of which are in
spanish). After that the error reappears form time to time.

I guess that the database is sending the data in a non UTF8 format
when the error appears, but I can not understand:

- Why it happens sometimes and not always?
- If it is a problem with oracle (or cx_oracle) why is it manifested
afer reloading apache?

PLEASE help me with any idea or suggestion!!

I am using
Redhat ES release 4 U7
Apache 2.0.52-41.ent.2
python 2.3
django1.0
mod_python 3.2.1
cx_oracle 4.4

These are my basic models/views/templates:

prueba1.models.py
-------------------------
from django.db import models

class PeriodosAcademicos(models.Model):
peri_ccod= models.IntegerField(db_column='PERI_CCOD',
primary_key=True)
peri_tdesc = models.CharField(max_length=100,
db_column='PERI_TDESC')
class Meta:
db_table = 'PERIODOS_ACADEMICOS'

prueba1.views.py
-----------------------
from prueba1.models import PeriodosAcademicos
from django.shortcuts import render_to_response

def test(request):
periods = PeriodosAcademicos.objects.all()
return render_to_response('test.html',{'periods':periods})

prueba1/templates/test.html
-------------------------------------
<h1>Test</h1>
{% for period in periods %}
<li>{{period.peri_tdesc}}
{% endfor %}

This is the full error transcript:
TemplateSyntaxError at /test/

Caught an exception while rendering: 'utf8' codec can't decode bytes
in position 3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)

Original Traceback (most recent call last):
File "/usr/lib/python2.3/site-packages/django/template/debug.py",
line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.3/site-packages/django/template/
defaulttags.py", line 123, in render
len_values = len(values)
File "/usr/lib/python2.3/site-packages/django/db/models/query.py",
line 154, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.3/site-packages/django/db/models/query.py",
line 269, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.3/site-packages/django/db/models/sql/
query.py", line 206, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.3/site-packages/django/db/models/sql/
query.py", line 1714, in <lambda>
result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
File "/usr/lib/python2.3/site-packages/django/db/backends/oracle/
base.py", line 400, in fetchmany
return tuple([tuple([to_unicode(e) for e in r]) for r in
Database.Cursor.fetchmany(self, size)])
File "/usr/lib/python2.3/site-packages/django/db/backends/oracle/
base.py", line 411, in to_unicode
return force_unicode(s)
File "/usr/lib/python2.3/site-packages/django/utils/encoding.py",
line 70, in force_unicode
raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'utf8' codec can't decode bytes in position
3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)

Request Method: GET
Request URL: http://172.16.110.42/test/
Exception Type: TemplateSyntaxError
Exception Value:

Caught an exception while rendering: 'utf8' codec can't decode bytes
in position 3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)

Original Traceback (most recent call last):
File "/usr/lib/python2.3/site-packages/django/template/debug.py",
line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.3/site-packages/django/template/
defaulttags.py", line 123, in render
len_values = len(values)
File "/usr/lib/python2.3/site-packages/django/db/models/query.py",
line 154, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.3/site-packages/django/db/models/query.py",
line 269, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.3/site-packages/django/db/models/sql/
query.py", line 206, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.3/site-packages/django/db/models/sql/
query.py", line 1714, in <lambda>
result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
File "/usr/lib/python2.3/site-packages/django/db/backends/oracle/
base.py", line 400, in fetchmany
return tuple([tuple([to_unicode(e) for e in r]) for r in
Database.Cursor.fetchmany(self, size)])
File "/usr/lib/python2.3/site-packages/django/db/backends/oracle/
base.py", line 411, in to_unicode
return force_unicode(s)
File "/usr/lib/python2.3/site-packages/django/utils/encoding.py",
line 70, in force_unicode
raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'utf8' codec can't decode bytes in position
3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)

Exception Location: /usr/lib/python2.3/site-packages/django/template/
debug.py in render_node, line 81
Python Executable: /usr/bin/python
Python Version: 2.3.4
Python Path: ['/var/www/html/tide', '/var/www/html/tide/
reportes_test', '/usr/lib/python23.zip', '/usr/lib/python2.3', '/usr/
lib/python2.3/plat-linux2', '/usr/lib/python2.3/lib-tk', '/usr/lib/
python2.3/lib-dynload', '/usr/lib/python2.3/site-packages']
Server time: Wed, 18 Mar 2009 15:43:26 -0500

---------------
When it works, the page shows:
<h1>Test</h1>

<li>OTOÑO

<li>PRIMAVERA

<li>PRIMAVERA

<li>INVIERNO
...

[Ian Kelly]

On Mar 19, 4:55 pm, elm <ernesto.la...@gmail.com> wrote:
> I have bee trying to deal with an erratic error for the last 3 day. I
> hope you can rescue me!!!
>
> After experiencing erratic errors that appeared from time to time on a
> larger project (i.e. ViewDoesNotExist) I have reduced my project to a
> very simple application that runs on apache with mod_python and
> connects to Oracle via cx_oracle (4.3).
>
> after reloading apache I get the following error:
>
> ----
> TemplateSyntaxError at /test/
>
> Caught an exception while rendering: 'utf8' codec can't decode bytes
> in position 3-4: invalid data. You passed in 'OTO\xd1O' (<type 'str'>)

Indeed, that's the latin1 encoding of 'OTOÑO', not utf8. Maybe Oracle
is getting somehow getting the client encoding confused. Try setting
the environment variable NLS_LANG = '.UTF8' for the Apache process.

HTH,
Ian

[elm]

Thanks for the hint,

>
> Indeed, that's the latin1 encoding of 'OTOÑO', notutf8. Maybe Oracle

> is getting somehow getting the client encoding confused. Try setting
> the environment variable NLS_LANG = '.UTF8' for the Apache process.

I will explore that path. It clearly has some relation with the
latin1/utf8 encoding.

But what puzzels me is the inconsistency (sometimes the data arrives
in latin1 an sothers in utf8).

Any idea waht could be causing this erractic behaviour???

Whoy would different apache processes get different encondings form
the database???

THX

Ernesto

[Graham Dumpleton]

Are you sharing the web server with web applications written using
something else, such as PHP? If PHP is playing with the language
settings, you might be hitting an ordering problem. That is, it works
if any PHP code gets executed first in a specific process and doesn't
otherwise, or vice versa.

You also need to be careful about stopping and starting Apache using
sudo command without the -H option. Leave out the -H option and
environment from your personal account leaks into that of Apache. So,
things may work if you use sudo without -H, but then if system reboots
and Apache started at boot time, then it lacks that same environment
that you as a user have and which was leaking into Apache environment.
Thus things can then break.

Graham

[elm]

> Are you sharing the web server with web applications written using
> something else, such as PHP? If PHP is playing with the language
> settings, you might be hitting an ordering problem. That is, it works
> if any PHP code gets executed first in a specific process and doesn't
> otherwise, or vice versa.
>

Thanks Graham ... the system indeed has web applications in PHP. And
this could be a source of the conflict.

To be honest ... I am a little bit lost with the coding configuration
for receiving data from the DB. Where should the coding be
configured?

I assume that my Django settings (and the DB configuration) should
have the controls ... but is it also dependant on the apache, php,
mod_python configuration?

If php is messing around with the coding (and it is beyond my control)
how can I force UTF8 in my django applications?

I would appreciate further advice fellows!!

Ernesto

[Ian Kelly]

As far as I know, the NLS_LANG environment variable is the only way to
specify the client encoding in cx_Oracle. If that can't be relied
upon within your Apache process, then I think you would need to run
Django either as a CGI or under a separate Apache process.

Another possibility is to upgrade your cx_Oracle to version 5.0.1.
Then cx_Oracle will return strings as unicode to begin with, and the
particular problem you're seeing should go away. But it's only half a
solution, because strings passed in to cx_Oracle will still be utf8-
encoded and might still be mistaken for latin1. The fix to *that*
would involve compiling cx_Oracle 5.0.1 with the WITH_UNICODE option,
and then fixing up the backend so that it passes in unicode. I don't
think that's a major task, but it may require some knowledge of Django
internals.

Regards,
Ian

[Ian Kelly]

On Mar 20, 11:16 am, Ian Kelly <ian.g.ke...@gmail.com> wrote:
> As far as I know, the NLS_LANG environment variable is the only way to
> specify the client encoding in cx_Oracle.  If that can't be relied
> upon within your Apache process, then I think you would need to run
> Django either as a CGI or under a separate Apache process.

One other thing that didn't occur to me before. Are you already using
the prefork Apache MPM? If not, you should switch to it if possible,
and it may be sufficient to prevent the problem from occurring.

HTH,
Ian

[elm]

Thanxs for the advice Ian!!!

But unfortunately I haven't been succesful.

I have set the NLS_LANG to .UTF8 in the <Location> directive at the
apache conf file:

<Location "/tide/reportes_edd">
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE reportes_edd.settings
PythonOption django.root /tide/reportes_edd
PythonDebug On
PythonPath "['/var/www/html/tide', '/var/www/html/tide/
reportes_edd'] + sys.path"
PythonInterpreter reportes_edd
SetEnv NLS_LANG .UTF8
</Location>

But I get a strange behaviour: If I show the os.environ['NLS_LANG']
within my application, it shows '.UTF8' BUT the database still sends
me data encoded in latin1. I have also tried to set os.environ
['NLS_LANG']='.UTF8' in: settings.py, urls.py, myview.py ... and it
seems that the environment variable does not take effect.

I still get the right coding "sometimes" and the wrong one
"othertimes".

I don't have the control of the server and I cannot modify the apache
(honestly I don't know if it has prefork Apache MPM).

The server is a Redhat ES release 4 U7 which comes with python 2.3,
and they are not willing to upgrade it to python 2.5 (since it is not
supported by RedHat ES 4) ... so I can not move to cx_Oracle 5.01. I
am running out of options ...

This is my first project with this client on Django and I really want
to demosntrate that we can have reliable Django applications on their
platforms but my LAST resource would be to move the whole project to
PHP (Which is not what I whant to do ... What else could I try before
that last chance?)

Ernesto

[Ian Kelly]

Sorry, I don't have any other ideas. Have you verified that this only
happens running under apache and not the dev server? You might also
try the cx-oracle-users mailing list; maybe somebody there has
experienced a similar problem.

Ian

[elm]

Thanks ... I will keep trying

My temporary solution is to have my own apache process running on
another port (i.e. 8000) (Here we don't experience the coding
problems)
We have configured the main apache (on port 80) to act as a proxy to
my "private" apache ... and the user experiences a clean result. But
this not a long term solution ... so I will keep trying to identify
how to make the application to work cleanly on the main server.

Ernesto