UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1145: ordinal not in range(128)

[Annet]

After upgrading to web2py version 2.10.3 I get an error on fields
of type text when they contain accented latin characters, see
traceback.


Traceback (most recent call last):
  File "/Users/iannet/web2py_local/gluon/restricted.py", line 227, in restricted
    exec ccode in environment
  File "/Users/iannet/web2py_local/applications/my/controllers/landingpage.py", line 131, in <module>
  File "/Users/iannet/web2py_local/gluon/globals.py", line 393, in <lambda>
    self._caller = lambda f: f()
  File "/Users/iannet/web2py_local/gluon/tools.py", line 3440, in f
    return action(*a, **b)
  File "/Users/iannet/web2py_local/gluon/tools.py", line 3440, in f
    return action(*a, **b)
  File "/Users/iannet/web2py_local/applications/my/controllers/landingpage.py", line 116, in landingpage_form
    if form.process().accepted:
  File "/Users/iannet/web2py_local/gluon/html.py", line 2301, in process
    self.validate(**kwargs)
  File "/Users/iannet/web2py_local/gluon/html.py", line 2238, in validate
    if self.accepts(**kwargs):
  File "/Users/iannet/web2py_local/gluon/sqlhtml.py", line 1677, in accepts
    self.id_field_name]).update(**fields)
  File "/Users/iannet/web2py_local/gluon/packages/dal/pydal/objects.py", line 2117, in update
    ret = db._adapter.update("%s" % table._tablename,self.query,fields)
  File "/Users/iannet/web2py_local/gluon/packages/dal/pydal/adapters/base.py", line 988, in update
    raise e
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1145: ordinal not in range(128)


Why isn't my application backward compatible?

Kind regards,

Annet

[Willoughby]

Postgre?

https://groups.google.com/d/topic/web2py/LJwG9ec4o28/discussion

[Annet]

Hi,

Thanks for you reply, I read the post before posting my question,
I also read this post: https://github.com/web2py/web2py/issues/910

My problem is that I wasn't having this issue in:

Version 2.9.11-stable+timestamp.2014.09.15.23.35.11
Database drivers available: SQLite(sqlite3), MySQL(pymysql), PostgreSQL(pg8000),
MSSQL(pyodbc), DB2(pyodbc), Teradata(pyodbc), Ingres(pyodbc), IMAP(imaplib)

which as far as I can see also only has pg8000 available not psycopg2

Created by Massimo Di Pierro, Copyright 2007-2015
Version 2.10.3-stable+timestamp.2015.04.02.21.42.07
Database drivers available: sqlite3, imaplib, pyodbc, pymysql, pg8000

I prefer to keep my web2py installation as close to when its first unzipped,
this makes in easier to host it.


Kind regards,

Annet

[Willoughby]

Hmm...well, both your error and the issue you mention reference pydal, which is now a separate project and I believe has to be 'upgraded' separately as well.

Did you upgrade PyDAL?

PyDAL--The DAL is now an independent project

[Annet]

Hmm...well, both your error and the issue you mention reference pydal, which is now a separate project and I believe has to be 'upgraded' separately as well.

Did you upgrade PyDAL?

No, I just downloaded the latest version of the source code for normal users,
unzipped it, and created a new simple application.

Further more, I used appconfig.init to configure db, smpt and form styling, and
reused code from a previous application in db.py, a controller and some views.

I am not using GitHub, I am just using Web2py to develop a web application.

Should I downgrade to version 2.9.11 to solve this issue?


Kind regards,

Annet

[Willoughby]

The latest source is 2.10.4 so you could try that or if that fails, use 2.9.11.

[Antonio Salazar]

I have the same problem. PostgreSQL database, web2py 2.11.2-stable+timestamp.2015.05.30.16.33.24 source install.

Now any field update which has non-ASCII characters fails with "<type 'exceptions.UnicodeDecodeError'> 'ascii' codec can't decode byte 0xc3 in position X: ordinal not in range(128)"
By any field update, I mean custom forms, form grids, and even database administration.

This is the traceback after failing to change a user's name to "José" with database administration:

Traceback (most recent call last

):
  File "S:\Extranet\Website\Fuentes\web2py\gluon\restricted.py", line 227, in restricted
    exec ccode in environment
  File "S:/Extranet/Website/Fuentes/web2py/applications/extranet/controllers/appadmin.py", line 704, in <module>
  File "S:\Extranet\Website\Fuentes\web2py\gluon\globals.py", line 412, in <lambda>
    

self._caller = lambda f: f

()
  File "S:/Extranet/Website/Fuentes/web2py/applications/extranet/controllers/appadmin.py", line 343, in update
    if form.accepts(request.vars, session):
  File "S:\Extranet\Website\Fuentes\web2py\gluon\sqlhtml.py", line 1686, in accepts
    self.id_field_name]).update(**fields)
  File "S:\Extranet\Website\Fuentes\web2py\gluon\packages\dal\pydal\objects.py", line 2020, in update

    ret = db._adapter.update("%s" % table._tablename,self.query,fields

)
  File "S:\Extranet\Website\Fuentes\web2py\gluon\packages\dal\pydal\adapters\base.py", line 996, in update
    raise e
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 57: ordinal not in range(128)

I don't recall having this problem with pre 2.10 versions

[Massimo Di Pierro]

It is a bug in pg8000. You need to install psycopg2

[Paolo Valleri]

According to my tests, the current PG adapter works only with psycopg2 and python2.7

It is broken with both pg8000 and psycopg2 on python 3.x

A possible fix for pg8000 and python2.7 is here https://github.com/ilvalle/pydal/commit/d5450938ae606a3e85deffa831899a35146f5e64

I've posted an issue for psycopg2 with python3.x here https://github.com/psycopg/psycopg2/issues/331#issuecomment-111909256

Paolo

[Antonio Salazar]

Is it still recommended to use psycopg2 instead of pg8000 after this fix?

[Paolo Valleri]

In general the best choice is psycopg2 however, the issue with pg8000 has been fixed

I've written an initial basic unittest https://github.com/web2py/pydal/blob/master/tests/base.py#L72 to check possible issues with unicode

 Paolo

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/AE6NVmLPCwk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Massimo Di Pierro]

I think we should remove pg8000 unless this unicode problem is fixed. It just causes problem to new pgsql users. psycopg2 is solid.

 Paolo

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

[黄祥]

+1

[Annet]

+1

[Paolo Valleri]

Massimo the mentioned issue has been fixed in pydal. Have you found other different issues?

Paolo

 Paolo

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

[Massimo Di Pierro]

Thanks for letting me know. I did not try that yet.

[Ben Lawrence]

Hi Paolo,  I get the error "<type 'exceptions.UnicodeDecodeError'> 'ascii' codec can't decode" using the latest web2py download (2.10.4-stable+timestamp.2015.04.26.15.11.54 (Running on Apache/2.4.7 (Ubuntu), Python 2.7.6)). This occurs even with “” characters in the on-board wiki.

Could you tell me the steps of the easiest way to fix this? (I don't mind which i use pg8000 or psycopg2) Is it a simple matter of 

sudo apt-get install python-psycopg2

?

[Paolo Valleri]

Hi Ben, the latest web2py download doesn't include the fix yet. You have to either manually update pydal or wait for a newer release of web2py

 Paolo

[Ben Lawrence]

Or just
sudo apt-get install python-psycopg2

as Massimo said, web2py will use psycopg2 before looking for pg8000    :-)    great!!