TypeError: decoding Unicode is not supported

[Ian Wilson]

I have no idea why this doesn't work. I can pull the same data from
the TG console and move into lists and its fine but I can't do it
within a TG method. Does anyone have any ideas?

My uri looks like this:
sqlobject.dburi="mysql://busysignal:********@localhost/busysignal?unix_socket=/var/run/mysqld/mysqld.sock&use_unicode=1&charset=utf8&sqlobject_encoding=utf-8"

Also my my.cnf is littered with utf8s.

It works on my box at home but not on the server and I think my home
box is running x86 and the server is running amd64, both gentoo linux.
I don't know if arch has anything to do with it.

It doesn't matter if I proxy is behind apache or I use it directly.

Any ideas would be very very much appreciated as I have tried many
things and broken many things in the process.

-Ian

Page handler: <function _wrapper at 0x2afd30150668>
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cphttptools.py",
line 105, in _run
self.main()
File "/usr/lib64/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cphttptools.py",
line 254, in main
body = page_handler(*virtual_path, **self.params)
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/identity/conditions.py",
line 275, in _wrapper
return fn( *args, **kw )
File "<string>", line 3, in getPlants
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/controllers.py",
line 326, in expose
output = database.run_with_transaction(
File "<string>", line 5, in run_with_transaction
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/database.py",
line 246, in so_rwt
retval = func(*args, **kw)
File "<string>", line 5, in _expose
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/controllers.py",
line 343, in <lambda>
mapping, fragment, args, kw)))
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/controllers.py",
line 367, in _execute_func
output = errorhandling.try_call(func, *args, **kw)
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/errorhandling.py",
line 71, in try_call
return func(self, *args, **kw)
File "<string>", line 3, in getPlants
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/controllers.py",
line 165, in validate
return errorhandling.run_with_errors(errors, func, *args, **kw)
File "/usr/lib64/python2.4/site-packages/TurboGears-1.0b1-py2.4.egg/turbogears/errorhandling.py",
line 106, in run_with_errors
return func(self, *args, **kw)
File "/home/ian/workspace/busysignal/busysignal/controllers/admin/inventory/plant.py",
line 132, in getPlants
log.debug(str([p.id for p in context[0:limit]]))
File "/usr/lib64/python2.4/site-packages/SQLObject-0.7.2a_r2023-py2.4.egg/sqlobject/sresults.py",
line 155, in __iter__
return iter(list(self.lazyIter()))
File "/usr/lib64/python2.4/site-packages/SQLObject-0.7.2a_r2023-py2.4.egg/sqlobject/sresults.py",
line 163, in lazyIter
return conn.iterSelect(self)
File "/usr/lib64/python2.4/site-packages/SQLObject-0.7.2a_r2023-py2.4.egg/sqlobject/dbconnection.py",
line 776, in iterSelect
select, keepConnection=True)))
File "/usr/lib64/python2.4/site-packages/SQLObject-0.7.2a_r2023-py2.4.egg/sqlobject/dbconnection.py",
line 705, in __init__
self.dbconn._executeRetry(self.rawconn, self.cursor, self.query)
File "/usr/lib64/python2.4/site-packages/SQLObject-0.7.2a_r2023-py2.4.egg/sqlobject/mysql/mysqlconnection.py",
line 77, in _executeRetry
myquery = unicode(query, self.encoding)
TypeError: decoding Unicode is not supported

[Max Ischenko]

Try to append sqlobject_encoding=utf8 to your sqlobject.dburi in
dev.cfg.

Max.

[Ian Wilson]

Thanks but I changed sqlobject_encoding=utf-8 to
sqlobject_encoding=utf8 and I still get the same error. What else
could I try?

-Ian

[Ed Singleton]

I had the same thing and fixed it by using sqlite instead of MySQL.
PostGre doesn't seem to have the problem either.

Ed

[Ian Wilson]

Yeah I was kind of looking to stay with mysql because I use it
everywhere else and I am more familiar with it than PostGre and
sqllite. There has to be something I can do to fix it.

[Kevin Dangoor]

I didn't follow the beginning of this thread, but a quick note that
you should be able to use unicode when you use UnicodeCol objects.
The UTF-8 encoding/decoding is handled in Python then.

Kevin

--
Kevin Dangoor
TurboGears / Zesty News

email: k...@blazingthings.com
company: http://www.BlazingThings.com
blog: http://www.BlueSkyOnMars.com

[Ian Wilson]

Hmm... do you think mixing StringCol and UnicodeCol could cause this?
I never thought about it but maybe I need to upgrade all the columns
to be Unicode instead of having mixed Unicode/String Cols. I'll see
what that does later today.

-Ian

[Claudio Martínez]

Try adding use_unicode=1 to your connection string. If it doesn't help
create a file my.cnf with:
[client]
default-character-set = utf8

And add read_default_file=/path/to/my.cnf to your connection string.

Of course, try using UnicodeCol columns.

[Claudio Martínez]

I hate replying myself, I meant trying those parameters only.

charset= and sqlobject_encoding= only gave me problems.

Also, including the file my.cnf fails silently when the path is wrong
and can be a major pain when debugging (try it relative and abosolute
paths).

[ben.ha...@gmail.com]

Last I heard the guy who maintains the mysqldb at cheeseshop of pypi is
having a bit of difficulty with
knowing about this problem drop him an email his address if there. If
we bug him enough he will either wither or fix it. Be nice though
because he does seem a little worn.

[Dianne Marsh]

I had this problem too and ended up switching to all UnicodeCols and
then I still have times when I need to call str() on the DB result from
my controller to eliminate the error. I'm using MySQL as well.

Dianne

[Ian Wilson]

Yeah none of that seemed to work. Any more ideas?

-Ian

[Max Ischenko]

Ian Wilson wrote:
> Yeah none of that seemed to work. Any more ideas?

I'm afraid we ran out of the decent ideas. Personally, I use the
following dburi and the patch included below to make it all work. IIRC
SQLObject developers even merged this patch into codebase but only in a
dev branch, not the stable version TG uses (and requires)

sqlobject.dburi="mysql://...?client_encoding=utf8"

Index: sqlobject/mysql/mysqlconnection.py
===================================================================
--- sqlobject/mysql/mysqlconnection.py (revision 1744)
+++ sqlobject/mysql/mysqlconnection.py (working copy)
@@ -19,6 +19,10 @@
self.user = user
self.password = passwd
self.kw = {}
+ if kw.has_key('client_encoding'):
+ self.client_encoding = col.popKey(kw, 'client_encoding')
+ else:
+ self.client_encoding = None
for key in ("unix_socket", "init_command",
"read_default_file", "read_default_group"):
if key in kw:
@@ -47,6 +51,8 @@

if hasattr(conn, 'autocommit'):
conn.autocommit(bool(self.autoCommit))
+ if self.client_encoding:
+ conn.query('SET NAMES ' + self.client_encoding)

return conn

[Ian Wilson]

I am sorry. Also I should actually start reading exception traces for
once. I think the problem is more of the data going IN than the data
coming OUT. I was doing a query like this:
Plant.select(Plant.q.genus==searchWord) and it turns out searchWord
was unicode. And I don't know why it is unicode because I stopped
using a Unicode validator and starting using a regular String
validator.

So I think my problem is happening because of this:
>>> unicode(unicode('', 'utf-8'),'utf-8')

Traceback (most recent call last):

File "<stdin>", line 1, in ?

TypeError: decoding Unicode is not supported

AND because changing:
myquery = unicode(query, self.encoding)
To this:
if type(query) != unicode:
myquery = unicode(query, self.encoding)
else:
myquery = query
in the file "sqlobject/mysql/mysqlconnection.py" fixes the problem.

I really don't know if this is a "fix" or avoidance of the problem
because my knowledge is minimal. Thank you everyone for your
responses sorry if my previuos posts were misleading and I am really
confused but its working now.

-Ian

On 10/31/06, Max Ischenko <isch...@gmail.com> wrote:
>
>

[Max Ischenko]

Ian,

On 11/1/06, Ian Wilson <vengful...@gmail.com> wrote:

So I think my problem is happening because of this:
>>> unicode(unicode('', 'utf-8'),'utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: decoding Unicode is not supported

AND because changing:
                        myquery = unicode(query, self.encoding)
To this:
                    if type(query) != unicode:
                        myquery = unicode(query, self.encoding )
                    else:
                        myquery = query
in the file "sqlobject/mysql/mysqlconnection.py" fixes the problem.

Indeed. I was writing about this problem some time ago(*), just got confused about the  strack trace you've been showing.

* - http://maxischenko.in.ua/blog/entries/89/sqlobject-unicode-and-ascii-error/

 

I really don't know if this is a "fix" or avoidance of the problem
because my knowledge is minimal.    Thank you everyone for your
responses sorry if my previuos posts were misleading and I am really
confused but its working now.

Glad to hear it finally works for you.

[Kevin Dangoor]

It shouldn't, unless you have unicode objects heading into one of the
StringCols. The UnicodeCol doesn't do anything other than utf-8 (by
default) encode on the way to the database and decode on the way out.

Kevin

[JimmyDean]

Ian,
Just wanted to thank you for posting this 'fix'.
I'm pretty new to both Python and TG and this problem was occurring in
the TG wiki tutorial.
I'm going to reference this thread as a comment in the tutorial to help
anyone else who runs into this.