Error UnicodeDecodeError in sql.py

[Jose]

The database server is MS-SQLServer

The following was working (is a simplification):

def test():
form = SQLFORM(db.DomiciliosSolicitud)
return dict(form=form)

in any of the updates stopped working. I do not know how long, because
that portion is used infrequently. Now we are migrating the server and
test that everything works well I found the error.

The error is as follows:

Traceback (most recent call last):
File "/var/web2py/gluon/restricted.py", line 173, in restricted
exec ccode in environment
File "/var/web2py/applications/autogestion/controllers/
solicitud.py", line 167, in <module>
File "/var/web2py/gluon/globals.py", line 96, in <lambda>
self._caller = lambda f: f()
File "/var/web2py/applications/autogestion/controllers/
solicitud.py", line 31, in prueba
form = SQLFORM(db.DomiciliosSolicitud)
File "/var/web2py/gluon/sqlhtml.py", line 618, in __init__
inp = self.widgets.options.widget(field, default)
File "/var/web2py/gluon/sqlhtml.py", line 194, in widget
options = requires0.options()
File "/var/web2py/gluon/validators.py", line 359, in options
self.build_set()
File "/var/web2py/gluon/validators.py", line 343, in build_set
records = self.dbset.select(*self.fields, **dd)
File "/var/web2py/gluon/sql.py", line 3018, in select
return self.parse(db,rows,self.colnames)
File "/var/web2py/gluon/sql.py", line 3040, in parse
value = value.decode(db._db_codec)
File "/usr/local/lib/python2.6/encodings/utf_8.py", line 16, in
decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbf in position 0:
unexpected code byte

Jose

[mdipierro]

The problem is not so much with this code but with the fact that there
seems to be non-unicode characters in one of your records. It comes up
because of the fields in the SQLFORM as an IS_IN_DB validator.

I have made a modification in trunk that will allow you hopefully to
get the data out of the database anyway.
You should try find out what field and record that is and correct it
anyway.

Massimo

[Jose]

On 12 ene, 17:01, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The problem is not so much with this code but with the fact that there
> seems to be non-unicode characters in one of your records. It comes up
> because of the fields in the SQLFORM as an IS_IN_DB validator.
>
> I have made a modification in  trunk that will allow you hopefully to
> get the data out of the database anyway.
> You should try find out what field and record that is and correct it
> anyway.
>
> Massimo

if you know what field, the problem that it is a legacy database. With
the change in the trunk stayed as it was before.
Thanks.

Jose

[mdipierro]

If it is a legacy database try use

DAL('mssql2://...') instead of DAL('mssql://...')

Does it mean you can read the data?

> Thanks.
>
> Jose

[Jose]

On 13 ene, 13:46, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> Does it mean you can read the data?

Yes, but it shows strange characters. I do not know how to change the
charset, and the administrator of the database does not know (or not
want) to tell me what charset it uses.

the connection is as follows:
SQLServer < FreeTDS < unixODBC < pyodbc < web2py

Jose

[mdipierro]

Look into gluon/sql.py/SQLDB/__init__ and the different between

DAL('mssql2://...') and DAL('mssql://...')

the difference is in the charset

[Jose]

change to:

db = DAL("mssql://..., db_codec='latin1')

now it looks all right. With mssql2 fails.

[mdipierro]

yes because mssql2 forces utf8. Good!