exception message encoded in utf8

[jean-philippe dutreve]

Hi all,

I use SQLAlchemy-0.4.2p3, postgreSQL 8.2.4 (UTF8 configured) and
psycopg2.
I have no issue with unicode DATA in and out of the database.

My problem is that when an IntegrityError is thrown, the exception
message is a string encoded in utf8.
And the logging module throws an UnicodeDecodeError.

Is there a way that the exception message is type of unicode instead
of string?

Thanks for help,
jp

[Michael Bayer]

logging module itself throws UnicodeDecodeError ? are you sending
exception messages using logging.debug() or similar ? my impression
is that you'd want to decode those manually doing something like
string.decode('utf-8').

[Michael Bayer]

of course my collegue suggests, "maybe we should decode exception
messages within SA using engine.encoding", *after* I push send....but
I want to make sure im understanding how you're getting it to raise
the error in the first place since we're not able to trip up logging
on this end.

[jean-philippe dutreve]

On 7 mar, 02:29, Michael Bayer <mike...@zzzcomputing.com> wrote:

> logging module itself throws UnicodeDecodeError ?

yes, in logging.format: ... = "%s" % msg
with msg the exception message encoded in utf8 and the default
encoding is ascii.

> are you sending exception messages using logging.debug() or similar ?

exactly: log.error("... : %s", e.message)

my impression
> is that you'd want to decode those manually doing something like
> string.decode('utf-8').

yes, it works fine, but it's pain to do this in each try/except.
Another solution is setting utf8 as the default encoding in
sitecustomize.py.
It's better centralized, but has sitepackage effect.

But a better way IMHO is that the DB driver or SA returns unicode
exception message.

[Sylvain Prat]

Sorry to ressurrect this thread but the problem is still there. Since SQLAlchemy knows the encoding used to communicate with the database, it can properly decode the error strings returned by the database to unicode. So, I think it should be SQLAlchemy's responsibility to convert the error strings to unicode, not the user's responsibility. Could we open a bug for that in the tracker?

Sylvain

[Michael Bayer]

OK as I said earlier, I'm not able to reproduce this.    So I'd need that reproduction case in order to do anything.   To be honest it sounds more like a psycopg2 bug, since psycopg2 does the decoding in most cases nowadays and even works with Python 3, so for it to be raising an exception with the "bytes" type for the message is certainly a bug.   But would need to see a real world example to get a feel for it.

--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/OOEvbKoo63cJ.
To post to this group, send email to sqlal...@googlegroups.com.
To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.

[Nico C.]

I think I can help reproduce this, but one has to configure the base system in a non C or english locale.
E.g. the system I work with is in french: it's default locale is fr_FR.UTF-8, hence the postgresql server
I installed on it runs with that locale too, by default. One can check with the `SHOW lc_messages;` SQL query.

Hence, with any error I trigger on the server side, like an IntegrityError or a ProgrammingError, comes
with a message from the server, which is encoded in that locale. For example many of messages translated
in french, come with «» (french guillemets) instead of double quotes which have psycopg or SQLAlchemy
choke on it. I would agree that is looks more like psycopg issue. It also rarely impacts production environments,
which are generally not set with fancy locale (at least I don't do that).

For reference, some ways to mitigate this are:

- reconfigure your postgresql serveur and have it use another default (requires "administrative" access to reconfigure the server);
- reconfigure your session so it use another session (cf. code snippet below, requires to be superuser which is rarely convenient);

Regarding the required priveleges cf. https://www.postgresql.org/docs/current/static/runtime-config-client.html.

@listens_for(_engine, 'connect')
def first_connect_callback(dbapi_connection, connection_record):
    # Registers composite types so psycopg2 know how to
    # handle them when they appear in resultsets.
    cursor = dbapi_connection.cursor()

    cursor.execute("SET lc_messages TO 'C';")
    cursor.execute("COMMIT;")
    cursor.execute("BEGIN;")  # Make sure you start a new transaction (may not always be appropriate)