Re: patch to DBD::ODBC for odbc_utf8_on

[Peter J. Holzer]

On 2009-11-04 10:33:59 -0800, Noel Burton-Krahn wrote:
> We connect to Aster (http://www.asterdata.com/, based on PostgreSQL)
> via unixODBC. We found that Aster always returns utf8 strings, but
> when they got to perl, multibyte utf8 sequences were not translated
> into unicode characters. So, the character single character
> "\x{263A}" from the database would be sent to perl as a two character
> string like "\x{26}\x{3A}".

Just a nitpick:

The character "\x{263A}" is "\x{E2}\x{98}\x{BA}" in UTF-8.

hp

--
_ | Peter J. Holzer | Auf jedem Computer sollte der Satz Ludwigs II
|_|_) | Sysadmin WSR | eingeprägt stehen: "Ein ewig Rätsel will ich
| | | h...@wsr.ac.at | bleiben, mir und andern."
__/ | http://www.hjp.at/ | -- Wolfram Heinrich in desd

[Martin Evans]

Noel Burton-Krahn wrote:
> Hi Martin,
>
> Here's a patch against DBD::ODBC from subversion to fix utf8 handling
> for Aster databases.

>
> We connect to Aster (http://www.asterdata.com/, based on PostgreSQL)
> via unixODBC. We found that Aster always returns utf8 strings, but
> when they got to perl, multibyte utf8 sequences were not translated
> into unicode characters. So, the character single character
> "\x{263A}" from the database would be sent to perl as a two character
> string like "\x{26}\x{3A}".
>

> My solution was to added a "odbc_utf8_on" flag to the dbh and sth
> handles. If the odbc_utf8_on flag is set, DBD::ODBC will call
> svUTF8_on() on all strings returned from the driver before returning
> them to perl.
>
> I looked at the unicode support in DBD::ODBC, but couldn't get it to
> work with Aster since Aster does not return characters as SQL_C_WCHAR.
>
> I made an exception so odbc_utf8_on would not affect SQL_C_BINARY
> columns, but I haven't been able to test it.
>
> Thanks for all your work on this module. We use it a lot. I hope my
> contribution is useful. Please let me know if you need anything else
> from me, or if there's another way to solve this problem
>
> Noel Burton-Krahn
> no...@burton-krahn.com <mailto:no...@burton-krahn.com>

Thanks Noel.

I will look at this as soon as I can and get back to you.

The ODBC unicode support in DBD::ODBC requires a driver to support the
so called "Wide" functions - SQLxxxW and has little to do with
supporting SQL_C_WCHAR. Columns may be bound as SQL_C_WCHAR even in non
unicode builds of DBD::ODBC in which case they are returned as is from
the driver.

I am not aware of there being any existing way of handling ODBC drivers
that return UTF-8 encoded data and to this day I still cannot understand
how drivers can do this within ODBC and still support APIs like
SQLGetData see http://www.martin-evans.me.uk/node/20#unicode and
http://www.easysoft.com/products/data_access/odbc_oracle_driver/unicode.html#odbc.

However, if your driver returns UTF-8 encoded data consistently and you
are happy this patch works I'll try and incorporate it.

BTW, I only just noticed your CC list - I guess this is related to:

"Enquisite selects Aster Data to Scale its Worldwide Search Data Network
November 2, 2009"

Martin
--
Martin J. Evans
Easysoft Limited
http://www.easysoft.com