mysql2 and swedish characters

[jeb]

When using the mysql2 gem with my excising databases all non-standard
chars gets messed up:

från is shown as frÃ¥n in the browser

Is this possible to fix or do I have to continue using the old mysql-
gem?

[Frederick Cheung]

IIRC mysql2 forces use of utf8. If you're elsewhere telling the
browser that you're using a different character set then you'd get
unwanted results.

Fred

[jeb]

I think it's more a question of how the data is stored in the
database.
In my layout i have:
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
and in application.rb:
config.encoding = "utf-8"

:-) j

On 30 Juli, 20:15, Frederick Cheung <frederick.che...@gmail.com>
wrote:

[Frederick Cheung]

On Jul 30, 8:31 pm, jeb <jo...@ibiz.se> wrote:
> I think it's more a question of how the data is stored in the
> database.
> In my layout i have:
> <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
> and in application.rb:
> config.encoding = "utf-8"
>

Also possible. If previously the database connection was configured as
some latin1 variant and your columns were also latin1 but you were
stuffing utf8 data into them then switching to mysql2 would cause data
to appear messed up.

Fred

> :-) j
>
> On 30 Juli, 20:15, Frederick Cheung <frederick.che...@gmail.com>
> wrote:
>
>
>
> > On Jul 30, 7:00 pm, jeb <jo...@ibiz.se> wrote:
>
> > > When using the mysql2 gem with my excising databases all non-standard
> > > chars gets messed up:
>

> > > från is shown as från in the browser

[jeb]

That seems to work, the problem is when reading the data using mysql2.
In the database å is stored as Ã¥.
When I write to the database using mysql2, without having changed it,
it writes å.


On 30 Juli, 22:17, Frederick Cheung <frederick.che...@gmail.com>

[Frederick Cheung]

On Jul 30, 9:21 pm, jeb <jo...@ibiz.se> wrote:
> That seems to work, the problem is when reading the data using mysql2.

> In the database å is stored as å.

> When I write to the database using mysql2, without having changed it,
> it writes å.
>

What does mysql think the column types are ? Before using mysql2, what
encoding was set in your database.yml ?

Fred

[Nathaniel Brown]

== Welcome to Vidli

Vidli ( http://vidli.com ) is the first of its kind Open Source Video
eCommerce platform. Specifically designed for video content owners that
are looking to setup a distribution storefront. Vidli is built on the
Ruby on Rails v3 framework.

Out of the box, Vidli integrates with PayPal Express checkout for
payment processing and Amazon S3 for video streaming and storage.

== Getting Started

Download the source code

git clone http://github.com/vidli/vidli

Copy the database.yml.example file to database.yml. Update with your
login and password to your MySQL database.

cp config/database.yml.example config/database.yml

Configure your AWS settings

cp config/amazon_s3.yml.example config/amazon_s3.yml

Create the development and production buckets on AWS via
http://console.aws.amazon.com that matches your config file.

Upload the lib/crossdomain.xml file to each bucket. Make sure
permissions have Everyone -> Open/Download.

Signup at PayPal for a developer account

http://developer.paypal.com

Setup your Vidli app config and ActiveMerchant for PayPal payments

cp config/vidli_config.yml.example config/vidli_config.yml

== Try it out

Visit http://demo.vidli.com to try out the Vidli app for yourself.

== Who is Vidli for?

- Video content creators
- Video distributors

== Vidli comes with

- Simple, easy to use video content management system
- Non-commercial use JW Player video player (when you go live this will
need to be purchased)
- Seamless integration with PayPal Express
- Integration with Amazon S3 for video storage and playback

== License

Vidli is released under the MIT license.

[jeb]

No encoding set in database.yml, cp1252 set in mysql. By it self I
guess.



On 30 Juli, 22:37, Frederick Cheung <frederick.che...@gmail.com>

[Frederick Cheung]

On Jul 30, 9:50 pm, jeb <jo...@ibiz.se> wrote:
> No encoding set in database.yml, cp1252 set in mysql. By it self I
> guess.
>

that sounds like you should change the encoding for your existing
tables/columns to utf8

One way of doing this is to alter the columns to blobs and then back
to a text column with the correct encoding (as documented at the
bottom of http://dev.mysql.com/doc/refman/5.0/en/alter-table.html)

[jeb]

Yes, that seems to work, but takes some time.
Thanks!

On 30 Juli, 23:58, Frederick Cheung <frederick.che...@gmail.com>
wrote:

> On Jul 30, 9:50 pm, jeb <jo...@ibiz.se> wrote:
>
> > No encoding set in database.yml, cp1252 set in mysql. By it self I
> > guess.
>
> that sounds like you should change the encoding for your existing
> tables/columns to utf8
>
> One way of doing this is to alter the columns to blobs and then back
> to a text column with the correct encoding (as documented at the

> bottom ofhttp://dev.mysql.com/doc/refman/5.0/en/alter-table.html)

[michael_teter]

Bummer, this didn't work for me.  I think I'm having a similar problem.

I was using mysql gem, database was utf8, but I was getting an occasional template encoding error.  That prompted me to update my driver to mysql2 (which solved my template problem), but resulted in formerly "correct" characters getting botched.

Björn became Björn.  Or maybe it always was that, but mysql driver was somehow presenting it as we want to see it.  Regardless, going to mysql2 left me with a lot of screwed up names :(

I also tried the column -> blob -> text trick, and sadly that didn't appear to change anything.

Do you think there's some simple algorithm I can use to map screwy things to their correct versions?

Thanks!

[michael_teter]

More info...

In rails console, mysql driver shows this for the column value: Bj\xF6rn
mysql2 driver shows this for the column value: Björn

F6 hex is the UTF8 code for o umlaut, so the mysql(1) view seems more correct than the mysql2 view of the column value...

[Frederick Cheung]

On Sep 14, 11:56 am, michael_teter <michael.te...@gmail.com> wrote:
> Bummer, this didn't work for me.  I think I'm having a similar problem.
>
> I was using mysql gem, database was utf8, but I was getting an occasional
> template encoding error.  That prompted me to update my driver to mysql2
> (which solved my template problem), but resulted in formerly "correct"
> characters getting botched.
>
> Björn became Björn.  Or maybe it always was that, but mysql driver was
> somehow presenting it as we want to see it.  Regardless, going to mysql2
> left me with a lot of screwed up names :(
>
> I also tried the column -> blob -> text trick, and sadly that didn't appear
> to change anything.
>

I once had a case where through an accident of history, some data was
doubly bad, so I had to go through the text->blob->text approach
multiple times

F6 is actually a latin1 code point, which suggests that you still
haven't got things in utf8 properly

Fred

[michael_teter]

Ahh, I'll make a few more blob/text round trips and see if that helps.

Yeah, the F6 thing was created by mysql(1) driver.  Now mysql2 apparently doesn't understand.

Thanks!