problem storing UTF8 data to the DB

[mason]

Hello,

I am still RedBean newbie, but it is almost working for me. I got my
web app talking to it, sending user-input data to PHP backend, which
uses redbean to save to MySQL DB. Great! Easy!

The only problem is, my Japanese text is getting corrupted when
written to the DB. Can't tell for sure, but looks like it is getting
converted to latin1, or similar.

Now, for testing I output same data to browser as well. Browser gets
UTF8, but DB gets gibberish. Is there any special thing I need to do
with regard to UTF8 for RedBean? I know PHP has poor support for
Unicode, so it might be more of a generic PHP thing, not specific to
RedBean. But just in case, I ask.

Code is like this:
-----
// gets raw POST data (test UTF8 text from frontend web app):
$raw_data = file_get_contents("php://input");

// use utf8 string as title for bean
$testDoc = new PublishedDocument; // simple test class, uses redbean
$testDoc->setTitle($raw_data); // should set to the utf8 string data
$id = $testDoc->save();

// for test, output bean's title back to web browser :
$output = $testDoc->getTitle();
header("HTTP/1.1 200 OK");
header("Content-Type: text/html");
header("Content-Length: " + strlen($output));
echo($output);
-----

(Now I hope this google group can deal with the UTF-8 characters
below...)

When I see the string output in the web browser, it is proper UTF-8
encoded "日本語日本語ブラブラブラ。"

However, when I dump the database, or look at it in Sequel Pro
database access app, the string looks like "日本語日本語ブラブ
ラブラ。".

Is there something else special I need to do to make this work?

Thanks for any pointers!

P.S.
Database was created like this:

mysql> CREATE DATABASE workfro DEFAULT CHARACTER SET utf8 COLLATE
utf8_general_ci;

...and dumpfile for that table looks like:

DROP TABLE IF EXISTS `publisheddocumentbean`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `publisheddocumentbean` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

[desfrenes]

I don't know about Redbean's internal on this subject but the general
rule for php/mysql is to have
1/ your fields/tables declared as utf8
2/ "set names utf8" sql command before any request (unless your my.cnf
states utf8 as the default connection charset)

[gabor de mooij]

Can you see if the table collation is utf8?
Else, I have to adjust the query writer... good find..! can you report this bug at github if possible?

[gabor de mooij]

Hi
This issue has resulted in a fix; can you please verify whether this works?

Issue:
http://github.com/buurtnerd/redbean/issues/unreads#issue/1

RedBean 0.6.5.1 contains the fix:
http://redbeanphp.com/downloads/redbean0651.zip

Hopefully this solves the problems, if not please tell me! ;)
Thanks for your feedback, this is the only way to make RedBean a better product! Thanks for investing time in testing RedBean!

[dkornishev]

Does the software infer charset or use DB default? Any way to control
it?

On Sep 10, 1:35 pm, gabor de mooij <gabordemo...@gmail.com> wrote:
> Hi
> This issue has resulted in a fix; can you please verify whether this works?
>
> Issue:http://github.com/buurtnerd/redbean/issues/unreads#issue/1
>
> RedBean 0.6.5.1 contains the fix:http://redbeanphp.com/downloads/redbean0651.zip
>
> Hopefully this solves the problems, if not please tell me! ;)
> Thanks for your feedback, this is the only way to make RedBean a better
> product! Thanks for investing time in testing RedBean!
>

> On Thu, Sep 10, 2009 at 2:47 PM, gabor de mooij <gabordemo...@gmail.com>wrote:
>
> > Can you see if the table collation is utf8?
> > Else, I have to adjust the query writer... good find..! can you report this
> > bug at github if possible?
>

[mason]

Hello,

I downloaded the latest 0.6.5.1 version, but still had the same
problem. But THEN, I spent some hours searching the net and trying out
suggestions from various blogs. Finally, I had edited MySQL's /etc/
my.cnf file in a way that made everything work (with the latest
RedBean 0.6.5.1). So perhaps this was not a RedBean problem, or
perhaps there were multiple problems.

Anyway, my simple test app is now writing correct Japanese text as
UTF-8 to the database. For reference, I post the entire contents of my
my.cnf file below:
-----
[mysqld]
default-character-set=utf8
default-collation=utf8_general_ci
character-set-server=utf8
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'

[client]
default-character-set=utf8
-----

Hope this helps somebody!

Thanks again.

On Sep 11, 2:35 am, gabor de mooij <gabordemo...@gmail.com> wrote:
> Hi
> This issue has resulted in a fix; can you please verify whether this works?
>
> Issue:http://github.com/buurtnerd/redbean/issues/unreads#issue/1
>
> RedBean 0.6.5.1 contains the fix:http://redbeanphp.com/downloads/redbean0651.zip
>
> Hopefully this solves the problems, if not please tell me! ;)
> Thanks for your feedback, this is the only way to make RedBean a better
> product! Thanks for investing time in testing RedBean!
>

> On Thu, Sep 10, 2009 at 2:47 PM, gabor de mooij <gabordemo...@gmail.com>wrote:
>
>
>
> > Can you see if the table collation is utf8?
> > Else, I have to adjust the query writer... good find..! can you report this
> > bug at github if possible?
>

[gabor de mooij]

RedBean uses a default for this: UTF8 Unicode. It is supposed to support all possible charactersets. There is no way to control this (yet).

[gabor de mooij]

Your database must include the collate utf8 statement also I believe, unfortunately RedBean cannot do this for you

[dkornishev]

Really? Is this since version 0.6.5.1? Version 0.6 created tables
with varchar columns with latin1_swedish_ci (which is the db default)
Once I changed default collate for the DB to utf8_unicode_ci it used
that instead.

I'll try version 651 today

On Sep 11, 1:45 am, gabor de mooij <gabordemo...@gmail.com> wrote:
> RedBean uses a default for this: UTF8 Unicode. It is supposed to support all
> possible charactersets. There is no way to control this (yet).
>

[dkornishev]

Looks like it defaults to utf8 in latest version, awesome.

However (there is always a however :))

it created columns with type longtext :o
The values were in Cyrillic and 5 characters long.

[gabor de mooij]

Can you turn on debug mode (last kickstart param) and send me the output plz. Dont know what happened...
Have you stored a large text in that column, else keepInShape() will autom. adjust the column for you..

[dkornishev]

Sent via email form.

On Sep 11, 8:44 am, gabor de mooij <gabordemo...@gmail.com> wrote:
> Can you turn on debug mode (last kickstart param) and send me the output
> plz. Dont know what happened...
> Have you stored a large text in that column, else keepInShape() will autom.
> adjust the column for you..
>