Hello, django users.

I tried entering a Cyrillic text for the poll question attribute, while
playing with example polls application. This resulted in a bunch of question
marks on the view page, instead of the expected "Как дела?" poll question,
which was in Cyrillic.
My firefox character encoding is set to UTF-8;
the following command:
$ env | grep -i utf
yields to
LANG=en_AU.UTF-8

Has anyone faced that problem before?
Am I supposed to switch something else in django or MySQL to get the
cyrillic text or am I doing something stupid?

Hello Anton,

In my experience, text can get broken on each and every step.

First, I would try to see whether the browser sends it correctly using a
dumping HTTP proxy (netcat or a simple perl script). Then, check whether
it is written correctly to the database (e.g., issue a SELECT in mysql).
Then, see if it is displayed correctly.

With kind regards,
--
Baurzhan Ismagulov
http://www.kz-easy.com/

On Dec 9, 11:25 pm, Baurzhan Ismagulov <i...@radix50.net> wrote:

*The netcat part*:
Here's what I did:
1. opened an admin interface, went to
http://localhost:8000/admin/polls/poll/add/ and filled it with some
cyrillics.
2. ran shell command:
$ nc -l -vv -p 8000 localhost > cyrillic_trouble
3. pushed "Save and continue editing" on the admin page.
Thus I got browser's POST request in cyrillic_trouble file. The POST's
question parameter taken from that  file looks this way:

question=%D0%9A%D0%B0%D0%B3+%D0%B4%D0%B8%D0%BB%D0%B0%2C+%D0%BA%D1%80%D0%BE% D1%81%D0%B0%D1%84%D1%87%D0%B5%D0%93%3F

which is URL encoded text I did entered on the admin page.

So is it supposed to be like this?

I netcated the response to
GET /admin/polls/poll/3/ HTTP/1.1
Host: localhost:8000

The troubled line (the one with question marks) looks like this:
<input type="text" id="id_question" class="vTextField required"
name="question" size="30" value="??? ?????" maxlength="200" />

*The MySQL part*:
I did:
<code>
mysql> select * from polls_poll;
+----+--------------------+---------------------+
| id | question           | pub_date            |
+----+--------------------+---------------------+
|  1 | what's up?         | 2006-12-05 15:05:00 |
|  2 | how are you today? | 2006-12-03 15:22:00 |
|  3 | ??? ?????          | 2006-12-09 12:24:13 |
+----+--------------------+---------------------+
</code>

the third row was inserted via admin interface and as you can see it's
just question marks.

then I tried to ran the following SQL:
mysql> insert into polls_poll (`question`, `pub_date`) values ('Ты
используешь Джанго?', NOW());

now it's displayed ok in the mysql client, but the admin interface
shows a different style garbled text instead:
Ты Ð¸Ñ Ð¿Ð¾Ð»ÑŒÐ·ÑƒÐµÑˆÑŒ
Джанго?
which is
Ты используешь Джанго?

I'm desperately stuck for now. Hope someone will push me towards.

 m> then I tried to ran the following SQL:
 mysql>> insert into polls_poll (`question`, `pub_date`) values ('Ты
 m> используешь Джанго?', NOW());

 m> now it's displayed ok in the mysql client, but the admin interface
 m> shows a different style garbled text instead:
 m> Ты Ð¸Ñ Ð¿Ð¾Ð»ÑŒÐ·ÑƒÐµÑˆÑŒ
 m> Джанго?
 m> which is
 m> Ты используешь Джанго?

Here UTF-8 text has been treated as Latin1 and encoded again as UTF-8.

--

Anton Daneika напиша:

Which mysql version are you using?

I'm using mysql 5.0.24a and django from svn - utf-8 cyrillic text works
fine out of the box in a browser with a default codepage of utf-8. The
system locale is mk_MK.utf8.

To get mysql client in the console working with utf-8 (doesn't have
anything to do with Django) I have this in /etc/my.cnf:

[client]
.
.
default-character-set = utf8

[mysqld]
.
.
character-set-server=utf8
collation-server=utf8_unicode_ci
init_connect='set collation_connection = utf8_unicode_ci;'

--
Glisha
The perfect OS, MS-DOS!
No patches, no root exploits for 21 years.

On 12/11/06, Georgi Stanojevski <gli...@gmail.com> wrote:

$ mysql --version
mysql  Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (i486) using readline 5.1

I'm using mysql 5.0.24a and django from svn - utf-8 cyrillic text works

I use Django 0.95 official version

To get mysql client in the console working with utf-8 (doesn't have

Well, I tried this conf modification.
Before it I could do
mysql> insert into polls_poll  (`question`, `pub_date`) values ('Часто ли у
вас возникают проблемы с кирилицей?', NOW());

Then the select statement would produce a correct cyrillic output, but after
the proposed conf modification I get garbled text in mysql client...

:)

welcome to the wonderful world of charsets :)

(sorry, i have no idea how this works in mysql(i'm a postgresql user),
but if you see it garbled in mysql-client, that does not mean things got
wrong... unfortunately there are many-many factors that may affect these
things)

so, good luck, and don't give up :)

gabor

Thank you for support :)
It helps to keep myself up.

On 12/11/06, Gábor Farkas <ga...@nekomancer.net> wrote:

This looks good, at least I have no problems with Cyrillic characters
passed to the server in that way.

Could you please od the value part, to eliminate the possibility of the
terminal - shell - catting tool chain scrambling the output?

Could you please od this, too?

Aha. So, what is the output of "locale" in the shell you started mysql
in? Does mysql inherit it? What is the encoding of the database you are
writing to?

With kind regards,
--
Baurzhan Ismagulov
http://www.kz-easy.com/

well at last the problem is solved.

i was up to dig into the od'ing everything on my way, but decided to try to
mess with MySQL and it helped.

1.
at first i did what Georgi Stanojevski suggested:

To get mysql client in the console working with utf-8 (doesn't have

2.
then i restarted the mysql server with (i use ubuntu):
$ sudo /etc/init.d/mysql restart

3.
from mysql client
mysql> alter table polls_choice convert to character set utf8;
mysql> alter table polls_poll convert to character set utf8;

and finally the input of cyrillic in the admin area started to work!

the interesting fact is that in the same time running the SELECT statement
from mysql client yields:
mysql> select * from polls_choice;
+----+---------+-------------+-------+
| id | poll_id | choice      | votes |
+----+---------+-------------+-------+
|  1 |       1 | ??? ?????   |     0 |
|  2 |       1 | ????? ????? |     0 |
+----+---------+-------------+-------+

to output of question marks, which i suppose displays, that the client
doesn't handle utf8.

I want to thank all the people, who tried to help me. I will try my best to
be helpful as well
:)