Re: Unable to storing unicode data using django

[Tom Evans]

On Fri, Apr 12, 2013 at 6:38 AM, sastry mln <sast...@gmail.com> wrote:
> Hi All,
> I have a mysqldb with utf8 charset and utf8_general_ci collation. The
> tables are also explicity defined to use the same charset/collation
> I have a django model with CharField.
> I want to store unicode characters (hindi language chars)
> when i tested the model form django admin shell, I got error saying that
> the character is unrecognised.
>
> I read the unicode section of the documenation, but i did not understand
> clearly.
>
> can someone please post a snippet about how to let the web application
> allow unicode data entry using (A) from django shell (b) From the data
> entry screen created through the modelAdmin forms
>
> regards
> mln
>

Can you show an example of both cases please - error traceback from
admin page when you get this error, and traceback (and code) from
doing it in the shell.

Unicode usually just works, particularly from the admin. Tracebacks
will probably show why it failed in this case.

Cheers

Tom

[Anshum Verma]

Utf8 is default support in django unless you are explicitly typecasting to str. Please provide details of traceback and code snippets to debug this.


Sent from my Windows Phone

---

From: sastry mln
Sent: ‎12-‎04-‎2013 17:29
To: django...@googlegroups.com
Subject: Unable to storing unicode data using django

Hi All,

  I have a mysqldb with utf8 charset and utf8_general_ci collation. The tables are also explicity defined to use the same charset/collation

  I have a django model with CharField.

  I want to store unicode characters (hindi language chars)

  when i tested the model form django admin shell, I got error saying that the character is unrecognised.

  I read the unicode section of the documenation, but i did not understand clearly.

  can someone please post a snippet about how to let the web application allow unicode data entry using  (A) from django shell (b) From the data entry screen created through the modelAdmin forms

regards

mln

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[mln sastry]

Hi Anshum,

  Thank you for responding.

  I defined a Book model as shown below and tried to save it from the admin shell.

  I am using python2.7.3 , django1.5, and python mysql driver 1.2.3.

  I am not sure if my model data type is wrong or if any qualifier is missing. Please help.

 

  I got this error:Warning: Incorrect string value: '\xE0\xA4\xB6\xE0\xA5\x8D...' for column 'bookName' at row 1"

class Book(models.Model):

    bookId = models.AutoField(primary_key=True)

    bookName = models.TextField(max_length=200,null=False,)

    bookShortName = models.CharField(max_length=10,null=True,)

    isDeleted = models.NullBooleanField  (null=True,)

 

gita4 = Book(bookName= "श्रीमद्भगवद्गीता",bookShortName="गीता",isDeleted=False)

gita4.save()

Traceback (most recent call last):

  File "<console>", line 1, in <module>

  File "C:\py\lib\site-packages\django\db\models\base.py", line 546, in save

    force_update=force_update, update_fields=update_fields)

  File "C:\py\lib\site-packages\django\db\models\base.py", line 650, in save_base

    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "C:\py\lib\site-packages\django\db\models\manager.py", line 215, in _insert

    return insert_query(self.model, objs, fields, **kwargs)

  File "C:\py\lib\site-packages\django\db\models\query.py", line 1661, in insert_query

    return query.get_compiler(using=using).execute_sql(return_id)

  File "C:\py\lib\site-packages\django\db\models\sql\compiler.py", line 937, in execute_sql

    cursor.execute(sql, params)

  File "C:\py\lib\site-packages\django\db\backends\util.py", line 41, in execute

    return self.cursor.execute(sql, params)

  File "C:\py\lib\site-packages\django\db\backends\mysql\base.py", line 120, in execute

    return self.cursor.execute(query, args)

  File "C:\py\lib\site-packages\MySQLdb\cursors.py", line 204, in execute

    if not self._defer_warnings: self._warning_check()

  File "C:\py\lib\site-packages\MySQLdb\cursors.py", line 117, in _warning_check

    warn(w[-1], self.Warning, 3)

Warning: Incorrect string value: '\xE0\xA4\xB6\xE0\xA5\x8D...' for column 'bookName' at row 1

[Anshum Verma]

This seems to be coming from MySQL . Can you check what is the default charset for the Book table in mysql and also that of the individual columns. You can share the output of show create table from mysql.

Regards,
Anshum

Sent from my Windows Phone

---

From: mln sastry
Sent: ‎13-‎04-‎2013 11:07
To: Anshum Verma
Cc: django...@googlegroups.com
Subject: Re: Unable to storing unicode data using django

[mln sastry]

Hi Anshum,

 At first I modified the db charset with below commands. It did not help.

ALTER DATABASE mybooks DEFAULT CHARACTER SET = 'utf8' DEFAULT COLLATE = 'utf8_general_ci'

ALTER TABLE   book  DEFAULT CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'

 It worked after I tried this

     ALTER TABLE   book   CONVERT TO CHARACTER SET 'utf8' COLLATE 'utf8_general_ci'

[Anshum Verma]

The issue is when you alter table all the columns switch to original charset which is usually latin1. If you run the command show create table mybooks you will be able to see that.

Sent from my Windows Phone

---

From: mln sastry
Sent: ‎13-‎04-‎2013 20:14

[Tom Evans]

On Sat, Apr 13, 2013 at 6:37 AM, mln sastry <sast...@gmail.com> wrote:
>
>
> Hi Anshum,
> Thank you for responding.
> I defined a Book model as shown below and tried to save it from the admin
> shell.
> I am using python2.7.3 , django1.5, and python mysql driver 1.2.3.
> I am not sure if my model data type is wrong or if any qualifier is
> missing. Please help.
>
> I got this error:Warning: Incorrect string value:
> '\xE0\xA4\xB6\xE0\xA5\x8D...' for column 'bookName' at row 1"
>
> class Book(models.Model):
>
> bookId = models.AutoField(primary_key=True)
>
> bookName = models.TextField(max_length=200,null=False,)
>
> bookShortName = models.CharField(max_length=10,null=True,)
>
> isDeleted = models.NullBooleanField (null=True,)
>
>
>
> gita4 = Book(bookName=
> "श्रीमद्भगवद्गीता",bookShortName="गीता",isDeleted=False)

This is not specifying a unicode string, it is specifying a byte
string. A unicode string starts u"Some text". Django correctly handles
unicode text, but if it is given byte strings, what happens depends
upon your environment - mostly, it will just not work.

There is an excellent document describing python 2.x and it's unicode handling:

http://docs.python.org/2/howto/unicode.html

however, the only things you really need to do:

1) Specify unicode strings as unicode by prefixing with u, eg
u"श्रीमद्भगवद्गीता"
2) Any python file that you include non-ascii characters in must
specify the encoding the file is in, by adding a special comment to
either the first or second line in the file:
# -*- coding: utf-8 -*-

Cheers

Tom