Unicode text in a ListItem

[david guez]

Hello,

Since I updated my linux box to the latest Mageia version (Mageia 5), one of the program I developed with wxpython is broken, and the cause seems very strange to me.

Shortly, I'm feeding a ListCtrl with unicode strings and, when these strings contains special chars (like a french é), the list box insert a blank line instead.

A reproduced the bug in a short program (joined file).

Note that there is a commented line 

# wx.SetDefaultPyEncoding('utf8')

that I tried to use, but the bug is the same with or without this line...

my version of python is 

Python 2.7.9 (default, Dec 14 2014, 10:12:16) 

[GCC 4.9.2] on linux2

>>> import wx

>>> wx.VERSION

(3, 0, 1, 1, '')

>>> wx.GetDefaultPyEncoding()

'ISO-8859-1'

and my Mageia system is configured to "french / france"

The same program used to work fine with me with the last Mageia release....

Does anybody can give me a clue on what I'm doing wrong ?

Many thanks in advance.

D. Guez

[Werner]

The only way I got this working with Py2.x is using a bit of a hack,
which is using "sys.setdefaultencoding('utf-8')", see the enclosed
changed file.

Maybe you had used sitecustomize.py before?

Werner

[david guez]

Thank you Werner... I tried it and it does not help, nor adding the lines

import sys

sys.setdefaultencoding("utf-8")import sys

sys.setdefaultencoding("utf-8")

to the python27\lib\site-packages\sitecustomize.py file...

I'm a bit lost!

Thank you anyway...

Cheers,

D. Guez

[Vlastimil Brom]

> --

Hi,
sorry, if I am missing something obvious, but I don't understand the
purpose of the explici decoding at line 18
r=row.decode('utf8')
Normally the communication of text content within the gui, I'd expect
to just pass the unicode objects directly.
If I disable this by using r=row instead and suply the expected
unicode literals for python 2 u"..." at lines 31-33, especially
self.frame.add_line(u'test éé')
everything seems to be working, but this is on windows 7, I
understand, there might be platform differences.

hth,
vbr

[david guez]

Hi,

Actually, the real code get the data from a sqlite database and the result of the query command can be str or unicode according to the string content. This is the reason why I explicitly changed the format to utf8.

But I tried a modified version ( r=row instead of r=row.decode('utf8') and self.frame.add_line(u'test éé ') and the problem is the same.

Do you mean that for you, the first version of my code reproduce the bug where the second does not ? If so, it could give me a clue on what's happen...

Thank you for your help,

david

[Vlastimil Brom]

> --
Hi,
sorry for the confusing notice - no, both versions work for me (they
appear to be equivalent), I just thought, that is would be more
convenient to work with unicode objects rather than encoded strings,
but your notice about a db as the original data source explains it.
I hope, someone will have a better explanation of the problem.

regards,
vbr

[Werner]

Hi,

On 6/28/2015 0:33, david guez wrote:
> Thank you Werner... I tried it and it does not help, nor adding the lines
> import sys
> sys.setdefaultencoding("utf-8")import sys
> sys.setdefaultencoding("utf-8")
>
> to the python27\lib\site-packages\sitecustomize.py file...
>
>
> I'm a bit lost!
> Thank you anyway...

I just tried again the changed sample I attached last time and it does
work for me on Windows 7 with Py2.7 and either wxPython 2.8.12.1 Unicode
or wxPython 3.x Phoenix.

I can make it work either with the setdefaultencoding trick or by
adapting the change Vlastimil suggested (not using decode line).

Did you check that you use a Unicode build of wxPython? As of 2.9 there
are only Unicode builds, but with 2.8 there where ANSI and Unicode builds.

How are you getting the data out of sqlite? Directly or are you using
e.g. SQLAlchemy?

Werner

[david guez]

Hi,

Tell, my wx version is 3.0.0.1, so I guess it should be a unicode version, and I'm getting the data directly from the sqlite database, which returns me str or unicode data according to the string content.

Does the unchanged version (those I sent) fails as it does for me ?

Anyway, the line

print type(r), r , "=>",itm.GetText()

shows me that I'm feeding the ListCtrl with unicode string which is well formated but not well taken into account by ListIm.

I realize that I didn't sent the output of my program. As you can see on the console windows, the second line is feed with well formatted data (unicode string, well printed on the screen) but is emptied when passing through the SetText method

(see the output line in the console window :

<type 'unicode'> test éé =>

showing that the r variable is a unicode string corresponding to the "test éé" value but, after I gave it to the ListItm, the result is an empty sring.

Le mardi 30 juin 2015 15:55:58 UTC+3, werner a écrit :

Hi,

[Werner]

On 6/30/2015 15:24, david guez wrote:

Hi,

Tell, my wx version is 3.0.0.1, so I guess it should be a unicode version, and I'm getting the data directly from the sqlite database, which returns me str or unicode data according to the string content.

Does the unchanged version (those I sent) fails as it does for me ?

Just tried your original version and it works for me too with wxPython 2.9.5 on Win7 and on Linux Mint 17 with wxPython 2.8.12 Unicode.

Note that your original version does not feed unicode but it feeds strings, see lines 31-33, to the listctrl.

Are you sure you are using wxPython 3.x?

What about with the enclosed version?

Werner

[david guez]

Hi Werner,

that's right, the line 31-33 get str or unicode data, but this corresponds to the fact that sqlite returns me str / unicode content. The thing is that I'm ensuring that I'm feeding the ListCtrl with unicode content via the line 

 r=row.decode('utf8')

and I'm sure that I do this okay because the output of my code for the print type(r) is indeed <unicode>.

I'm also confident that the french page of the utf8 charset is present on my system since I can print the é char  on a terminal configured with the utf8 encoding.

I tried your version of the code which also gives me white lines (see the attached screenshot).

thx...

D. Guez

[Werner]

Hi,

On 6/30/2015 22:54, david guez wrote:
> Hi Werner,
> that's right, the line 31-33 get str or unicode data, but this
> corresponds to the fact that sqlite returns me str / unicode content.
> The thing is that I'm ensuring that I'm feeding the ListCtrl with
> unicode content via the line
> r=row.decode('utf8')
> and I'm sure that I do this okay because the output of my code for the
> print type(r) is indeed <unicode>.
> I'm also confident that the french page of the utf8 charset is present
> on my system since I can print the é char on a terminal configured
> with the utf8 encoding.
>
> I tried your version of the code which also gives me white lines (see
> the attached screenshot).

Only difference I can see is with your 'locale', I have "en_US.UTF-8"
but you have "fr_FR".

What do you get when you add:

print(wx.GetDefaultPyEncoding())
print(sys.getdefaultencoding())

On my Mint I get:
UTF-8
ascii

Werner

[david guez]

hi,

same as you... many thanks for all the time you give to me by the way :-) 

[david@localhost ~]$ python

Python 2.7.9 (default, Dec 14 2014, 10:12:16) 

[GCC 4.9.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import wx,sys

>>> print(wx.GetDefaultPyEncoding())

UTF-8

>>> print(sys.getdefaultencoding())

ascii

Werner

--
You received this message because you are subscribed to a topic in the Google Groups "wxPython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wxpython-users/56SMg8t4mMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wxpython-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

           | bye !  |
  David GUEZ   gue...@gmail.com

[Werner]

Hi,

On 7/1/2015 22:54, david guez wrote:

hi,

same as you... many thanks for all the time you give to me by the way :-) 

[david@localhost ~]$ python

Python 2.7.9 (default, Dec 14 2014, 10:12:16) 

[GCC 4.9.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import wx,sys

>>> print(wx.GetDefaultPyEncoding())

UTF-8

>>> print(sys.getdefaultencoding())

ascii

It is really strange, for me, that your 'print' line shows the correct text for the variable 'r' but shows blank for 'itm.GetText()', if it is encoding issue it would normally show some strange symbols.

Just to be sure, when you run the program you don't get any error? 

Werner

[david guez]

Actually, I think the process is that one :

1 - I get in r a well encoded utf8 string.

2 - then I transfer it to the ListItem via the SetText method

3 - the SetText method for an unknown reason try to convert or to validate the string and fails to do it, so it set actually the text to nothing

4 - I'm retrieving the text that the SetText method actually set, which is empty because of the validation above mentioned

my problem is : does this conversion / validation fails because my input is not the expected one or is it a bug of wxpython.... I guess this is a bug but very hard to reproduce in other systems

[Werner]

On 7/2/2015 13:34, david guez wrote:
> Actually, I think the process is that one :
> 1 - I get in r a well encoded utf8 string.
> 2 - then I transfer it to the ListItem via the SetText method
> 3 - the SetText method for an unknown reason try to convert or to
> validate the string and fails to do it, so it set actually the text to
> nothing
> 4 - I'm retrieving the text that the SetText method actually set,
> which is empty because of the validation above mentioned
>
> my problem is : does this conversion / validation fails because my
> input is not the expected one or is it a bug of wxpython.... I guess
> this is a bug but very hard to reproduce in other systems

Just for a test, can you try this:
if isinstance(row, str):
r=row.decode('utf8')
else:
r = row
self.lbDocuments.InsertStringItem(0, r)

I.e. not using ListItem, to see if it is a ListItem issue or ...?

Werner

P.S.
Could you also try using wxPython Phoenix?
http://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip