cegui & unicode strings

[Krešimir Špes]

when I write smtg like this:

loading_text=cegui.WindowManager.getSingleton().getWindow("loading_text");
loading_text.text=u'Učitavam mapu'

the second non-ascii character doesn't get displayed correctly:
http://img91.imageshack.us/img91/3076/snapshot3dw9.jpg

the .py file is saved as UTF8

[Krešimir Špes]

so, I tried the python way, but still, the unicode character gets
drawn incorrectly in cegui:

s = u'U\u010ditavam mapu' #\u010d - č
loading_text.text=s

[Krešimir Špes]

and the same thing happens with utf-8:

s='\xc4\x8dikara'

the font I use has the charachers I need, and the C++ version of my
program uses the same resources and unicode chars get displayed
correctly.

[Krešimir Špes]

so, any clues? anyone? this is really important for my project :/

[Lakin Wecker]

Does CEGUI support the use of unicode strings in C++? 

Lakin

[Roman Yakovenko]

On 5/21/07, Lakin Wecker <lakin....@gmail.com> wrote:

Does CEGUI support the use of unicode strings in C++? 

Good question: are you using\build bindings with same dll\library as with C++ application?

Lakin

On 5/21/07, Krešimir Špes < kresim...@gmail.com > wrote:

so, any clues? anyone? this is really important for my project :/

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[Krešimir Špes]

yes, I am using the same library. I'm running this on Python-Ogre RC1
linux (Debian). I'll test it on Windows and let you know.

[Andy Miller]

I suggest that you will have to create a cegui.string first and use it
to set the text attribute.

Unfortunetly I won't be able to test for a day or two

Andy

[Krešimir Špes]

I've tried cegui.String but to no avail. it would seem something very
strange is going on with unicode strings :/

1) I've set a unicode string with special chars in the layout file:
'čikara' and it was displayed correctly and output of
loading_text.text.c_str() returned u"čikara"

2) when set u"čušo" in code, I get u"Ã\x84Â\x8duÃ…Â\xa1o" from the
c_str() ?

3) when I set a unicode utf8 string ('\xc4\x8dikara') I get: u"Ä
\x8dikara"

I've yet to try this on windows though.

[Krešimir Špes]

I've tried it on windows, same thing. utf8 string results in
displaying the wrong character..

[Krešimir Špes]

[Roman Yakovenko]

On 5/24/07, Krešimir Špes <kresim...@gmail.com> wrote:
>
> I've tried it on windows, same thing. utf8 string results in
> displaying the wrong character..

I will try to find some time to understand what is going on.

[Andy Miller]

The fix is to create a CEGUI String using the 'cstr' name value..

 

       s=CEGUI.String(cstr='\xc4\x8dikara')
       editorWindow.text=s
       

 

Cheers

Andy

 

[Roman Yakovenko]

On 5/24/07, Andy Miller <nzmi...@gmail.com> wrote:
> The fix is to create a CEGUI String using the 'cstr' name value..
>
> s=CEGUI.String(cstr='\xc4\x8dikara')
> editorWindow.text=s

And this will fix the bug? If so we can add next code to CEGUI __init__.py file

def String( s ):
if isinstance( s, str ):
return _cegui_.String( s )
elif isinstance( s, unicode ):
return _cegui_.String( cstr=s )
else:
raise some exception

What do you think?

[Andy Miller]

In fact I've probably got this wrong - sorry.. 

 

What I really need is a complete test program as I don't seem to have the necessary character maps installed (or enabled) for my testing to be valid..

 

Further investigation is required :)

 

Andy

 

On 24/05/07, Roman Yakovenko <roman.y...@gmail.com> wrote:

[Andy Miller]

OK - so more testing and I have a work around -- unfortunately I need to understand a lot more about unicodes etc to create a real fix (unless of course someone else has the time :) )

 

        ## Test with the Euro symbol
        special=CEGUI.String(num=1,code_point=8364) # this makes a single char string with the unicode character
       
        ## and let's check some appending etc..
        f=CEGUI.String("Start")
        temp=f.append(special,0,1) ## append the unicode string
        end = CEGUI.String("End")
        temp=temp.append(end,0,3)
        editorWindow.text=temp  
       

Cheers

Andy

 

[Krešimir Špes]

I've tried this and it works, albeit a poor workaround :/

Init_Load=cegui.String(num=1,code_point=0x010d)
Init_Load.append('ikara')

there's really not that much to unicode as one might think, it is
basicly an universal code page that contain EVERY single character in
existance. Thusly, a single character has to have 32 bits.

there are many unicode transformation formats, most comonly used would
be utf8 which is an ascii extension that works in a way that all codes
lower then 128 are treated as ascii and all larger are unicode chars
that span two or more bytes.

cegui in c++ comonly uses utf8 for cegui.String assignment which is
why I'm puzzeled as to why '\xc4\x8d' results in cegui transforming
this into something different :? So, we should concentrate on making
cegui understand python utf8 strings to solve the problem.

also, cegui can take int32* which coresponds I believe to python's
unicode strings, but they get transformed wierdly as well

[Krešimir Špes]

I've managed to work-around the issue by using cegui.String.append()
however the process is preety slow and all my localisation data is
stored in utf8 which means I have to first convert it to utf32 before
I can use append.

Can someone please tell me how to debug this? I would gladly do it,
but I don't know how :/ By default, cegui takes utf32 unsigned char*
strings however it seems to be having a problem with converting python
strings to unsigned char

[Andy Miller]

Can you send me some UTF8 strings (with descriptions etc of what they are meant to show) as there seem to be multiple implementations and it's something I'm "learning"...

 

It should be reasonably simple to fix once I understand it better

 

Cheers

Andy

[Krešimir Špes]

I do believe I've come to the bottom of it :)

1) 0x010d is a unicode codepoint of 'č' (c with an inverted ^ above
it)
2) '\xc4\x8d' is a utf8 transformation of that character into a
standard 8-bit string.
3) s=cegui.String('\xc4\x8d') results into two diferent characters:
196 & 141 (C4 & 8D)

s[0] returns 196, s[1] returns 141 and s[2] returns 0. however, s[0]
should've been 296 (010D) and s[1] : 0

so, my educated guess is that in String.__init__ each ascii charcter
is taken and inserted seperately into the string which is okay with
ascii chars but not with utf8 chars which have to be first converted
into unicode codepoints before insertion.

can you post cegui.String constructor code please?

On May 27, 3:08 pm, "Andy Miller" <nzmill...@gmail.com> wrote:
> Can you send me some UTF8 strings (with descriptions etc of what they are
> meant to show) as there seem to be multiple implementations and it's
> something I'm "learning"...
>
> It should be reasonably simple to fix once I understand it better
>
> Cheers
>
> Andy
>

[Andy Miller]

Not quite sure what you mean by constructor code -- it's all open source ( built on the standard CEGUI code base) and/or Python-Ogre generated code..

 

What is best is for me to get a simple C++ demo working, then look at the difference for Python..

 

Cheers

Andy

 

[Andy Miller]

I'm looking at the CEGUI 'String' class and indeed it needs some work :).. 

 

There are a number of unsigned char *'s used as arguments and these don't automatically translate well across to Python - infact I think I was lucky it worked at all...  So I'm doing some testing and will hopefully update the SVN over the next day or 2 with a fix

 

Cheers

Andy

 

[Krešimir Špes]

hooooray :D

if you need any more info on unicode and utf-X, don't hesitate to
ask :)

[Andy Miller]

I've updated the SVN with an initial change -- not yet fully tested as I'm moving development machines and have to do a complete rebuild..

 

You need to create a CEGUI string and then 'assign' it with a python string: 

cs = CEGUI.String()

cs.assign ('\xc4\x8dikara')

 

You can look in hand_made_wrappers (in code_generators/CEGUI) and look at the StringAssign function (at the top) to see what I'm doing...

 

It only deals with utf8 encoded strings at the moment -- could you check if it works for you.

 

On another note -- I do have a namespace bug/problem with the code generator for CEGUI that I'll look to fix later today, basically you can run generate_code multiple times and get different source for certain classes :(..

 

 

Andy