[ENCODING] Problem with encoding

[Tomek]

Hello everyone,

I have a problem with encoding. I use Java, and Appium for my iOS app. Now app which i am testing will support 8 languages, and in those 8 languages one is special: JAPANESE.

I have a problem that when i for example gt text form some element on my app suing simple:

driver.find....getText()  , I get the japanes text, but it is not the same as in Original Localziable.strings file which is set to app.

Probably there are some issues with encoding, so, could You tell me waht could be wrong:

1. Is appium has set some encoding ? if yes, which one?

2. Is it fault of webdreiver and its encoding? and how to change it if it is possible?

Some other ideas?

[bootstrap online]

It may be an issue of UTF-8 vs UTF-16. I don't have a localized iOS
app to test with though.

> --
> http://appium.io
> ---
> You received this message because you are subscribed to the Google Groups
> "Appium-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to appium-discus...@googlegroups.com.
> Visit this group at http://groups.google.com/group/appium-discuss?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Tomek]

But in my opinion its not connected with utf16. In my java classes everywhere i have utf8. Localizable.strings file has no encoding, and in project in xcode what i see there is utf8 encoding set.

Problem i have with some chracters from japanese, like for example,

We have letter "i" With dot at top and this letter is in localizable file. But when i get label with this text using gettext on webelement i get instead of it two chars

"i" without dot and lonely dot. But they together looks like the this one mentioned before.

I dont know how to move it. It looks like appium or webdriver has something doing with those characters,

[bootstrap online]

Are you willing to share a build of your app with Sauce Labs so they
can reproduce and fix the problem? Alternatively, is there a test app
that can reproduce the issue? If there's a way to repro the error then
I'm sure it'll get fixed quickly.

[Tomek]

So i had problems with sharing my app because of privacy etc. etc. (I am only a QA of this app), but what i can is to share Localizable.strings file with japanese translation and with key. If someone knew XCode or/and have some testApp (I don't know objective-C language at all unfortunaltely), he can use it those two added translations in his app, and it will be much quicker than creating test app by me.

in Attachment there is Localizable.strings file with translations for Japanese.

There are only 2 of several keys with characters like that. 

The issue is: When it is set in app, applciaiton on ipad works propelry, it looks like text is translated properly and we see good char, but if I get this text using getText() method, I get this text (the frist one), which is longer than this original form file, becuase somehow Appium split the last CHAR into to char (split DOT and JapanesCharacter ???).

The same issue is with the second key, but there is more characters with this Japanes DOT and the end of letter so this String is change to more longer one , and some characters are split into 2, char and dot....

Can You or somebody help me/us in this case?

Thanks for responses 

Tomek

[bootstrap online]

Hi,

I built an iPhone app using the provided Localizable.strings and
reproduced the issue. The string value is directly from Apple's test
framework so it's not a bug in Appium.

I normalized the unicode composition and then the strings matched as
expected. Here's an article on how to do that in Java.
http://docs.oracle.com/javase/tutorial/i18n/text/normalizerapi.html

The Ruby code I used is here.
https://gist.github.com/bootstraponline/e5cbff8208cea324889b

I hope that helps.

[Tomek]

Hello,

Thanks a lot ! :) I've tried this normlaizing and it works fine for japanese and also for other languages. 

Great work, Thanks a lot one more time.

Tomek

[dan.do...@lookout.com]

I'm seeing a similar issue in the Spanish version of our app.  A string from localizable strings file is in UTF8 and the string as read through appium is in UTF8.  The string is displayed correctly in the app, but reading the string through appium gives an extra byte and of course this causes failure because the expected title from the app does not match the string read from disk:

The char: ó

Read from appium:
111
204
129

Read from file:
195
179

So what's really the fix here and why is the string as read through appium different than what's in the app?

[bootstrap online]

Are you normalizing?

> Visit this group at http://groups.google.com/group/appium-discuss.

[Dan Doveralba]

Trying to understand the problem first. I have a UTF8 localizable.strings that's given to the app and I read the localizeable.strings file from the disk with File.open(file_name, 'r:UTF-8'). I'm not understanding why the string would be represented differently read from Appium vs. read from the file.

But no, not normalizing yet. Is it something like this?

string_read_from_file.bytes.pack('c*').force_encoding('UTF-8')

string_read_from_appium.bytes.pack('c*').force_encoding('UTF-8')

That doesn't seem to impact the bytes in the string so maybe I'm missing something.

[Dan Doveralba]

unicode_utils gem did the trick.

UnicodeUtils.nfc(string_read_from_file) == string_read_from_appium

If I wanted to do this across the board, could I monkey patch web driver so that all text returned was normalized?

[Dan Doveralba]

Answering my own question…I was able to patch WebDriver to always normalize and all the test code stays the same. Awesome.

module Selenium
module WebDriver
class Element
def text
UnicodeUtils.nfc bridge.getElementText @id
end
end
end
end

[Dan Doveralba]

I ended up patching alerts too.

module Selenium
module WebDriver

class Element
def text
UnicodeUtils.nfc bridge.getElementText @id
end
end

class Alert

def text
UnicodeUtils.nfc @bridge.getAlertText
end
end

end
end