Re: Yandex Translate issue

[Jos]

This seems to be an issue on our side; could you please give us a few words with the problematic characters and their English translations so that we can do some testing? Many thanks for your report.

[BlankWorld Apps]

Some words with this problem:

ışık = light

çocuk = kid, child

ağaç = tree

bağış = donation

[Jos]

Thank you; I'll file it as an issue and do a bit of digging. No ETA though. Thanks!

[Sven Wiebus]

I ran into the same issue while translating words which include polish and other languages' special characters. This most likely has to do with somehow messed-up UTF-8 encoding. What I saw is, that if you replace the special characters with for instance slashes ("/") you most of the time receive the same false translation results, which gave me the impression that the special characters are ignored by Yandex because they are encoded/ sent as some "crap characters" rather than "valid letters".

To test this theory of sometimes defective UTF-8 encoding I implemented a very simple Yandex translator myself by putting together some Text- and Web-Blocks in AI2 in order to compare its results against the results of your build-in Yandex translator blocks. The emulated translator does not have the issue which the special characters (i.e. it translates all the problematic examples correctly).

Maybe you find it helpful to compare the Yandex-API URLs (simple HTTP-GET URLs) my small emulator calls against the URLs your Yandex-Blocks generate for the sames words. I attach the small demo-project to this message. To get it work you will have to assign your Yandex translate API Key to the global variable "yandex_api_key" on top of the project.

Regards
Sven

[Jos]

I found the problem, guys. My fault for not encoding the string going up the server. There will be a fix in the next release.

[Sven Wiebus]

Thanks a lot Jos!

[Sven Wiebus]

I found another issue with the build in Yandex translate blocks which do not appear with my own Yandex translate emulator. First I though the issue was a matter of the number of the text to be translated. But by performing further tests I realized that it also happens with shorter sentences if they contain specific words, while these words itself or in different context are translated without error messages: This happens also for words with just plain ASCII characters, so most probably this time its not an encoding issue.

Here are three examples. When I try to translate the following two snippets from German to English;

• Meine Damen und Herren
• geehrte Herren

I receive this notification alert from the build in AI2 Yandex translator blocks:

Error 2203: The response from the Yandex translation service cannot be parsed; please try again later.

However, by just flipping the two words in the last example to

• Herren geehrt

the error alert does not show up anymore and I receive a translation from the AI2 translator:

When I translate the same 3 sentences with my own emulated Yandex translator all three snippets translate just fine without problem. So the "parsing"-error from the AI2 translator blocks do not seem to be due to malformed Yandex XML-responses.

I append again the ".aia"-project file of my small test App amended by the above 3 examples.

Please note: To get App work you need to assign your own Yandex translate API Key to the global variable "yandex_api_key" on top of the project.  

[Sven Wiebus]

Jos,

I had a brief look on Github into "YandexTranslate.java" and found that the

Error 2203: The response from the Yandex translation service cannot be parsed; please try again later.

(ErrorMessages.ERROR_TRANSLATE_JSON_RESPONSE)

in YandexTranslate.RequestTranslation(...) is caused by an exception thrown by JSON, so outside your direct area of control. I have checked the Yandex-JSON-API responses to the three small examples from my post above via Web-Browser (Firefox 31) and got the following results:

Text to translate: Meine Damen und Herren
URL: https://translate.yandex.net/api/v1.5/tr.json/translate?key=<your Yandex key goes here>&lang=en&text=Meine%20Damen%20und%20Herren
Response: {"code":200,"lang":"de-en","text":["Ladies and gentlemen"]}

Text to translate: geehrte Herren
URL: https://translate.yandex.net/api/v1.5/tr.json/translate?key=<your Yandex key goes here>&lang=en&text=geehrte%20Herren
Response: {"code":200,"lang":"de-en","text":["dear gentlemen"]}

Text to translate: Herren geehrt
URL: https://translate.yandex.net/api/v1.5/tr.json/translate?key=<your Yandex key goes here>&lang=en&text=Herren%20geehrt
Response: {"code":200,"lang":"de-en","text":["Mens honored"]}

So at least when called manually in a Web-Browser the responses to all three examples look syntactically equal, i. e. no obvious reason why two of these translations end up in AI2 with the above JSON parser exception leading to the 2203-error message, while the third does not:

{"code":200,"lang":"de-en","text":["Ladies and gentlemen"]}
{"code":200,"lang":"de-en","text":["dear gentlemen"]}
{"code":200,"lang":"de-en","text":["Mens honored"]}

Is it possible for you for debugging reasons to include the result of a call to je.getCause() to the AI2 error message in the

catch (JSONException je) {

form.dispatchErrorOccurredEvent(YandexTranslate.this, "RequestTranslation",
ErrorMessages.ERROR_TRANSLATE_JSON_RESPONSE);

}

branch of your code to hopefully see what the JSON parser stumbles upon on the first two of the above examples?

Thank you very much in advance!
Sven

[Jos]

Hi Sven, I am trying to duplicate this problem but your app is working fine for me. Is it still happening?

[Sven Wiebus]

Hi Jos,

yes, the problem still persists. Below is a screenshot from my Samsung Galaxy S2 running my small demonstration App in "AI Companion 2".

When trying to translate the German snippet "geehrte Herren" the error message "2203" (ERROR_TRANSLATE_JSON_RESPONSE) is shown by your Yandex translator method

AsynchUtil.runAsynchronously(new Runnable()) {
   ...

   catch (JSONException je) {
      form.dispatchErrorOccurredEvent(YandexTranslate.this, "RequestTranslation", ErrorMessages.ERROR_TRANSLATE_JSON_RESPONSE);
   }
}

caused by an JSONException thrown by either;

final String responseCode = jsonResponse.getString("code");   or
org.json.JSONArray response = jsonResponse.getJSONArray("text");   or
final String translation = (String)response.get(0);

within your method

private void performRequest(String languageToTranslateTo, String textToTranslate)

However, requesting the same translation through my own emulated Yandex translator works fine, so this error is not caused by a malformed Yandex response, but by the JSON class used in your translator.

If you do not receive this same error message, can it be that you already use a version of AI2 where you fixed the missing UTF-8 encoding, while I still use the MIT online version which not yet include this fix? I doubt this is the reason because the text snippet only contains pure ASCII characters, so UTF-8 or non UTF-8 encoding should the not change the result from the Yandex server. But who knows?

Here is the Screenshot. Marked in red the result of your translator, marked in blue the result of my emulated translator.

[Jos]

Hi Sven, yes, I am testing with the new changes; it seems to fix this issue too. It should be out in the next release.

[Sven Wiebus]

Hi Jos, when do you think the next release will become available?

[Jos]

Unfortunately, I have no idea. Also note that this needs to be a release with a change to the Companion; if you hear of a release with no Companion, then the change will not be in.