Tech Agreement between Google/MIT/Ai2/Yandex

[moon]

Hi guys,

Is there any knowledge about the technical agreement between let's say AI (maybe MIT) & Yandex ?

Don't need to know all the details just

if any limitation then

• Number of requests allowed /day ? month? seconds ?
  
• or per App ?
  
• Max size of a given request in characters? words?
  

and equally important but it's probably not part of the agreement : what are the structure(s) that Yandex can handle (at least for AI). 

Have been through some pages from Yandex but nothing could help me identifying whether they can preserve some structures.

i.e : if I send a csv structured request, will I get it back as csv? 

of course I considered Steve's recommendations for one column csv file. But when it goes for large csv files with many rows and columns it ends with a headache.

From my understanding of their business, their engine considers the structures (HTML, XML, Json etc...) but how to get that working from our end ?

Any ideas/tips will be appreciated.

M

ps: I used to append whatever csv files I have into a unique text file with specific characters that allow me parsing (files, columns & rows) but it's not longer working now. I get rejected with the 2202 ERROR.

[SteveJG]

Regarding Yandex   

Probably the following applies:

Translate API — Troubleshooting — Yandex Technologies

The daily request limit is 1,000,000 characters. The monthly limit is 10,000,000 characters. To increase the request limit, switch to the fee-based version of the service.

Fees for using the paid service are listed on the Pricing page. If you agree to the fees, use the feedback form to send a request to switch to the fee-based version.

MIT probably will respond to this thread if the limits differ.

A Yandex 2202 error:  Error 2202: The Translation service is not available; Please try again later.   So, that means you cannot connect, either because the service is down (my experience is Yandex is not totally reliable)  or your request might contain too long a string to translate.   You need to experiment using your data.  Make a separate Project to experiment with what strings are possible and permitted.

Yandex has nothing to do with Google or MIT App Inventor.  It is a Russian translate service.

[moon]

Hi Steve, thx for the reply,

Firstly, just need to tell that what I'm doing for the moment is testing/probing before considering to include anything in my App.

Using the link you provided I sent an XML formatted file but still got the same error.

In fact, small texts without any special characters are treated correctly.

When is comes to texts that include some characters that are frequent (#, |) we get that 2202 Error.

The bottom line is that the error code transmitted does NOT reflect the real situation.

I might move to use Google's option, but that requires probably some investment that I'm going to start now. Unless there is another option.

M

[Chris Ward]

Hi moon

I think you will find the Google service to be similar. It's your job to remove characters that are not included in words and send the service a squeaky-clean file.

[Chris Ward]

[moon]

Hi Chris,

Thx for the suggestion.

Actually I deliberatly added the special characters in order to be able to parse the answer from Yandex.

In my case I have several CSV files (not all the same size however several lines & columns each).

The best/easy way I found is to amalgamate the whole things in a single "phrase i.e text" pass it to Yandex and then parse in order to retrieve the items correctly.

All that after having cleaned the text from double spaces, & \n.

I used the # to separate the columns & the pipe to retrieve the lines.

Seems I'm in a dead end. :-(

Unless.....

M

[Chris Ward]

Hi Moon

Does your app ultimately display the text in a tabulated/spreadsheet format?

[Chris Ward]

...by the way, pipe is commonly be used to separate columns :)

[SteveJG]

Carlos, have you tried URL encoding your sent text using the Web component?

[moon]

No Chris all that is about managing the texts displayed in the app.

The main constraint is that the texts (downloaded as CSV from G.Sheet thru a request) are dynamic.

In other words some users (admin) may add columns and rows (data), the others (very final end-users) need to have access to the latest status of the "database" translated to whatever language they're comfortable with.

I can put an example:

Consider an app that provides means for the end-users to answer questions related to some services.

On the other hand you got several entities that delivers those services. Those entities need (for whatever reason) to ask specific questions related to the service they deliver.

You end-up with a table that is dynamic because the entities may add new questions, reformulate them or even having new entities entering the system.

here we are. You got the idea. At least I hope so

thx again for the follow-up

M

[moon]

for a guy coming from Unix/Linux ithe pipe | is also used for another purpose ;)

M

[moon]

Nope because from the link you sent, they clearly mentioned that we need to be careful cuz the length of that encoding may grow a lot ....

Should I ?

even if I need to learn how to do that.

M

[SteveJG]

:)    OK, don't try.    Just note that from Yandex "Note. All special characters must be escaped."      and that includes commas, pipes etc.

[moon]

Meanwhile I tried :-) and I confirm that it didn't work.

Bottom line, I do think that the only option is to generalise your suggestion for the one column csv file. A double loop that will kill the processor :-)

[moon]

Hi

Not answering myself, just following up as I did several tests.

At the end I managed to get Yandex accepting the structure that includes those special characters !

Can post the blocks and/or the aia if it can help any saving time in case facing the same.

The strange thing is that from the algorithm I run, it seems that each post can't accept more than 10,000 characters.

If over then it returns 2202 

Don't know if anyone can confirm or not.

Thx

M

[Chris Ward]

Hi moon

Certainly would be good for all if you could share your blocks.

I'm wondering if the 10,000 characters limit is a new limit specific to the free service - since there are huge numbers (400,000 unique monthly active users) of people using App Inventor, that alone probably generates a lot of demand. Then again, 10,000 is a long way off 1 million, so perhaps it has more to do with the amount the translator can deal with in one session. You could ask Yandex.

[moon]

Okay here we are.

Did another set of tests just to be sure again and I confirm that over 10,000 it's rejected.

I included in my texts some emails, URLs, ref to files etc to be sure all what might go through the pipe works.

As you say this is way beyond the stated limit !! and so frustrating !

Meanwhile I'm asking Yandex now.

Keep all informed as it looks like a serious limit.

M

[Evan Patton]

Please note that the 1 million character limit is a daily limit, not a per-request limit. My guess is that the 10k character limit you are running into is probably a server configuration issue, since the content of the request is sent as part of the HTTP URI to the server. Most HTTP servers will have some hard limit on the length of a URI, and my guess is that you are hitting it. In that case, consider breaking your request into more manageable chunks.

Regards,

Evan

[moon]

Hi Evan

Thanks for sharing your thoughts/guesses.

I asked Yandex and still waiting for any answer that we can consider as a basis for our future developments.

Meanwhile, as you suggested I started yesterday working on the preparation of batches. 

Not optimal because of the asynchronous process, and since I need those translation timely, I have to keep the processor doing ... nothing ....until I get all the translations.

M

ps: I'll get back to all once I get an answer from Yandex.

[Evan Patton]

Is this translation step the same for all users, or are the inputs different per user? In the former case, you might want to consider doing the translation offline and then having the different translations as assets in your project rather than running it as part of your app.

Regards,

Evan

[moon]

Hi Evan

"Unfortunately" this is specific to the user (International University Campus)

Even though was thinking about making an offline translation in all the European languages. It's an option, however not dynamic as in my case the content will change according to the University events/moments during the day... and I wont control that every time a change/add is done, just impossible in a large Campus where almost everyone, if not many,  can add/remove/change an information/news/data at any time.

I know my logic is kind of brut since it doesn't consider only what changed. The fact is that I'm not using a proper database, just several Google spreadsheets.

M

[TimAI2]

Could always consider a google apps script ;)

https://developers.google.com/apps-script/reference/language/language-app

[moon]

For sure Tim, that was one of my options early in the fight, however have been told that I’ll face the same :-)

See the previous answers I got.

If it’s something that you consider worth investing then I’ll go for it. Why not.

What matters at the end is having something working properly.

M

[TimAI2]

Nothing in the quotas for Google Services:

https://developers.google.com/apps-script/guides/services/quotas

Only thing that could cause a problem is the limitation on the number of executions at 30

for Google Cloud things appear different:

https://cloud.google.com/translate/quotas

???

[moon]

Thx Tim,

From my reading of your links and others they states this:

Translation API is optimized for translation of short requests. The recommended maximum length for each request is 2K. 

Translation API will reject very large requests (with a 400 INVALID_ARGUMENT error) regardless of the available quota.

What's the meaning of 2K ? I don't know. is it 2Kbits ? is it 2000 characters ???

All in considering the figures they put they are way ahead of Yandex, therefore may be worth for me investing.

Hope to find quickly  the right guide that goes straight to the point very.

In case you have that I'll appreciate following the route and not reinvent the wheel.

M

[TimAI2]

Without fully understanding how your app is setup to communicate the data and changes, I would do something like this:

Base data stored in google Sheet / csv style

Each language required has its own google sheet /tab translated from the base data

Google Apps script will up date translations for any changes to base data (but only those items that change)

So far all this happens in google sheets and apps script

User selects their preferred language as a part of setup

This is communicated to the google sheet, which pulls down the correct language csv to populate fields / data

If user CRUD (create/update/delete) any data this info is fed back to the google sheet/s and the base data and language datasets are updated by apps script 

[Chris Ward]

Hi Moon

The recommended maximum length for each request is 2K

Since that's a measure of length, it means 2000 characters in total. 

[moon]

Hi Tim

Thx for the idea.

Since I have a bounded number of languages (27) I ought to consider your option.

I'll start a test and see how I can deal with G.scripts to make my CSVs updated whenever there is a change/add.

It looks to me that I need to have a cron-like (Unix ref) process that run in the background to ensure that whatever change is made the translation occurs.

I'll have a deep thought about it

Thx again very much

M

[moon]

Wow that's then weird

At the same time it's a recommendation ! whatever it means.

M

[TimAI2]

Just use a trigger on the google apps script (assumes you are running it from the sheet) to watch for a new submission / append.

You could use a hidden to user google form for users to submit entries, which makes this a bit easier to act on a form submit.

[moon]

Thx Tim, I'll work on it.