Has anyone seen the output from Rozetta and other Japanese MT vendors?
Kevin Johnson
I've also known my fair share of MT vendors, who are increasingly notorious for over-promising on quality. Maybe we're in the middle of a MT revolution or maybe a lot of it is just hype for our pair.
Kevin Johnson
Dan Lucas
--You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/89425891-6568-441a-962d-82e7798a4458%40googlegroups.com.
石崎敏明
Kevin Johnson
(There was a PR Times article from 2018 that had some side-by-side translations on it, and these were honestly terrible. But it seems Rozetta is now planning to release a "99% accurate" MT service in 2020, so we'll see whether it's snake oil or the real thing.)
JON JOHANNING
>
> (There was a PR Times article from 2018 that had some side-by-side translations on it, and these were honestly terrible. But it seems Rozetta is now planning to release a "99% accurate" MT service in 2020, so we'll see whether it's snake oil or the real thing.)
Jon Johanning
jjoha...@igc.org
Matthew Schlecht
JON JOHANNING
>
> (Which should all still constitute one sentence. One sentence that must be tied together properly.)
Great example of putting sentences into parentheses!
Jon Johanning
jjoha...@igc.org
Herman
modern AI-based MT systems, would typically refer to a score given to
the output of such a system by a modern AI-based MT evaluation system.
Such a system (e.g. BLEU) may work, for example, on the basis of N-gram
matching to a set of reference (human) translations.
So while the idea is, on one hand, that the accuracy of machine
translation is equivalent to the closeness of its output to human
translation, an interesting feature of these metrics is that the score
for a highly skilled human translator who makes no mistakes would not
necessarily be 100% and may actually be lower than that of a good MT
system -- because the output of the highly skilled translator may be
less of an averaged match to the set of (possibly mediocre) human
translations used as a reference for calculating the score.
Thus, claims that a given MT system has achieved 99% accuracy,
human-like performance, better than human performance, etc., are not
necessarily sheer advertising puff, and may be scientifically backed --
with the (likely unmentioned) proviso that "accuracy" in this case may
have a somewhat different meaning from what would typically be
understood by that term in other contexts.
Herman Kahn
Jon Johanning
I think "accuracy", in the context of evaluating the performance of
modern AI-based MT systems, would typically refer to a score given to
the output of such a system by a modern AI-based MT evaluation system.
Such a system (e.g. BLEU) may work, for example, on the basis of N-gram
matching to a set of reference (human) translations.
So while the idea is, on one hand, that the accuracy of machine
translation is equivalent to the closeness of its output to human
translation, an interesting feature of these metrics is that the score
for a highly skilled human translator who makes no mistakes would not
necessarily be 100% and may actually be lower than that of a good MT
system -- because the output of the highly skilled translator may be
less of an averaged match to the set of (possibly mediocre) human
translations used as a reference for calculating the score.
Thus, claims that a given MT system has achieved 99% accuracy,
human-like performance, better than human performance, etc., are not
necessarily sheer advertising puff, and may be scientifically backed --
with the (likely unmentioned) proviso that "accuracy" in this case may
have a somewhat different meaning from what would typically be
understood by that term in other contexts.
This could be called a “scientifically-backed” evaluation system, but it's a rather strange kind of science, I would say. It looks to me more like pseudo-science disguised by using a lot of fancy math to make it look like science.
The purpose of scientific activity is generally considered to be a search for truth, i.e., an understanding of reality. If it is asserted that an MT system shows a "human-like performance" and indeed may be even a "better than human performance," although we can see in experience (which is supposed to control scientific research) that MT products are often not as good as what skilled human translators can do (without considerable post-editing by humans, at least), and in many cases these translators judge that it would have been better for them to have done the translation from the start than to do the post-editing, I consider this not very congruent with reality.
I don't deny that end clients are often satisfied with MT products which are "good enough," as it is often said, for the clients' purposes. A lot of translation work that used to be done by humans when there were only human translators is now entrusted to MT, and the customers are happy with the results — they are cheap and fast. But as we know, these customers need to consider very carefully in which cases it really would be worthwhile for them to hire human translators to do the job. But these suggestions by the MT companies that their systems can replace human translators due to their tremendously superior speeds and cost reductions are the essence of false advertising, it seems to me.
The crux of the matter, I continue to believe, is that translation cannot really be done without understanding the meanings of the source texts, and computers, using AI or not, can’t possibly understand meaning. What they can do is simulate translation, and often that satisfies the customers.
Jon Johanning
Christiane Feldmann-Leben
Hi there,
I haven't followed this discussion closely, interesting as it is.
But now it seems to me that it is worth telling you about a
lecture at the last BDÜ conference three weeks ago (BDÜ is the
Association of Interpreters and Translators in Germany). It was
about product data sheets and a company took the trouble to train
8 different MT machines, that means to train with customer
terminology. The results were sobering for every translator. There
were different error categories from spelling to wrong translation
with appropriate weighting, and the number of errors was analysed.
In all cases, the number of errors dropped from about 20% to less
than 5% and, above all, mistranslations improved to 0%. The pie
charts previously looked green with lots of orange to red (red was
poor). Afterwards there was no red at all any more and actually
one saw essentially a green cake. Conclusion: Product data sheets
can be left to the machines with training in the future.
I agree that there are a number of types of text where MT reaches
its limits and nothing will work without qualified translators -
or rather experts in the field. Japanese will also be problematic
for a while. And especially tapeworm sentences - eternally long
sentences - will pose problems for even longer. I often find the
verb wrongly placed in posttediting. But I believe that this is
only a matter of time until enough data is available.
So Rosetta's planning 99% accuracy. Well, they will make it, at
least for certain types of text. And the verb at the end won't be
a problem.
And just for the record, this text was written in German and
translated by DEEPL. I haven't reworked anything at all that I
would do otherwise. I have already seen a few passages that I
probably would have formulated differently, for example the end of
the previous sentence. So 99% probably not, but judge the English
nevertheless times. Hm, this sentence was really creepy, but I
became more colloquial. I would be interested.
A nice day from Germany,
Christiane
--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/CALnYxcxmFA51co3pRV7dZT%2BSvPcMeksuNiPZBFiQ4PBAWS%3DZHA%40mail.gmail.com.
Dr. Christiane
Feldmann-Leben
Chemieübersetzerdienst
Böcklerstr.
1
D-76275 Ettlingen
Phon +49 (0)7243/217218
www.chemieuebersetzerdienst.de
in...@chemieuebersetzerdienst.de
Mark Spahn
Hi Christiane. It would have been interesting if you had included your
original German-text note, so that your readers can compare it with its
MT version.
-- Mark Spahn (West Seneca, NY)
Herman
> Well, they will make it, at least for certain types of text. And the
> verb at the end won't be a problem. And just for the record, this
> text was written in German and translated by DEEPL. I haven't
> reworked anything at all that I would do otherwise. I have already
> seen a few passages that I probably would have formulated
> differently, for example the end of the previous sentence. So 99%
> probably not, but judge the English nevertheless times. Hm, this
> sentence was really creepy, but I became more colloquial. I would be
> interested.
>
text through DEEPL, translating it back and forth between different
languages that I have some knowledge of, but then it said that "Due to
high traffic on the free DeepL translator, the service is limited at the
moment.", so I switched to Google Translate. Anyhow, things were going
relatively well in the sense that the meaning of the text was being more
or less maintained, albeit with some degradation. But then I threw Latin
into the mix (which I did "take" at one point, although my knowledge of
it now is, admittedly, rather poor), and that sort of messed some of the
text up into semi-gibberish, leading back to the following English:
"Well, at least some organization types can be supported. The last word
is that it doesn't matter. For information translated by DEEPL to be in
German. Otherwise I don't have to start over. The reason why you can't
see what you can't do is, as an example, another way, at the end of the
grace of the previous judgment is already past. 99% due to UK party
oblivion. This sentence is scary, but at most. I am interested"
However, further iterations seemed to improve the quality of the text
into something that made sense again:
"Well, at least some types of organizations are compatible. The last
suggestion is that it doesn't matter. Use DEEPL for German translation.
Otherwise, there is no need to start over from the beginning. For
example, he doesn't see what he can't do because it happened at the end
of a pre-estimated grace period. 99% due to forgetting things in the UK.
This prayer is terrible. be interested"
Now, on the one hand, this sort of underhanded activity could be, for
example, leveraged to impeach the very process of machine translation as
being inherently unreliable, but on the other hand, the process of going
through a number of machine translation iterations between a handful of
languages seemed to separate the wheat from the chaff, bringing into
clear relief what in retrospect may have been the underlying intended
meaning or message of the initial text: "Use DEEPL for German translation."
Herman Kahn
JON JOHANNING
I agree that there are a number of types of text where MT reaches its limits and nothing will work without qualified translators - or rather experts in the field. Japanese will also be problematic for a while. And especially tapeworm sentences - eternally long sentences - will pose problems for even longer. I often find the verb wrongly placed in posttediting. But I believe that this is only a matter of time until enough data is available.
Christiane Feldmann-Leben
Yes, garbage in, garbage out is certainly something, I am seeing
quite often. And sometimes the system draws funny conclusions -
which at least gives me a good laugh sometimes. An example:
asparagus is Spargel in German, and then there is the enzyme
asparaginase which is Asparaginase in German. However, I found
"Spargelinase". I loved that. Another one was "glass sons" instead
of "glass fibers" for "fils de verre". It took me quite a few
seconds before I understood what had happened.
The point, however, is that clients can train the machines with their own material and terminology lists (and the discussions turns to implementing rules as well). And if the clients have and use good translations, done by all of us, the machine will become frighteningly good. So in the end, our own good work turns against us.
This time, all errors are mine entirely. I wrote in English. ;-)
Christiane
--
dav...@gol.com
From: hon...@googlegroups.com <hon...@googlegroups.com> On Behalf Of JON JOHANNING
- Japanese is definitely problematic; how much longer it will be is hard to say.
Forever, would be my guess. Since Google translate still does the following.
OJさんは妻らを殺したとされました。
OJ was alleged to have killed his wives.
This is an in-principle problem that can’t be fixed by “more training data”, or anything else, without doing AI right. And the giddy insanity of contemporary AI hype means that very few people even understand the phrase “doing AI right”. (FWIW, the book “Rebooting AI” (highly recommended) does talk about what it would mean to do AI right.)
David J. Littleboy
Tokyo, Japan
Kevin Johnson
Kevin Johnson
Herman
> Great post, Christiane! It was like a short story with the reveal at the
> end...
>
> There are a couple things that are nagging me here. One, my
> understanding is that DeepL is trained on the EU corpus and therefore
> has absolute reams of high-quality side-by-side translations to work
> with. Meanwhile, as we all know, the full of body of Japanese-English
> work is garbage, and your average Japanese academic paper (or company
> website, or even your average Japanese-English dictionary, etc.) has a
> terrible English translation. So the texts may not be there (but,
> conversely, standards may be quite low).
>
> My second concern is that, as far as I know, German and English are
> extremely similar languages, making the task of automating such
> translations much easier. My understanding is that it's been much easier
> for Europe to make use of MT, and a ton of the huge success stories I
> hear are in the German-English pair specifically. I'm sure there the EU
> corpus enables similar results for many European pairs, especially
> closely related ones.
>
with, and plus the question of what level of similarity is to be
characterized as extreme vs slight is somewhat subjective and depends
also on what other pairs of languages one is comparing the given pair
to. I would say, though, that German and English are not so similar that
the translation between German and English would necessarily be easier
than, say, between English and Japanese, simply by virtue of greater
similarity at, e.g., morphological, syntactic and lexical levels. I
would say that the greater success that has apparently been had with MT
of languages based on the EU corpus (and the like) may derive more from
similarity of what is said than similarity in the means of saying it. In
other words, it is not the case that speakers of different languages
necessarily say 100% the same things but only do so in a differently
encoded manner, but rather, for various reasons -- including not only
differences of a phonological, morphological, syntactic or other such
purely linguistic nature, but also differences of a historical,
cultural, political, economic, philosophical, etc., nature -- speakers
of different languages may say different kinds of things.
Even if the languages are extremely different at a purely linguistic
encoding level, provided that a sufficient pattern of correspondences
can be established on the basis of an adequate bilingual corpus to begin
with, this type of difference would not be a source of difficulty for an
MT system. The difficulty would come from an absence of 100% or nearly
100% corresponding terms, expressions, etc., not from the fact that the
equivalent terms and expressions are formally dissimilar. This point
would be exemplified, for instance, by comparing the case of
(1) translating the language of a Chinese-speaking practitioner of
Western medicine into language of an English-speaker practitioner of
Western medicine, versus (2) the case of translating the language of a
Chinese-speaking practitioner of Chinese medicine into the language of
an English-speaking practitioner of Western medicine. Assuming that the
source language content remains invariant, the only way to establish a
system of translation in case (2) that would work as well as in case (1)
would be to in effect devise a consistent body of "terrible English
translation", and then accept that pattern of terrible English
translation as the correct translation.
Herman Kahn
石崎敏明
--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/cb7ad5c2-eac1-e306-e56e-18a971bb348b%40lmi.net.
Herman
> Jon Johanning writes:
>
> <snip>
> “Enough data” is a real problem for MT development. I’m not sure that
> the developers have yet solved the famous problem we have heard of in
> the past: the enormous amounts of translation data needed to train the
> systems is usually scraped from web sites, which are often translated by
> raw MT, without expert post-editing, so it’s a garbage-in, garbage-out
> situation.
>
> 私はAI開発や著作権の詳しい知識がありませんが、無知を覚悟で一言、二言。
> 仮に、データ・ソースがウェブサイトでなく書籍だったら、garbage-in,
> garbage-outにはなりませんよね。
> 例えば、GoogleがBig Shortのオリジナルと日本語訳の両方をインプットして、
> MTに活用したとすると、
> ゴミどころか、Beyond Meatのような商品性のある製品を複製することになります。
>
> こうなると、「金融・資産運用の オタク的な知識」という囲いの中にある
> 「人間保護区域」に住んでいる私は、柵を失ってロボットに対し全く無防備にな
> ります。
>
> 結論：書籍データの扱い次第では、今、そこにある危機はロゼッタではなく
> Googleである可能性も。
>
といっても、ソースデータが翻訳ミスをたくさん含んでいるのがGarbage、とは限らない。
例えば、
「年末やお正月のすき焼きにたっぷりの笹がきごぼうをぜひ加えてさっと煮て下さい。」
といったレシピからの一文に対して、Google Translateが
"Be sure to add plenty of mushrooms to the end of the year and New
Year's sukiyaki and simmer."
との結果を出す根拠は、「笹がきごぼう」を
mushroomsにした誤訳がインターネット上に多い、ということではなく、「すきやきにたっぷりのきのこを」「きのこたっぶりのすきやき」などの文がインターネット上に多いからである。
即ち、「すきやきにたっぷりのきのこ」等が、「すき焼きにたっぷりの笹がきごぼう」に対してGarbageとなる。
Herman Kahn
石崎敏明
mushroomsにした誤訳がインターネット上に多い、ということではなく、「すきやきにたっぷりのきのこを」「きのこたっぶりのすきやき」などの文がインターネット上に多いからである。
即ち、「すきやきにたっぷりのきのこ」等が、「すき焼きにたっぷりの笹がきごぼう」に対してGarbageとなる。
Herman
Translate
のようなAIを使ったMTならば、笹掻きごぼう≒きのこ、という関連付けができて、笹掻きごぼうをmushroomsと訳してしまう可能性が高い。即ち、対象文が訓練データのパターンに合わない場合です。
Herman Kahn
Jens Wilkinson
From: hon...@googlegroups.com <hon...@googlegroups.com> On Behalf Of JON JOHANNING
- Japanese is definitely problematic; how much longer it will be is hard to say.
Forever, would be my guess. Since Google translate still does the following.
OJさんは妻らを殺したとされました。
OJ was alleged to have killed his wives.
Yes, that’s really difficult, because if you don’t know the context you might get it wrong. In fact I asked a couple of Japanese people who didn’t know who it was about and they guessed it involved bigamy. AI is still very bad at understanding context.
Herman
>
>> On Dec 12, 2019, at 4:16, dav...@gol.com wrote:
>>
>>
>>
>> Of *JON JOHANNING
>>
>> * Japanese is definitely problematic; how much longer it will be is
>>
>> Forever, would be my guess. Since Google translate still does the
>> following.
>>
>> OJさんは妻らを殺したとされました。
>>
>> OJ was alleged to have killed his wives.
>>
>>
> Yes, that’s really difficult, because if you don’t know the context you
> might get it wrong. In fact I asked a couple of Japanese people who
> didn’t know who it was about and they guessed it involved bigamy. AI is
> still very bad at understanding context.
>
suggest that the AI got it right in this case. My sense is that nearly
all instances of "妻らを" that may be found in the corpus would be intended
to signify "wife and others", but in the specific case above, the
wording 「OJさんは...とされました」would seem to imply a certain specificity
that would be incompatible with reading 妻ら as "his wife and other
unspecified persons", thus leading to the interpretation "wives".
I would not say that AI is bad at understanding context, though.
Strictly speaking, AI doesn't really understand anything, but if one
takes understanding context to mean translating the same term
differently depending on the context, then that is precisely what AI
based MT systems tend to do. As an example, from Google Translate:
OJさんは妻らない = OJ doesn't have a wife
PKさんは妻らない = PK is not wife
Herman Kahn
dav...@gol.com
It’s not that “AI is bad at understanding context” it’s that current AI explicitly refuses to even try to work on what it would mean to understand context.
The recent book “Bootstrapping AI” discusses this, and talks about what AI needs to be doing. My take is far more jaundiced than said book: the things AI is doing now can’t, in principle, do what they are claiming/trying to do, and the things that would make AI possible (common sense reasoning) are much harder than the book’s authors think.
David J. Littleboy
Tokyo, Japan
--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/6F833150-74C3-4615-9E7E-06AB8A9F28E2%40gmail.com.