DARPA wants GALE to do your job
Bruce Talbot
massive new MT program, GALE (Global Autonomous Language Exploitation).
Their ultimate goal is 95% accuracy, leaving that final 5% to us human
translators, I suppose. Question is, if it becomes super-cheap to get 95%
accuracy, will there be such a big boost in translation volume that we HTs
can still find plenty of work fixing the funny 5%?
http://arstechnica.com/news.ars/post/20061109-8186.html
The article ends with a bit of encouragement:
approaching 95 percent accuracy—even for casual discussion in noisy
environments—remains a huge challenge. DARPA has a stated goal of
"eliminating the need for linguists and analysts," but that day may still be
years away.
Bruce Talbot
Exploiting my language since 1980
James Sparks
> Their ultimate goal is 95% accuracy,
I've always wondered what such figures really mean. Does the above
mean that one out of twenty words is translated incorrectly? Or that
all of the words are translated 95% correctly (whatever that would imply)?
James Sparks @ not worried, but wouldn't enter this profession now at age 25
Steven P. Venti
> Bruce Talbot wrote:
>> Their ultimate goal is 95% accuracy,
> I've always wondered what such figures really mean.
A longer but only marginally more informative article on this subject is
available at <http://www.msnbc.msn.com/id/15547807/>.
The MSNBC article mentions how that figure was determined, saying that
"DARPA judges scored the computer translations by counting the number of
human edits that the sentences needed in order for them to have the
correct meaning."
It makes no mention, however, of what method was used to determine how
many "non-edits" there were, how a percentage might otherwise be
calculated, or even--pray tell--what a "100% accurate" translation might
be. Doh.
------------------------------------------------------------------------
Steve Venti
spv...@bhk-limited.com
No electrons have been harmed, truths distorted, nor personalities
defamed in the posting of this message.
------------------------------------------------------------------------
James Sparks
> A longer but only marginally more informative article on this subject is
> available at <http://www.msnbc.msn.com/id/15547807/>.
Thanks for that.
> It makes no mention, however, of what method was used to determine how
> many "non-edits" there were, how a percentage might otherwise be
> calculated, or even--pray tell--what a "100% accurate" translation might
> be.
Yes, and it even says this:
"GALE's goal is to deliver, by 2010, software that can almost instantly
translate Arabic and Mandarin Chinese with 90 to 95 percent accuracy.
That might be impossible. Humans might not even be that precise."
So, I guess the translations you and I are pumping out are less than
90% accurate. <g>
I think these numbers are yet another example of Mark Twain's three
kinds of lies: "lies, damned lies, and statistics."
James Sparks @ back to my inaccurate work
Karen Sandness
On Nov 10, 2006, at 1:47 PM, Bruce Talbot wrote:
> approaching 95 percent accuracy―even for casual discussion in noisy
> environments―remains a huge challenge. DARPA has a stated goal of
> "eliminating the need for linguists and analysts," but that day may
> still be
> years away.
The idea of saving money by eliminating job categories or reducing
their scope is in vogue now. You can see it in hospitals where more and
more nursing tasks are given to non-R.N.'s or in supermarkets where
they are trying to phase in automated checkout lines or in banks where
the ATM has replaced most of the functions of the teller.
Back in the early 1960s, there was even talk of eliminating school
teachers through the use of "teaching machines," which were really
automated drilling programs. They worked something like this:
"Japan is an island country located off the coast of northeast Asia.
Its capital is Tokyo."
1. Where is Japan located?
a. Near Mexico
b. In Europe
c. Off the coast of Northeast Asia
d. In the United States.
2. What is its capital?
a. Tokyo
b. New York
c. Canberra
d. Paris"
I'm exaggerating only slightly here. The drills were just that
simple-minded.
Eventually, even the proponents were forced to admit that teaching was
much more multi-dimensional than they thought, not to mention the fact
that children hated the teaching machines even more than they hated any
individual teacher.
It's odd, but I've never seen proposals to eliminate DARPA researchers
(it must be terribly expensive to write and debug the code needed for
a workable J<>E translation system) or hospital administrators or
supermarket managers, or school administrators, just the employees who
actually do the hands-on work.
Anyway, I think it's going to be an awfully long time before the AI
types can produce publishable J<>E translation. Potential clients will
soon wise up to the fact that computers don't try to ascertain the
writer's intentions and can't handle ambiguity very well, and that the
result is a process even more expensive and crazy-making than the
earlier practice of using native English rewriters to patch up the
dictionary-dependent prose of 英米文学 graduates.
Head-shakingly yours,
Karen Sandness
Steven P. Venti
> The idea of saving money by eliminating job categories or reducing
> their scope is in vogue now.
That is part of it, too, but isn't much of the impetus for this kind of
research coming from the fact that certain monolinguals have suddenly
deemed it important to keep an eye on the flow of information available
in Arabic, Chinese, and other languages for which there are relatively
few Americans who are---How can I say this diplomatically?---not only
multilingual but also fit the right educational and security profiles?
BTW, just out of curiosity, we all know what kind of accuracy Google's
J<>E translation tool achieves, but how about translation tools in
languages that are more closely related to English. Have the
Spanish<>English or German<>English translation tools improved to the
point where these markets have been impacted?
It's easy to find articles on the Web surveying the history of machine
translation and making glib assessments such as ". . . [Although] recent
improvements of translation engines and microprocessor speed have made
machine translation decent[,] [y]ou'd never want to use a machine
translation on your company brochures or to write copy for a Web page."
<http://www.glreach.com/eng/ed/art/rep-eur20.php3>
What's harder though is to find critical assessments of the accuracy of
such systems.
-----------------------------------------------------------------
Steve Venti
spv...@bhk-limited.com
Oh, you know all the words, and you sung all the notes,
But you never quite learned the song, she sang.
--Mike Heron
-----------------------------------------------------------------
Karen Sandness
found a website that offered machine translations of articles from
European newspapers. (I lost that bookmark after a hard disk crash.)
Anyway, the translations--whether from German, French, Spanish, or
Italian-- were so bad that I could hardly get through an article
without giggling. The problems lay in overly literal translation of
idioms and in inability to make informed choices about ambiguous
phrasing. The most memorable example of not being able to process
ambiguity was reference to "the Russian space station Me" in an article
"translated" from German.
Even if a system could somehow be programmed to recognize and translate
correctly every idiom in both languages, writers would still come up
with new metaphors and similes and cultural references that would yield
nonsense when "translated" literally.
Flying planes can be dangerously yours,
Karen Sandness
David Farnsworth
Technology, to see how they translate the names of their departments (as you
know, every institution has their own official names for their departments).
Well, the web page is useless, since they have decided to be up with the
times (and save money on translators, I suppose) by having their ENTIRE WEB
PAGE machine translated.
This is an engineering college, and one would suppose that precision in
content would be important to them... (sigh)
Anyway, look up the college's web site on the Internet for some giggles.
My favorite was "original skull practice" for 独自相談会. (I have no idea how one
derives from the other...)
David Farnsworth
Tigard OR (or maybe Bull Mountain) 97224
> --
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.409 / Virus Database: 268.14.2/528 - Release Date: 11/10/2006
>
>
Alan Siegrist
> My favorite was "original skull practice" for 独自相談会. (I have no idea
> how one derives from the other...)
I had never heard of "skull practice" before, but according to St. Google, a
"skull practice" or a "skull session" involves teaching strategy to an
athletic team. I suppose it is some sort of jock slang, but it seems to have
currency at least in some parts of the US:
http://www.timesnews.net/article.php?id=3658534
| Fletcher asked if that applied to "skull practice," referring to football
| players watching films and the coach drilling X's and O's on the
| chalkboard and otherwise devising plays and devious game strategies.
|
| School board member Betty Cornett said she didn't know what "skull
| practice" is, but 4:30 p.m. means 4:30 p.m. as far as she was concerned.
I keep wondering why "skull"? Could it be because it involves the coach
beating strategy into the players' thick skulls? I still don't know, but
maybe I am following along a little.
Still, it is a huge leap from 相談会 to "skull practice" and only a poor MT
system with a flawed dictionary could come up with this.
Regards,
Alan Siegrist
Orinda, CA, USA
<AlanFS...@comcast.net>
David Farnsworth
It refers to a part of the athletic team practice where they use their heads
(brains) instead of their bodies. But, as you say, that is remarkably far
from 相談会. (heh)
David Farnsworth
Matthew Schlecht
From: "Alan Siegrist" <AlanFS...@Comcast.net>
To: <hon...@googlegroups.com>
Sent: Saturday, November 11, 2006 3:31 PM
Subject: Skull practice (was RE: DARPA wants GALE to do your job)
> David Farnsworth writes:
>
>> My favorite was "original skull practice" for 独自相談会. (I have no idea
>> how one derives from the other...)
>
> I had never heard of "skull practice" before, but according to St. Google,
> a
> "skull practice" or a "skull session" involves teaching strategy to an
> athletic team. I suppose it is some sort of jock slang, but it seems to
> have
> currency at least in some parts of the US:
>
<snip>
>
> Regards,
>
> Alan Siegrist
******************************************
In the industrial life sciences world (my former life), I have
heard/read "skull session" used as a synonym for "brainstorming session",
It is the sense of putting brains/heads/skulls together to share expertise
and solve problems.
I have never heard of "skull practice" though.
Matthew Schlecht
Mika Jz
> Still, it is a huge leap from 相談会 to "skull practice" and
> only a poor MT
> system with a flawed dictionary could come up with this.
To us translators' amusement and a sigh of relief ;-)
David Farnsworth wrote:
> It refers to a part of the athletic team practice where they
> use their heads
> (brains) instead of their bodies.
聞いたばかりの新しい言葉ですが、あと
全身を動かすことに慣れた人たちが
「グラウンドに出て」 ぶつかりながら行う練習に対比して
メンバーそれぞれの 「skull の殻の中で」
右に左にびゅんびゅん走り回りながら「一緒に練習を行う」イメージもわきま
す。
運動していた学生時代を思い出しました．．．
Mika Jarmusz ＠運動不足
天気もいいし、ちょっと外へでてみるか
Salem, Oregon USA
Jacob Dunlap
> James Sparks @ not worried, but wouldn't enter this
> profession now at age 25
Ouch.
Jacob Dunlap @ just turned 26
ericfe...@excite.co.jp
Double ouch.
Eric Fetterman @ soon to turn 24. (Although, in all seriousness, I doubt the ability of computers to ever be able to do this with any level of real functionality, considering how complicated human languages are)
Geoff Trousselot
Imagine a word processor that as you type, searches for ambiguities in
your language, and continually asks you to clarify the ambiguities, and
keeps this knowledge as embedded information in the document. I think
through such a process, it would be possible to create material that
could be translated by machine.
Geoff Trousselot
Evan Emswiler
>
> Imagine a word processor that as you type, searches for ambiguities in
> your language, and continually asks you to clarify the ambiguities, and
> keeps this knowledge as embedded information in the document. I think
> through such a process, it would be possible to create material that
> could be translated by machine.
>
> Geoff Trousselot
I could imagine this working for certain specialized applications,
like government documents or patents (things that you know will be
translated and that have a set style and vocabulary), but the general
public is never going to put up with such a thing, so its uses would
seem rather limited.
I'm not too concerned that computers will become as good as human
translators in my lifetime. I'm am, however, slightly concerned that
people will start to think that machine translation is "good enough"
and question the value of paying translators to do something that a
machine will do for free. This would not only have an effect on our
profession, it could lead to some costly misunderstandings. When
something is 95% correct, it can be difficult to tell which is the 5%
that is not.
Evan Emswiler
Evan Emswiler
> I'm am, however, slightly concerned
In case anyone was keeping score, this was the 5% of my message that
was incorrect. ;)
Evan Emswiler
Edward Lipsett
> When
> something is 95% correct, it can be difficult to tell which is the 5%
> that is not.
I wonder how many human translators achieve 95% accuracy.
--
Edward Lipsett
Intercom, Ltd.
Fukuoka, Japan
Tel: 092-712-9120
Fax: 092-712-9220
trans...@intercomltd.com
http://www.intercomltd.com
Malcolm James
> I wonder how many human translators achieve 95% accuracy.
By eliminating ambiguity? <g>
Malcolm
________________________________________________
Malcolm James
Fontaine Limited, Kyoto
Japanese to English translation by native speakers
web: http://www.translation.co.jp
Steven P. Venti
> I wonder how many human translators achieve 95% accuracy.
I wonder how many can *define* accuracy, as it relates to translation,
of course.
I can tell you what *I* think it is with regard to J>E translation, and
I can even give a rudimentary description of how I might go about
quantifying translation accuracy in a J>E translation. But I can also
well imagine that not all translators would agree with me and that not
everything I might include in that definition would necessarily apply to
translation accuracy in directions other than J>E.
Mark Spahn
Edward Lipsett wrote:
> I wonder how many human translators achieve 95% accuracy.
I wonder how many can *define* accuracy, as it relates to translation,
of course.
I can tell you what *I* think it is with regard to J>E translation, and
I can even give a rudimentary description of how I might go about
quantifying translation accuracy in a J>E translation. But I can also
well imagine that not all translators would agree with me and that not
everything I might include in that definition would necessarily apply to
translation accuracy in directions other than J>E.
------------------------------------------------------------------------
Steve Venti
Edward Lipsett
>> I wonder how many human translators achieve 95% accuracy.
>
> I wonder how many can *define* accuracy, as it relates to translation,
> of course.
While a description of accuracy would be nice (and no doubt trigger a flame
war, in passing...), the point is that any condemnation of machine
translation because it produces error is sort of strange.
Sure, machine translation will produce error, and IMHO it currently produces
more error than professional translators, but I'm confident that no matter
how you define translation accuracy there are an awful lot of translators
with accuracy rates under 95%.
Evan Emswiler
>
> I wonder how many human translators achieve 95% accuracy.
Which of course brings us back to the question of what it means to be
"accurate." (And I was just repeating the numbers used in the original
article -- I don't actually think the accuracy of a translation can be
represented as a percentage.) Of course, at the moment, machine
translation of Japanese can't even produce grammatically correct
English in most cases, much less English that accurately reflects the
content of the original, so I think they have a ways to go.
But even if human and machine translation had similar rates of
accuracy, I can't help but think that a human translator would be more
like to be accurate where it really counts. A human being can tell
what part of a sentence or document is the most important and focus on
that, while a machine (at least for the foreseeable future) cannot.
Evan Emswiler
Minoru Mochizuki
Research Projects Agency, I presume) of DoD is to detect
communications between hostile parties, say, terrorists.
If my guess is correct, the object and the technique they use for
their purpose are far different from those of us, regular human
translators. We human translators seek ultimate accuracies in
terms and contexts in the process of translation (although it often
seems as difficult as DARPA's effort in some cases).
For DARPA, anything that is close to potential threat to the
society is their target.
I remember the time when I was learning the advanced theory of
automatic control in one of the classes I attended as a graduate
student. One of the chapters of the book we were reading was
about detecting enemy flying objects from the signal containing
white noise the radar feeds back. Perhaps, DARPA's effort is
something to similar as they try to comb through zillions of telephone
conversations, email messages, and published materials.
Let's not waste our time comparing the two entirely different
efforts. DARPA's effort is not really translation per se.
Minoru Mochizuki
----- Original Message -----
From: "Bruce Talbot" <talb...@verizon.net>
To: <hon...@googlegroups.com>
Steven P. Venti
> Please do go ahead and define how you would quantify J>E translation
> accuracy.
It's very simple. You show the translation together with the original to
any number of other translators, then ask them if the translation was
accurate or not. Divide the number of those who responded affirmatively
by the number of all who were asked.
Although the real point of my earlier post was just to prompt Ed to
finish his thought, my personal opinion on this matter is that
"translation accuracy" is a misnomer. Translations are not measurements;
they are replications. Thus, it makes a lot more sense to me to talk
about "translation fidelity" than about "translation accuracy." In fact,
since it doesn't really make any sense to me to try to quantify "audio
fidelity," I can't really wrap my head around quantifying "translation
fidelity" either, especially since not all translations are prepared to
the same "specifications"---something that the "Bureaucracy gone mad?"
thread illustrates quite nicely.
I think that there are two aspects of translation fidelity, however, for
which quantities can be approximated. Language-driven fidelity can be
evaluated for its "literalness" and information-driven fidelity can be
evaluated for its "completeness." Since no translation is ever purely
one or the other, however, and since not even the two together form a
complete picture of overall translation fidelity, they are not
particularly useful for quantifying translation fidelity.
Probably more than you wanted to know, <g> but there it is.
-----------------------------------------------------------------
Steve Venti
spv...@bhk-limited.com
Tourists talking 'bout the madhouse, talking 'bout the ear,
The madman hangs in fancy homes they wouldn't let him near,
He'd piss in their fireplace,
He'd drag them through Turbulent Indigo. --J. Mitchell
-----------------------------------------------------------------
Karen Sandness
something that could probably be quantified, but also producing a new
text that will not either puzzle native speakers of the target language
or give them the giggles.
Recently another JAT Board member sent a photograph of a hotel sign
that translated お泊まりのお客様以外の方の入室は固くお断りしております as "I will refuse entering
a room in one other than the staying customer hard." The "translator"
got most of the individual words right, but a monolingual English
speaker seeing that sign would be baffled by the "translation."
How would you quantify its "accuracy"?
Literally yours,
Karen Sandness
Evan Emswiler
>
> Mark Spahn wrote:
>
> > Please do go ahead and define how you would quantify J>E translation
> > accuracy.
>
> It's very simple. You show the translation together with the original to
> any number of other translators, then ask them if the translation was
> accurate or not. Divide the number of those who responded affirmatively
> by the number of all who were asked.
Would you do this on a sentence-by-sentence basis? Or would you just
ask the translators if an entire manuscript was accurate? It seems
like mistake, or even a difference of opinion about an ambiguity,
could technically render an entire manuscript "inaccurate."
Evan Emswiler
Evan Emswiler
> It seems like mistake
Make that *one* mistake (like this one).
Evan Emswiler
Steven P. Venti
> Steven P. Venti <spv...@bhk-limited.com> wrote:
>> Divide the number of those who responded affirmatively
>> by the number of all who were asked.
Evan Emswiler wrote:
> Would you do this on a sentence-by-sentence basis?
Hmm, I had thought that the last two-thirds of that post pretty much
clarified the fact that the first paragraph should be taken as tongue in
cheek.
Or are you just jerking my chain? <g>
Evan Emswiler
>
> Hmm, I had thought that the last two-thirds of that post pretty much
> clarified the fact that the first paragraph should be taken as tongue in
> cheek.
>
> Or are you just jerking my chain? <g>
Well, I was trying to point out that it was a dumb idea, but I guess
it was supposed to be, so never mind. <g>
Evan Emswiler
Peter Durfee
> I am guessing the main objective of DARPA (Defense Advanced
> Research Projects Agency, I presume) of DoD is to detect
> communications between hostile parties, say, terrorists.
This is true. The end goal of DARPA is to replace *all* military and
intelligence linguists and analysts with computers, though--and if we
believe that to be possible we're certainly believing that a computer will
show up that can do all of our jobs too.
I'm not believing that, personally.
The recent IHT article covering this effort has a definition of "accuracy,"
by the way, for those keeping score at home:
DARPA judges scored the computer translations by counting the
number of human edits that the sentences needed in order for them
to have the correct meaning. By this measure, the results largely met
DARPA's demands of 75 percent accuracy for text translation and 65
percent for speech.
I still don't understand how you can count the number of edits and get a
percentage value out of that, but FWIW.
--
Peter Durfee
du...@gol.com
Tokyo
Peter Durfee
> The recent IHT article covering this effort
And then I didn't give a link to the article. Here you go:
http://www.iht.com/articles/ap/2006/11/07/business/NA_FIN_US_Pentagon_Transl
ation_Challenge.php?page=1
James Sparks
> DARPA judges scored the computer translations by counting the
> number of human edits that the sentences needed in order for them
> to have the correct meaning. By this measure, the results largely met
> DARPA's demands of 75 percent accuracy for text translation and 65
> percent for speech.
>
> I still don't understand how you can count the number of edits and get a
> percentage value out of that, but FWIW.
Not to mention that editing itself is highly subjective, and ten
editors of a long translation will almost certainly give ten different
scores to the same translation. Furthermore, all edits are not of the
same importance. Say one translator mistranslates アンチモン as
ammonia, while another translates ディスク装置 as "disk apparatus"
rather than the more standard "disk device." Each translation has one
edit, but they are far from equal in quality.
I think everyone on this list would probably agree that quantifying the
"accuracy" of a translation is very difficult at best, and completely
misleading at worst. That may be one of the few things we can all agree
on. <g>
James Sparks