GPT-4 has been released

[Tom Gally]

As you might have read, OpenAI released GPT-4 earlier today. A couple of hours ago, I subscribed to ChatGPAT Plus so that I could try GPT-4, and I’ve started testing it with translation and other tasks.

When asked to translate a passage from a Japanese novel into English, it produced a translation that, to my eye, is as good as the published human translation of the same text—maybe better in spots, depending on your tastes.

I put the results here:

https://www.gally.net/temp/202212chatgpt/translation/fiction4.html

I didn’t include the Google Translate or DeepL versions this time, as they are so much worse than GPT-4 that there’s no point in comparing them.

Tom Gally

Yokohama, Japan

[Geoffrey Trousselot]

Perhaps you could liken this to the invention of photography and what impact it had on artists. The photorealism is the closeness of the translation and the impressive level of naturalness for such close translation. I guess the translators can't go full-blown Cubist, but maybe the skill of tweaking will still be appreciated. 

Geoffrey Trousselot

--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/CACnS3Ndzug5tKvp8QGjLU%3DQ0RpmsgJ24_NkF0zXmvpMcEc%3Dv7w%40mail.gmail.com.

[seigo]

Thanks for posting this Tom.

You've probably taken this into account but just to be sure, did you consider the likely possibility that ChatGPT Plus had access to the published human translation of this text? And tweaked that according to your prompt?

I'd be interested to know how it does translating previously untranslated work.

Regards,

Seigo

[加納寛子]

Nice to meet you all. I am a researcher in Japan and I feel the need for ChatGPT literacy. However, I am of the opinion that it should not be banned, but actively utilised. I feel that examples of good use should be spread throughout the world. That is why we organised the ChatGPT Grand Prix. Are there any researchers who would like to be a member of the panel of judges? ? If you would like to be a judge, please send me the following three information.
Name
Affiliation
Expectations for ChatGPT

ChatGPT Grand Prix Entry Form

https://forms.gle/ZzsEWyzCD9DEa5tb8

Best wishes.

=-=-=-=-=-
Hiroko KANOH
ka...@cc.yamagata-u.ac.jp
National University Corporation Yamagata University, Institute of Arts and Sciences, Associate Professor
1-4-12 Koshirakawamachi, Yamagata-shi, Yamagata, 990-8560 Japan

2023年3月16日木曜日 10:44:58 UTC+9 seigo:

[jo...@johnfry.org]

Tom, someone on Reddit made a similar observation about GPT4's impressive translation of Chinese fiction:

https://www.reddit.com/r/ChatGPT/comments/11t389v/wow_gpt4_beats_all_the_other_translators/

John Fry

[Tom Gally]

Seigo wrote:

I'd be interested to know how it does translating previously untranslated work.

Okay, I tried it. Below is what GPT-4 did with a story that was published in the magazine 新潮 last week. Both translations are very good, though the first one reads more smoothly me. 

https://www.gally.net/temp/202212chatgpt/translation/fiction5.html

Some of the “Japanesey” things, like 年越し蕎麦 and 高校の美術部の後輩, are translated a bit too literally for my tastes, but many human translators would make similar choices.

I produced the text using OCR, so there might be some mistakes in the Japanese. Also, in the printed magazine the name of the character 虫生 is given in furigana as むしゅう. I converted that to 《むしゅう》 but I put it after the さん by mistake. That might have confused GPT-4 in the first version.

On the question of whether the translation of 門 might have copied from a previous translation: Yes, that’s possible, but I doubt it.

When ChatGPT was initially released last year, my first thought when I saw that it could produce coherent, well-organized essays from a simple prompt was that it was pasting together text from the Web. After subsequent testing by me and many others, though, I don't think any evidence of that has emerged. I suspect that the translation of 門 was original as well, too. However, I did tell it the title and author of the work, and it might have drawn on summaries or other information about the novel in English or Japanese—which, if true, would be impressive in its own way.

For the story from the most recent issue of 新潮, it must be translating from scratch.

Thanks to everyone for their comments.

Tom Gally

Yokohama, Japan

[Tom Gally]

Additional comments:

Obviously more testing is necessary, but my impression at this point is that GPT-4 is good enough at translation from Japanese to English to be able to produce translations of literary texts that could be published and sold commercially. Only light pre-editing, appropriate prompting, and light post-editing would be necessary, and the burden on human editors might be less than with a human-produced translation.

Last night, I tested it on an essay that I had translated for a book published a couple of years ago by Iwanami Shoten. I hate to say it, but its translation was just as good as mine.

Tom Gally

[Kevin Johnson]

Hi Tom,

Thanks for your persistence on this issue. I can tell you're really interested in this, although sometimes it feels like you started with the conclusion rather than working your way there scientifically.

I have way too much translation work in March (年度末) to spend a ton of time analyzing this translation, but I had a couple of thoughts about the part I read.

I want to be clear that I love the idea of AI creating a universal translator like Star Trek and indeed replacing all work. I think it's fascinating. I'll survive no matter what, unless I guess I don't survive, which would be bad.

Anyway, first of all, your prompt engineering feels a little weird to me. If the technology were to be used to translate a novel, for example, it would not be possible to tell it how to fix specific errors in the prompt like you've done here. It would be on the translator to figure out the gender based on the contextual clues. If ChatGPT can't do that, it's already starting at a big disadvantage. Not a good start.

Second, I stopped when I got to this passage because it's so bad:

"My brother, who lives in Hiroshima, came over with his wife and niece, and my younger brother also showed up with his live-in girlfriend and some year-end soba noodles, making the house even more lively than usual."

Compare to the original:

広島に住んでいる弟と、義妹と姪も顔を見せに来ていて、下の弟も同棲中の彼女を連れて年越し蕎麦を持って現れ、いつもよりさらに賑やかでした。

Imagine someone told you, "This is my brother and his wife and niece and my younger brother and his live-in girlfriend." You may reasonably respond, "I have no clue what is going on in this family" and you'd be right, but this is actually how ChatGPT decided to phrase it.

Both brothers are younger than the speaker, but the English translation implies one is older. The translation says he came over with "his wife and niece," which is wrong and jarring. He came over with his wife and daughter, or alternatively with "my sister-in-law and niece," which is actually the rendering I prefer because it maintains the author's choice of presenting everyone in terms of their relationship to herself.

"Year-end soba noodles" is just weird to an English reader with no cultural knowledge and completely dodges the harder translator decision of how to present it such that it's not confusing. As an aside, I would never say "live-in girlfriend" (sounds rude, like "live-in maid") and have never heard that phrase before in my life, but apparently other people use it, so we'll give it a pass to be a good sport. 

I scanned the rest and found the English stilted, wrong in parts, dropping key information in other parts. 雪ちゃんらしい is missing. "...who continued to draw even as an adult" sound like a statement of fact rather than (correctly) the author's inference drawn from the illustration. "Bared lips" is hilarious but it's not "retracted lips" and doesn't convey that the gums were showing (gummy smile). "...then shifted my gaze to the next card and involuntarily held my breath," oh brother. Most importantly, it was boring.

「絶対に連絡しろ」という無言の圧力を強く感じました。

...making me feel the silent pressure of "be sure to contact me." 

This is emotive and a bit funny in Japanese, but in English it's just blade and lame. It lacks any kind of punch.

I would suggest that something being grammatically coherent does not make it a good literary translation. Literary translators are often PhDs or novelists in their own right.

Now, this is an amazing technology and I think it will change the world. But if someone asked me to "lightly post-edit" this prompt-engineered monstrosity into a well-written story for a reduced rate, I'm afraid I'd "bare my lips" and refuse.

Anyway, I've managed to spend two hours on this, which is time I definitely don't have in March, so I have to get back to work! Please keep up the fun posts.

Kevin Johnson 

[Fred Uleman]

What happens if you give it a passage to translate today and then give it the same passage to translate tomorrow? Does it realize it already did this and just give you the same translation the second time or does it go back and unthinkingly* do the translation again with different results?

* yes, I know the machine is not "thinking" in any case. It's an expression. Indulge me.

- -- --- ---- --- -- -

Fred Uleman

[Warren Smith]

I got a note today from a major patent translation client of mine, a client that has been worth over $700,000 to me in the last 6 years (obviously accounting for a substantial part of my income).

 

"Is your rate open for discussion? For majority of our projects, we apply Machine Translation and Translation Memory, which help us to improve the translation efficiency and reduce the cost to 0.07 - 0.08 USD per EN new word [in Trados]. May I ask if it is acceptable for you?" [Note the bad English in the email from the client.]

 

The answer was, "No. I will not accept work at that rate." I sent them my journal article that talks about some of the legal issues with machine translation and translation memories (linked here), but I don't think they will read it.

 

The firm has recently undergone a change in management, but is in the top 10 largest language service providers in the world (joining TransPerfect in the race to the bottom, I think).

 

I don't know about everybody else, but I am aggressively working to leave the industry. Translation has been very good to me, but it is time to move on, I think.

 

Warren Smith

 

 

[Tom Gally]

Thank you, Kevin, for the thoughtful comments.

There’s so much to discuss about the implications of GPT-4 for translation and I am sure many others are interested, too, so I have decided to organize an informal online discussion about it. Here are the details:

Title: The Implications for Translation and Translators of Recent Advances in Large Language Models

Date and time: 10:00 a.m. to 12:00 noon Japan time, Friday, March 31

Format: Zoom

Language of discussion: English

Main language pair considered: Japanese <---> English

I will prepare a sign-up form and some more details later today or tomorrow and post the information here. Anyone interested will be welcome to participate. Speakers will be asked to use their real names. The session will be recorded and posted to YouTube later for people who cannot attend in real time.

Regarding Fred’s question:

What happens if you give it a passage to translate today and then give it the same passage to translate tomorrow? Does it realize it already did this and just give you the same translation the second time or does it go back and unthinkingly* do the translation again with different results?

There’s some randomness in the output of LMMs—the technical term is “temperature”—and they usually give a somewhat different response each time to the same prompt. At present, OpenAI’s GPTs do not have long-term memory about what they have done before. I assume such memory will be coming in the not-too-distant future, because one can imagine many applications where it would be useful.

There’s also a limit to the length of the prompts they will accept. I tried asking GPT-4 to translate a ten-page essay all at once, but it refused because of the length. I then had to break it up into pieces, and there were noticeable problems with cohesion because the later parts were translated without reference to the earlier parts. I assume this problem will also be resolved at some point, too.

Tom Gally

--

You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/758B6BC4BE3A4448B18E12E0FD96EC3E%40WarrenSmithDell.

[Dan Lucas]

The firm has recently undergone a change in management, but is in the top 10 largest language service providers in the world (joining TransPerfect in the race to the bottom, I think).

This is the bit that caught my attention. It is inevitable that such companies are going to be in the forefront of cost-cutting efforts. Scale (higher volume of business) comes at a cost to the LSP (lower prices), and the LSP must offset that by systematically working to reduce their supplier costs (freelancer rates). 

None of us is so deeply into the "isolated-freelancer-who-hasn't-worked-for-a-company-in-decades, utterly-divorced-from-the-hurly-burly-of-commerce" mentality on this mailing list that we don't realize that this is entirely normal, right?

Anyway, you (Warren) declined, which was the right thing to do. Others don't, because they wrongly perceive these companies to be the best people with which to do business, regardless of the terms. And so it goes.

Last week, a representative of one of the top three LSPs contacted me to talk to me about "freelance opportunities". I agreed to talk on condition that (a) they confirm before we go any further that they will accept rate X, which is the minimum I charge my existing clients, and (b) that they have a job waiting. And that was that - end of dialogue. There was no job. They were not willing to pay. It was just an attempt to boost some or other KPI of theirs.

That's fine by me. Like Kevin Johnson elsewhere in this thread I am busy this month and April/May are going to be frantic. A couple of clients are making noises about June/July and beyond that I have just taken a booking for August (!). None of my clients in Japan and Europe feature in the top 20, but they're good customers. I support these clients rather than their top 20 rivals, and they send me regular work. Demand is growing, not in a straight line, but steadily over the long term.

Maybe in five years the industry will have been transformed, and I'll be going cap in hand to anybody who will give me a job, but I kind of doubt it. In the back of my mind is my experience when I started out in 2015. The feedback then was nearly all negative (CAT tools! Plummeting rates! MT! Plagues of locusts darkening the skies!). It hasn't stopped me or others earning a good living during the intervening seven years.

Now I'm seeing similar wall-to-wall negativity, the very uniformity of which makes me think the fear is overblown. I am not a reflexively contrarian person, but a consensus this strong is unlikely to be correct.

In the financial markets there are always people who take the negative view, who see every fluctuation as a secular downturn rather than a cyclical movement. "That guy," we used to chuckle to each other in front of the Bloomberg "has predicted nine of the past two market crashes." Well, some people have predicted nine out of the past two translation industry implosions.

This industry is going to change and keep changing as technology evolves, like every other industry. Nowhere is safe. There will be threats and opportunities. Some participants will be agile enough to keep up, and for some with other options the game will no longer be worth the candle. I think the job I do will today have changed in five years but I'll still be earning money, and that's about all I can say. Maybe it's just because I'm one of these internal locus of control people.

But, as a final point, if anybody is confident in the strength of their views, I'd be happy to see them put their money where their mouth is. If you've got skin in the game, feel free to make a public prediction for the future of this industry here or on a new thread.

• Clearly specify your metric of choice
  
• Forecast a verifiable value
  
• Give us a precise end date
  

Let's see how you do. Let's see if your certitude is justified.

(Warren - this last is not aimed at you! I just glommed onto your comment about the big LSP as a starting point. And you are putting your money where your mouth is by investing in a new profession, which seems entirely reasonable to me.)

Anyway. Busy. Need to get back to work. Have a good weekend everybody.

Regards,

Dan Lucas

PS And yes, of course everybody should have a Plan B...

[Jon Johanning]

I'm currently taking Corinne's Training for Translators course "March Marketing Madness." It's giving me lots of ideas for finding clients and agencies, more than I can work on right now, but I have plenty to try out later. What's clear to me, anyway, is that that there are many ways to find clients that I never thought of, and I suspect that is true of others, too.

Probably, one can stay in the profession despite the competition from all these new silicon-based doohickeys as long as one keeps trying new marketing methods. (Can chatbots tell us better marketing methods? Perhaps.) Why I see happening is a long continuation of the trend of customers for translation who can settle for very fast and cheap but not-so-great work, which has been going on as long as computers have been in the game, so that a lot of work that we used to do (and I recall fondly the lazy, hazy days of typewriters, snail mail, and even passing source texts and translations hand-to-hand) has disappeared. So we will have to spend a lot more time beating the bushes for clients and agencies (technically speaking, "marketing"). But these doohickeys may give us more time to do that.

A note on Tom's very informative posts on his AI experiments. I don't do fiction translation, so they don't relate precisely to my efforts. But it strikes me that the judgments about whether "silicon translations," I call them, are better or worse than human ones involve aesthetic considerations, which don't enter into technical translation so much: patents, medical stuff, and other dry subjects where extreme accuracy is the main factor. What I wonder about, where these fields are concerned. is whether the doohickeys make so many mistakes (and the proponents of the silicon gadgets admit that they get a lot of stuff wrong, or even "hallucinate") that post-editing their products will take lots of effort, and that may be where a lot of the work for us humans will come from in the brave new world. In other words, is there a lot of cherry-picking going on when we hear about all of the amazing new chatbot translations?

Best,

Jon Johanning

[Jon Johanning]

Sorry for all the mistypings, but I hope what I wrote is clear.

[Warren Smith]

This is useful, Jon. Thanks for posting. Where do we find this course by Corinne?

 

Also, about the machine translation of the novel chapter... I thought the silicon translation was pretty good, but my wife (an author) hated it. She *much* preferred the human translation, and felt that sentence structures were much more artistic, and that the prose flowed more smoothly. (I am not artistic enough to tell, personally.)

 

I have done some experiments with patent translation, and I am not hugely impressed. Yes, the translations by GPT-4 are better than I often see by some low-level J-E translators, but certainly not on par with a real professional. The problems are minor, actually, but there are odd choices of words that would be made by a greenhorn translator (such as "manuscript" instead of "document" in a sheet feeder for a copy machine), lots of problems with singular vs. plural, etc.

 

Constancy of word choices (data permanency) is an issue as well. For example, in the test I ran what was translated as "manuscript" in one spot was later on translated as "original" in, for example: "sheet-through type original feeding device (hereinafter referred to as ADF)" (シートスルー式の原稿送り装置 (以下、ADFともいう) ).  This data permanency issue is exacerbated because, at least at present, only fairly short hunks of a document can be translated in a single question. Thus consistency (which is critically important in patent translation) is currently a problem with GPT-4. (For what it's worth, it is because of these consistency issues that I will never use shared translation memories -- different translators make different choices (e.g., "manuscript," vs. "original," vs "document"), which then have to be reconciled when sentences show up in shared translation memory, but (as far as I have seen) there is no mechanism to compensate a translator for fixing the poor/inconsistent choices of earlier translators. The skilled translator spends more time fixing machine translations/shared-memory translations than it would have taken to translate from scratch, while having to subvert one's own style and preferred word choices to match the other's translations. This mixing of one's own translations with those of others produces errors. What a nightmare!)

 

On the other hand, machine translation and shared memories can be very efficient if you don't worry about quality... This produces huge financial incentives biasing low-quality translators to accept the outputs of machine translations, largely without fixing them. If your language skills are not good enough to recognize problems, you earn MORE than skilled/conscientious  translators who have to spend more time fixing problems. [This is not the same as with straight translation, where less experienced translators are often slower than more experienced translators, with problems that are caught by project managers quickly because they tend to be more obvious than the problems with machine translations that are only apparent to skilled translators...]

 

One would hope that the market would quickly screen out low-quality machine translations. But, alas, this is unlikely. The problem is that, just as with skilled and less-skilled human translators. many clients aren't sophisticated enough to know the difference! It alarms me that many of the top language service providers in the world choose to act as unsophisticated consumers of translation themselves, pretending that they can't tell the difference between low and high quality, thereby making it possible to buy cut-rate translations and send them on to their clients.

 

Shameful (but lucrative). One such firm (which I will not name because it loves to sue people!) has become a multibillion dollar company. Shameful!

 

W

 

---

To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/2a1f3ab8-e1aa-434e-95dd-15db016d31d0n%40googlegroups.com.

[Jon Johanning]

Corinne’s course is at https://trainingfortranslators.thinkific.com/users/sign_in, but I don’t think it can be joined midway. Her general website is https://www.trainingfortranslators.com/.

I have done some experiments with patent translation, and I am not hugely impressed. Yes, the translations by GPT-4 are betterthan I often see by some low-level J-E translators, but certainly not on par with a real professional. The problems are minor, actually, but there are odd choices of words that would be made by a greenhorn translator (such as "manuscript" instead of "document" in a sheet feeder for a copy machine), lots of problems with singular vs. plural, etc. 

 

Constancy of word choices (data permanency) is an issue as well. For example, in the test I ran what was translated as "manuscript" in one spot was later on translated as "original" in, for example: "sheet-through type original feeding device (hereinafter referred to as ADF)" (シートスルー式の原稿送り装置 (以下、ADFともいう) ).  This data permanency issue is exacerbated because, at least at present, only fairly short hunks of a document can be translated in a single question. Thus consistency (which is critically important in patent translation) is currently a problem with GPT-4. 

Yeah, it’s just not ready for prime time, although literary translators seem to like it.

(The skilled translator spends more time fixing machine translations/shared-memory translations than it would have taken to translate from scratch, while having to subvert one's own style and preferred word choices to match the other's translations. This mixing of one's own translations with those of others produces errors. What a nightmare!)

 

On the other hand, machine translation and shared memories can be very efficient if you don't worry about quality... This produces huge financial incentives biasing low-quality translators to accept the outputs of machine translations, largely without fixing them. If your language skills are not good enough to recognize problems, you earn MORE than skilled/conscientious  translators who have to spend more time fixing problems. 

Yes, if you hook up with clients who don’t care much about quality. But if they care very little, they can be satisfied with Google [redacted].

[This is not the same as with straight translation, where less experienced translators are often slower than more experienced translators, with problems that are caught by project managers quickly because they tend to be more obvious than the problems with machine translations that are only apparent to skilled translators...]

The PMs I encounter generally don’t know Japanese at all, so they only catch missing numbers and alphabetic letters.

 

One would hope that the market would quickly screen out low-quality machine translations. But, alas, this is unlikely. The problem is that, just as with skilled and less-skilled human translators. many clients aren't sophisticated enough to know the difference! It alarms me that many of the top language service providers in the world choose to act as unsophisticated consumers of translation themselves, pretending that they can't tell the difference between low and high quality, thereby making it possible to buy cut-rate translations and send them on to their clients.

They are unsophisticated consumers, at least for little, out-of-the-way, peculiar languages like 日本語. So I’m going to work much harder on hauling in direct clients than I have been.

Best,

Jon Johanning

[BTB]

Warren wrote:

Constancy of word choices (data permanency) is an issue as well. For example, in the test I ran what was translated as "manuscript" in one spot was later on translated as "original" in, for example: "sheet-through type original feeding device (hereinafter referred to as ADF)" (シートスルー式の原稿送り装置 (以下、ADFともいう) ).  This data permanency issue is exacerbated because, at least at present, only fairly short hunks of a document can be translated in a single question. Thus consistency (which is critically important in patent translation) is currently a problem with GPT-4. 

This is where 'prompt engineering' might come in handy. For example, you can ask it to translate a Japanese paragraph/page into English, using a list of preferred terminology, and it will (probably) stick to your style preferences. I also read somewhere where someone told GPT to refer to an external data link, and it could, but I have not experimented with this yet. Conceivably, one could set up an external glossary of terms / translation memory for GPT to link/refer to while translating something for you. 

Brian Boland

[Warren Smith]

"prompt engineering." Brilliant! That sounds like a great idea.

 

External data links with GPT-4 have not worked for me. I tried the prompt "Can you please translate JP2004021007A into English in a proper patent style?," and it gave me a page or two of some OTHER patent.

 

Warren

 

---

To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/6e38fd1f-dc84-4a60-8cde-deddea9c91bfn%40googlegroups.com.

[Dan Lucas]

Sounds very positive - there's always, always something more to learn!

Regards,

Dan Lucas

[TCM Lounge Sendai]

Dear members,

I have a question. I really wanna know about the average voice-over rates for the following 4 types:

1. Wild/Plain recording: Only one rule – to read the text. No consideration is given to length of time taken to read the text, or synchronizing the text with other media e.g., video. Audio is delivered as one file.
2. Segmented recording: Script is split into segments during the creation of Runsheet. Voice Talent reads the script in separate segments. Audio is delivered in multiple files.
3. Synchronized recording: Voice talent reads the text in sync with the on-screen action. In some cases, the Voice Talent attempts to sync the text with the lip-movements of the on-screen speaker or images on the screen if necessary. Audio is delivered as one file.
4. Minimum fee

Any input is welcome. I have checked some websites but couldn't find the average prices.

Thank you so much in advance.

Masatoshi Shoji