Translation by GPT-4 vs. Claude

[Tom Gally]

I was asked yesterday to translate a 19,000字 document into English, a set of guidelines to be distributed to students at the university where I am still employed part-time. The content is not particularly specialized, though the document contains some organization-specific terms and mildly complex formatting (headings, subheadings, a glossary of terms, URLs, etc.).

I asked both OpenAI’s GPT-4 and Anthropic’s Claude to translate the text into English. The prompts I tried were variations on “You are a skilled and expert translator of Japanese to English. Your translations are both faithful to the original and read smoothly and natural in English. I have attached a text file containing a text in Japanese. This document is [explanation of document's content and purpose]. Please translate this document into English."

To my surprise, I couldn't get GPT-4 to produce a sentence-by-sentence translation. Despite repeated attempts in both the regular web interface and the Playground, it would produce only summaries or partial translations.

ln contrast, Claude translated the entire document with no omissions. The English reads naturally, and it nailed a lot of organization-specific terminology. For example, it translated 教養学部 as "College of Arts and Sciences," which is the official English name at this particular university, rather than "Faculty of Liberal Arts," which both GPT-4 and Claude have given me in the past. It even knew the official English names of three dorms, which I would have had to check myself.

I will still need to check and polish Claude's translation, but having its version as a first draft will save me a huge amount of time.

Just for comparison, I also ran the text through DeepL. While its draft would have been usable as a starting point, DeepL is much sloppier than Claude and it continues to be unable to grasp the overall context of the document. While Claude translates 本学 consistently as the name of this particular university, DeepL uses either "this university" or, in a couple of cases, the name of another Japanese university.

Tom Gally

[Richard Sadowsky]

Thanks for the reminder about Claude (Claude 2 now), Tom! It's now
available in Japan, so you don't need to VPN into the U.S. I've just
tried it on some material, and it looks very promising.

Richard S.

> --
> You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/CACnS3Nf-UUEVSUygKBn1u7FV4gMDrL5Zb3eJfvunY8Je2yYzow%40mail.gmail.com.

[Tom Gally]

I should note that, as I actually started polishing that 19,000字 document that I had had Claude translate for me, I discovered about half way through that it had skipped a few sentences. The reason might be its decreasing accuracy after the middle of long prompts, as described in this AI Explained video:

https://www.youtube.com/watch?v=Auu8tGaTaNQ&t=816s

Tom Gally

[Brendan Craine]

Thank you, Tom!

That's a pretty incredible jump in pace for most translators, even with the added post-editing.

As ever, your reports fill me with a mixture of interest and creeping anxiety.

--Brendan

2023年11月16日(木) 19:30 Tom Gally <tomg...@gmail.com>:

--

You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/CACnS3Nf-UUEVSUygKBn1u7FV4gMDrL5Zb3eJfvunY8Je2yYzow%40mail.gmail.com.

--

=============================================
チーフエディター『エクスポート・ジャパン株式会社』
ブレンダン・クレイン （Brendan Craine）
email: brendan@export-japan.co.jp

[Tom Gally]

Here’s a post-mortem on my translation of that 19,000字 document—guidelines for students at a university—using GPT-4 and Claude.

As I mentioned previously, GPT-4 wouldn’t translate the document as a whole, so I used Claude’s translation as my first draft. As I reported later, there were a few omitted sentences in the middle of Claude’s translation, but overall it was good enough to use as my starting point.

When I translate a document by myself, my workflow normally looks something like this:

(1) Read the entire document (or the first part, if the document is long) to get an idea about its content.

(2) Translate straight through from beginning to end, looking up terminology and the like as I go along.

(3) Do one cross-check of my entire first draft against the original to make sure I got the meaning right and that I didn’t omit anything.

(4) Do two or three readings of my translation, looking for typos, omitted words, and places where the wording can be improved. Only when I am uncertain about a particular section do I go back and check the original Japanese again.

This time, I let Claude do step (2), which is the most time-consuming and tiring for me. Because I couldn't trust its terminology completely (though it did pretty well), I modified step (3) as follows:

(3') Do two cross-checks of the entire first draft against the original to make sure Claude got the meaning right and didn’t omit anything. When Claude did omit something or I was unhappy with its translation and couldn't immediately think how best to fix it, I pasted the entire paragraph into GPT-4 and asked for its translation. I kept a browser tab open with GPT-4 for the whole time I was working on the document. The first time I asked for a partial translation, I included a long prompt explaining what the document was and how I wanted it to be translated. When I asked it later to translate another section, I would just say “Here is another section from the same document” and it was able to maintain the context and tone in its translation.

If I was having difficulty thinking of a translation myself because of a complicated sentence structure or inability to think of English phrasing that satisfied me, I would ask GPT-4 for five translations of that same passage. It almost always came up with something I liked.

Step (4) was the same as usual.

My guess is that I finished the job in about 70% of the time I would normally need for such a translation, mainly because I outsourced step (2) to the AI.

My self-assessment is that the finished translation was slightly better than what I could have produced on my own. The main reason is that Claude and GPT-4 sometimes come up with phrasings that I recognize as good but wouldn’t have thought of myself. Every human translator (probably) develops something of a personal style and usually translates certain expressions in the same way even when other wordings would be equally good or even better. Asking the AI for ideas—especially multiple ways to translate the same passage—helped me break out of my personal translation rut a little bit.

Another reason for the slightly better quality is that, unlike me, Claude and GPT-4 make essentially no typos in their first drafts. In my final checks in step (4), I caught fewer omitted words, agreement errors, etc. than I typically find in a document that I translate entirely on my own.

Tom Gally

[Richard Sadowsky]

Tom,

Thanks for sharing your methodology. Of course, you are both the translator and to some degree the end client, so you can use AI to your heart's content. In the business world, this might not fly.

About step (4), about catching fewer omitted words, agreement errors, etc., I find that Grammarly does a great job fixing or suggesting fixes for those errors (that you only need click on, rather than type in).

Richard Sadowsky

--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/CACnS3NfjDYb1sLHuFpnfhNNj%2BJx55GxrJS2%3DXN9ckKaJ29cwGQ%40mail.gmail.com.

[Tom Gally]

Richard S. wrote:

Of course, you are both the translator and to some degree the end client, so you can use AI to your heart's content. In the business world, this might not fly.

Good point. Though maybe a better way to describe my current situation would be that I am an in-house translator with many years of experience working within the organization, including, in past years, serving on committees that wrote, debated, and approved the sorts of documents that I am now translating. It’s a very different perspective from what I had when I was freelancing, even for direct clients.

I am also being paid a monthly salary rather than by the job. Twenty years ago, I would not have quit freelancing to take such a job, but it’s just right for me now as a post-retirement gig, especially since I can work at home.

Tom Gally