One small comparison of DeepL and GPT-4

[Jon Johanning]

After viewing the video of Tom’s Zoom session on AI (thanks very much, Tom!), I have done a few trials of translation using GPT-4.

 

One of them was comparing GPT-4’s translation of a 日経 article on Kishi’s plan for increasing the supply of hydrogen as a non-carbon-based fuel with DeepL’s. This turned up a couple of things that I found interesting. This is only a glimpse into the subject, of course.

 

(1) The article referred to 脱炭素燃料, which both translations called “decarbonized fuel.” I certainly wouldn’t call hydrogen “decarbonized,” and neither would any other competent human translator, since hydrogen obviously never contains carbon in the first place. This is an example of these translating machines not knowing anything about the real world, I think.

 

(2) More seriously, DeepL fouled up a mention of a year, but GPT-4 got it right. 

 

The article says:

 

骨子では30年ごろの商用での実用化を目指し、サプライチェーンの構築を政府が後押しする方針も記した。

 

GPT-4 says: “The outline sets a policy to aim for commercial practicality around 2030 and have the government support supply chain construction,” which I think is OK. But DeepL has this: “The framework also states that the government will support the establishment of a supply chain with the aim of commercializing hydrogen around 1930.”

 

Come again??!! Why did it jump back a century?

 

What happened here, perhaps, is that 日経 made the original mistake with “30年ごろ,” which should have been written “2030年ごろ,” it seems to me. 

 

Back at the beginning of the article, it mentioned the year “2040年”: “2040年に現状の6倍の1200万トン程度に増やす方向で調整する。” And again, it’s in the 21st century: “…2017年に決めた「水素基本戦略」を改定する意向を表明した。同日の会議で骨子を示した。” So the century we’re talking about is clearly established.

 

Then we see: “水素の現状の供給量は年間約200万トン。30年に300万トン、50年に2000万トンを目指す方針を掲げてきた。” “年間” is OK, but then it gets sloppy with the year numbers: “30年に300万トン、50年に2000万トン…” 

 

Apparently, GPT-4 grasped that the century under discussion is the 21st, but DeepL, for some reason, slipped back to the 20th. I think it got confused by the intervening "30年" and "50年," and for some reason made this mistake about the centuries.

 

This looks to me an example of a point that people make about the two apps, that DeepL tends to go sentence by sentence, whereas GPT-4 is able to put the sentences into their whole context. But I would suggest that, in any case, a human being needs to be paying attention in order to decide which machine translation is right. You can’t blindly trust any machine.

Jon J.

[Jon Johanning]

Correction: I guess it is OK to refer to hydrogen as a 脱炭素燃料, because it is actually made mostly from coal and natural gas. I was thinking of electrolysis, which is used to make only a small percentage of it.

Jon J.

[Jon Johanning]

2nd correction: Grey H2 is made from fossil fuels and thus is not a decarbonized gas; blue H2 is partially decarbonized, because a Co2 capture and storage process reduces the carbon footprint by up to 90%.

https://www.iberdrola.com/about-us/decarbonized-economy-principles-regulatory-actions/decarbonised-fuels

Jon J.

[John Stroman]

Jon,

Thanks for the interesting comparison.

1. I wonder about the entire Japanese sentence that contains 脱炭素燃料. Depending on the context, a human J>E translator might use "carbon-free fuel" rather than the literal "decabonized fuel" as a higher level of abstraction to remain accurate within the context of the entire document or simply to avoid doing the chemical research that you did to find out the details. In other words, perhaps the writer of the original Japanese sentence was just throwing in a term that he/she did not sufficiently understand just because it sounded erudite. I've encountered many such instances over the years.

2. You point out that DeepL goes sentence-by-sentence without reference to context. I agree that DeepL sometimes makes vocabulary errors, especially in specialized technical terms. However, there has been a lot of buzz recently about CHAT-GPT and other AI software "hallucinating," and apparently in that very specialized field of AI, hallucinations are even divided into subtypes. In general, however, AI hallucination refers to making wild guesses when the corpus is insufficient by drawing in factual information from a variety of unrelated sources and plugging that information into the output so it appears intelligible and glib to an uniformed reader, but may be glaringly inaccurate or logically contradictory to an expert in the field. Warren's most recent post is another good example.

We can only conclude at this point that AI chatting software may eventually become a useful tool, but as my wife has correctly pointed out, a tool is only as good as the person who uses it. In the case of programming, AI may be able to write a computer program, and the program may even work, but it takes an expert programmer to recognize quality and elegance. The same is true of translation. Tom Gally has pointed out that in longer files CHAT tends to produce inconsistencies in  vocabulary, etc., which may be overlooked by someone who is seeking a quick answer, but would not be acceptable in a specialized document written in a single voice by a professional writer  (i.e., an experienced human translator). I think AI may eventually squeeze out most entry level and even mid-level translation work, but the high quality, specialized work will remain for humans with the required expertise, and the AI tools will only be as good as the people who use them.

John Stroman

[Tom Gally]

This is an interesting and important discussion.

One thing to keep in mind is that we have not yet been able to try sophisticated prompt engineering using OpenAPI’s plugins for GPT-4. When those become available—within days or weeks?—it should be possible to compile bilingual glossaries by comparing documents in both languages, automatically double-check those glossaries against other references, have GPT-4 refer to those glossaries when producing translations, have GPT-4 double- and triple-check its own translations, etc.—in other words, reproduce much of the process that an experienced translator goes through, with some extra checks to minimize hallucinations.

Below is a prompt I just tried several times with GPT-4. The glossaries it produced would be helpful to me if I were translating the document by hand. In the tests I did, though, it did not indicate uncertainty about any of the translations it produced.

=====

I am a translator of Japanese to English. When I am translating a long document, I find it useful to prepare a bilingual glossary of Japanese terms that appear in the source text and the English translations that I would use. An ideal glossary includes technical terms and proper names in the source text that do not appear in a medium-sized Japanese dictionary as well as the English translations of those terms and romanization of those names.

Please prepare such a glossary for the following text from a Japanese Wikipedia page. First, list all of the Japanese terms and names that fall into the category described above. Provide English translations for each of them. If you are very unsure about the appropriateness of a translation, mark it with two question marks (??); if you are a little unsure, mark it with one question mark (?); if you are confident, leave it unmarked.

 

=====

Below is the glossary it produced for the first few paragraphs of https://ja.wikipedia.org/wiki/%E9%BA%BB%E9%85%94

 =====

Japanese Terms:

麻酔 (ますい) - anesthesia
鎮痛 - analgesia
不動化 - immobilization
健忘 - amnesia
無意識状態 - unconscious state
全身麻酔 - general anesthesia
鎮静 - sedation
区域麻酔 - regional anesthesia
広義の局所麻酔 - local anesthesia (in a broad sense)
神経ブロック - nerve block
硬膜外麻酔 - epidural anesthesia
脊髄くも膜下麻酔 - spinal anesthesia
医学的 - medical
獣医学的 - veterinary
催眠薬 - hypnotic
解離性麻酔薬 - dissociative anesthetic
鎮静薬 - sedative
麻酔補助薬 - anesthetic adjuvant
神経筋遮断薬 - neuromuscular blocking agent
麻薬 - narcotic
鎮痛薬 - analgesic
合併症 - complication
心筋梗塞 - myocardial infarction
肺塞栓症 - pulmonary embolism
術後 - postoperative
再入院 - readmission
悪性高熱症 - malignant hyperthermia

Romanization of Names:

N/A (There are no proper names in the provided text.)

======

You can also ask it to prepare the glossary in CSV format for importing into spreadsheets or databases.

I tried this same task with Bing Chat. Although it apparently is using GPT-4 under the hood, it produced much shorter glossaries and refused to produce anything more when asked.

I haven’t yet tried giving both a text and a glossary to GPT-4 and asking it to use the glossary in its translation of the text. Maybe later.

A bit later: I asked GPT-4 itself for advice on how to use it for translation. Here is that exchange:

=====

Me:

I am a translator of Japanese to English, and I want to use you (GPT-4) to speed up and improve my translation process. What are some prompt techniques I should use with you when translating technical or other specialized documents?

GPT-4:

Using GPT-4 to assist with your Japanese to English translations can be highly beneficial, especially when dealing with technical or specialized documents. Here are some prompt techniques to optimize your experience:

Specify context: Begin by providing context about the document you're translating. Mention the document type, subject matter, and any relevant terminology. This will help guide the model to produce more accurate translations.

Use concise, clear instructions: Be specific and clear in your instructions when asking for translations. For example, instead of saying "Translate the following text," you could say, "Translate the following Japanese technical text on computer networking into English."

Break down complex sentences: When translating long or complex sentences, break them down into smaller chunks. This can help improve the accuracy of the translation by reducing the chance of misinterpretations.

Request explanations: For technical terms or concepts that may be difficult to translate directly, ask for an explanation in the target language. For example, "Explain the concept of '物理層' (physical layer) in computer networking in English."

Verify terminology: If you're unsure about the translation of a specific term, ask the model to provide alternative translations or the definition of the term in the target language. For example, "What are some alternative English translations for the Japanese term 'プロトコル'?"

Utilize glossaries: If you have an existing glossary of specialized terms, provide it as context to help the model understand and use the correct terminology for your translations.

Proofread and revise: After receiving the initial translation, go through the text and identify any errors or areas that need improvement. You can then ask the model for suggestions or alternative translations for specific phrases.

Iterate and refine: Don't hesitate to rephrase your prompts or ask follow-up questions to obtain the most accurate translation or clarification.

Remember that GPT-4 is a powerful tool, but it may still produce errors or misinterpretations, especially in highly specialized or technical contexts. Always double-check the translations, and consider consulting additional resources or human experts when necessary.

=====

Tom Gally

[Tom Gally]

Speaking of prompts, a person with the user name Cyan Finch Ling, commenting on a recent YouTube video of mine, shows some very elaborate prompts for translating a Japanese literary text into English. Asking GPT-4 to reflect on its translation seemed to lead to even better results.

The full thread is in the comments to this video:

https://youtu.be/8JUepj7wIl0

Click on the replies to Goyong’s comment. I posted a published human translation of the same passage to the thread, so you can compare it with GPT-4’s if you like.

For more depressing news, I just came upon the following discussions about a freelance writer who says she lost her job to ChatGPT:

https://old.reddit.com/r/freelanceWriters/comments/12ff5mw/it_happened_to_me_today/

https://news.ycombinator.com/item?id=35519229

Tom Gally

[Dylan Luers Toda]

Hi everyone,

I've been reading these threads on Chat GPT with interest to prepare for an upcoming talk. They are very useful. Thank you everyone.

Sorry to resurrect an old discussion, but a quick note about the term 脱炭素燃料. It doesn't mean decarbonized fuel. 脱炭素 refers to the movement toward a zero/low-carbon emission society, and 脱炭素燃料 is providing an instance of a fuel that helps realize such a society. This is most clearly illustrated by the following passage from the Agency for Natural Resources and Energy's website about 合成燃料 ("CO2とH2を合成して製造される燃料"), which says that this fuel is a 脱炭素燃料.

原料となるCO2は、発電所や工場などから排出されたCO2を利用します。将来的には、大気中のCO2を直接分離・回収する「DAC技術」を使って、直接回収されたCO2を再利用することが想定されています。CO2を資源として利用する「カーボンリサイクル」に貢献することになるため、「脱炭素燃料」とみなすことができると考えられています。

エンジン車でも脱炭素？グリーンな液体燃料「合成燃料」とは｜スペシャルコンテンツ｜資源エネルギー庁 (meti.go.jp)

Best,

Dylan

[John Stroman]

Dylan,

 

Thank you for your post on Honyaku with the additional clarification.

 

Although no one responded to what I wrote, I think that I made a somewhat OK guess by suggesting "carbon-free fuel" based on experience and intuition, even though the fuel itself is synthesized from CO2 and H2, which makes my suggestion of "carbon-free" illogical in Jon's short example. After reading your post, however, I now know the greater context, and I should have suggested something like "recycled carbon-containing fuel" prepared by the synthesis of CO2 and H2.

 

Hopefully, human translators will still have jobs a few years from now, but they will have to bring to the table the kind of research skill you have demonstrated, a good sense of the writer's intention (because not everyone is a good writer), and the ability to write clearly in the target language, in addition to the ability to utilize something like GPT and similar AI products as the need arises.

 

More power to you,

 

John Stroman

----------------

--
You received this message because you are subscribed to the Google Groups "Honyaku E<>J translation list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honyaku+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honyaku/1487874b-6f19-45c4-ae2d-7b7e75891a27n%40googlegroups.com.