Issue with News-commentary

[vinc...@yahoo.com]

Hi Tom / Barry,

When looking at WMT23 dataset  saw that you are giving links to NCv16 instead of v18.

I thought there was an issue in v18 but I realized there is the same issue in v17 and v16 (I am looking only at the ENDE file)

The following lines are repeated many times and with SOURCE = TARGET in English:

Following its targeted killing of Iran's second most powerful leader, the US could well find itself with no alternative but to devote more military resources to the Middle East, a path that could lead to additional Iranian provocations.
And that shift would occur at a time of growing challenges to US interests elsewhere in the world.
The targeted killing by the United States of one of Iran's top military leaders represents a significant symbolic defeat for the Iranian regime, but it does not augur all-out war.
After all, Iran and the US have already been at war for decades, and neither side has an interest in an uncontrolled military escalation now.
As in previous years, Project Syndicate asked its commentators to offer their best guesses about the events and trends that will define the coming 12 months.
If there is one common theme in this year's selection, it is that what happens in 2020 will weigh on the present for many years to come.

with line numbering starting at 1, you will see this block at line 134, 208, 423 ....

678 times altogether.

I am not sure if this has just been inserted in the middle of valid stories or if it replaced some lines in the stories.

Enjoy

[vinc...@yahoo.com]

there is also this sentence 617 times in the german side:

Enjoy unlimited access to the ideas and opinions of the world's leading thinkers, including weekly long reads, book reviews, and interviews; The Year Ahead annual print magazine; the complete PS archive; and more – All for less than $9 a month.

sometimes with the same in the english side but not always .....

[Barry Haddow]

Hi

Thanks for spotting this! I have added a fix to the extraction code, and I am re-running it. If all looks I well, I will update the release,

best

Barry

On 09/05/2023 14:57, 'vinc...@yahoo.com' via Workshop on Statistical Machine Translation wrote:

This email was sent to you by someone outside the University.

You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

--
You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/85491f7f-07de-481e-ab53-b21c61079cean%40googlegroups.com.

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[Barry Haddow]

Hi

I have regenerated the news-commentary corpus to fix the extraction. This fixes the error noted below, and also improves document alignment a bit, resulting in more aligned documents.

I created a new release here (https://data.statmt.org/news-commentary/v18.1/) , and will point the WMT task page at that release,

best

Barry

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336. --

You received this message because you are subscribed to the Google Groups "Workshop on Statistical Machine Translation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wmt-tasks+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wmt-tasks/3b5fc7da-f9f1-be9c-bdc5-1e8667a37f67%40ed.ac.uk.