Request for Assistance in Evaluating Machine Translation Output

[yaminaleyla moussaoui]

Dear SIGARAB NLP Community,

I hope this message finds you well.

I am currently working on a project involving the machine translation of poetic texts, and I would like to evaluate the output using one or more benchmark metrics (such as BLEU, METEOR, or BERTScore).

I would be very grateful if someone from the group could assist or guide me in applying one of these metrics — or suggest an approach that is more suitable for evaluating poetry translation, considering its stylistic and creative nature.

If anyone has experience with this or would be willing to offer help (even with a simple script or advice), please don’t hesitate to reach out.

Thank you in advance for your time and support.

Warm regards

Dr. 𝒴𝒶𝓂𝒾𝓃𝒶 𝐿𝑒𝒾𝓁𝒶 𝑀𝑜𝓊𝓈𝓈𝒶𝑜𝓊𝒾

Maître de recherche classe A

Centre de Recherche Scientifique et Technique pour le Développement de la Langue Arabe - CRSTDLA-

Unité de Recherche Tlemcen

https://www.crstdla.dz

 y.mou...@crstdla.dz

+213541831205

 

[A Light]

Dear Yamina,

I hope my email finds you well. 

Your project sounds fascinating, and it’s great to see growing interest in the evaluation of poetic translation in Arabic.

I’ve worked on this topic myself and would be happy to share some insights. Here is a link to a GitHub repository associated with a paper I co-authored on this subject: https://github.com/javierosorio/keep_it_local
The paper has been published, and you might find both the methods and code helpful.

In addition to that work, I’ve also used COMET in another research project and found it especially effective for capturing the nuanced meaning often present in poetry. Based on my experience, I highly recommend using COMET over traditional metrics like BLEU or METEOR for this kind of task. The official COMET GitHub repository is available here: https://github.com/Unbabel/COMET. Here is a link to a GitHub repository of another  paper I co-authored on this subject that had been accepted:  https://github.com/javierosorio/devil_in_the_details_mtsummit25

I’m genuinely interested in this area and open to future research collaborations.

Please don’t hesitate to reach out if you have any questions or would like to discuss further.

Warm regards,

--
You received this message because you are subscribed to the Google Groups "SIGARAB: Special Interest Group on Arabic Natural Language Processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sigarab+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sigarab/CAJP0JYY6yFz%2BGwSWkxi7NitJuZwM-DhxXUSesNNS5nUtOKETnA%40mail.gmail.com.