9 views

Skip to first unread message

Jul 18, 2019, 11:28:40 PM7/18/19

to Arben Sabani, tensor2tensor

Hi Arben,

t2t-bleu implementation (as well as SacreBLEU) uses the correct BLEU formula

(as in the original paper https://www.aclweb.org/anthology/P02-1040).

The formula in https://i.stack.imgur.com/MZygZ.png is wrong

because of the brevity penalty, which should be

BP = exp(min(0, 1 - r/c)) = min(1, exp(1 - r/c))

but the image uses

BP = min(1, c/r)

Except for the brevity penalty, the formula on the image is correct

because the geometric mean can also be expressed as the exponential of the arithmetic mean of logarithms,

see https://en.wikipedia.org/wiki/Geometric_mean#Relationship_with_logarithms

Martin

----- Original Message -----

> From: "Arben Sabani" <arben....@gmail.com>

> To: "tensor2tensor" <tensor...@googlegroups.com>

> Sent: Saturday, July 13, 2019 6:01:05 AM

> Subject: Formula used to calculate BLEU

> Hi Martin

> Hi everyone

>

> In the paper https://arxiv.org/pdf/1804.00247.pdf is mentioned that BLEU is

> calculated using "t2t-bleu" command.

> Does anyone know how the mathematical formula looks like.

>

>

> In https://www.aclweb.org/anthology/P02-1040 is this formula mentioned:

>

> [image: 2019-07-13 14_50_49-bleu.dvi.png] <about:invalid#zClosurez>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

> But there are other sources where the formula looks a bit different:

>

> <https://i.stack.imgur.com/MZygZ.png>

>

> --

> You received this message because you are subscribed to the Google Groups

> "tensor2tensor" group.

> To unsubscribe from this group and stop receiving emails from it, send an email

> to tensor2tenso...@googlegroups.com.

> To post to this group, send email to tensor...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/tensor2tensor/d20aaa29-8709-4f29-b112-900193c31881%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

t2t-bleu implementation (as well as SacreBLEU) uses the correct BLEU formula

(as in the original paper https://www.aclweb.org/anthology/P02-1040).

The formula in https://i.stack.imgur.com/MZygZ.png is wrong

because of the brevity penalty, which should be

BP = exp(min(0, 1 - r/c)) = min(1, exp(1 - r/c))

but the image uses

BP = min(1, c/r)

Except for the brevity penalty, the formula on the image is correct

because the geometric mean can also be expressed as the exponential of the arithmetic mean of logarithms,

see https://en.wikipedia.org/wiki/Geometric_mean#Relationship_with_logarithms

Martin

----- Original Message -----

> From: "Arben Sabani" <arben....@gmail.com>

> To: "tensor2tensor" <tensor...@googlegroups.com>

> Sent: Saturday, July 13, 2019 6:01:05 AM

> Subject: Formula used to calculate BLEU

> Hi Martin

> Hi everyone

>

> In the paper https://arxiv.org/pdf/1804.00247.pdf is mentioned that BLEU is

> calculated using "t2t-bleu" command.

> Does anyone know how the mathematical formula looks like.

>

>

> In https://www.aclweb.org/anthology/P02-1040 is this formula mentioned:

>

> [image: 2019-07-13 14_50_49-bleu.dvi.png] <about:invalid#zClosurez>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

> But there are other sources where the formula looks a bit different:

>

> <https://i.stack.imgur.com/MZygZ.png>

>

> --

> You received this message because you are subscribed to the Google Groups

> "tensor2tensor" group.

> To unsubscribe from this group and stop receiving emails from it, send an email

> to tensor2tenso...@googlegroups.com.

> To post to this group, send email to tensor...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/tensor2tensor/d20aaa29-8709-4f29-b112-900193c31881%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward

0 new messages

Search

Clear search

Close search

Google apps

Main menu