how to calculate the BLEU score

[shanw]

Hi,

In order to reproduce the result for WMT EN-DE 2014, could you tell me how you guys calculate the BLEU score, using t2t-bleu or multi-bleu.perl?

Thanks!

[Martin Popel]

I would suggest to use t2t-bleu or sacrebleu.

If you are brave and interested what method was used in the "Attention Is All You Need" paper, see
https://gitter.im/tensor2tensor/Lobby?at=5acfe16c7c3a01610dd81b46

Best,
Martin

> --
> You received this message because you are subscribed to the Google Groups
> "tensor2tensor" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to tensor2tenso...@googlegroups.com.
> To post to this group, send email to tensor...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tensor2tensor/e5ef204a-0ca5-42b2-ade7-d9ca7dced60a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[shanw]

Hi Martin,

Thanks for your reply. I have 4 gpus right now, I would like to use the latest version of t2t and use transformer, transformer-based to train an en-de model. How many steps do I need to train a sota model?

Thanks!

Shawn

在 2018年6月3日星期日 UTC-4下午3:13:16，Martin Popel写道：

[Martin Popel]

It depends on the batch size you can afford (which is limited by your GPU memory)
and several other hyper-parameters (big vs. base model,...).
Using the WMT training data (4.5 M sentence pairs), batch size>2k and 4GPUS,
I think at least 300k steps are needed.
It is always better to setup t2t-trainer with many more steps, plot the dev-set BLEU learning curve
and do early stopping (kill the training) when dev-set BLEU starts worsening or is good enough for your purposes.
I mean it may not be worth the money to run 4GPUs for another week, just to get extra 0.1 BLEU improvement.

>> > to tensor2tenso...@googlegroups.com <javascript:>.

>> > To post to this group, send email to tensor...@googlegroups.com

>> <javascript:>.

>> > To view this discussion on the web visit
>> >
>> https://groups.google.com/d/msgid/tensor2tensor/e5ef204a-0ca5-42b2-ade7-d9ca7dced60a%40googlegroups.com.
>>
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "tensor2tensor" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to tensor2tenso...@googlegroups.com.
> To post to this group, send email to tensor...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/tensor2tensor/ea05af7e-3b53-48b3-a236-fad06b5c7fdd%40googlegroups.com.

[shanw]

When I train the model, the evaluation is like this:

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Evaluation [10/100]

INFO:tensorflow:Evaluation [20/100]

INFO:tensorflow:Evaluation [30/100]

INFO:tensorflow:Evaluation [40/100]

INFO:tensorflow:Finished evaluation at 2018-06-03-23:33:04

Why does it stop at 40 instead of 100? Thanks!

在 2018年6月3日星期日 UTC-4下午4:29:48，Martin Popel写道：

[Martin Popel]

Most probably you dev test set contains just 40 eval batches. This is OK.
You can set eval_steps as you like, 100 is just a default value, see
https://github.com/tensorflow/tensor2tensor/blob/6c63/tensor2tensor/bin/t2t_trainer.py#L71

>> https://groups.google.com/d/msgid/tensor2tensor/ea05af7e-3b53-48b3-a236-fad06b5c7fdd%40googlegroups.com.
>>
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "tensor2tensor" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to tensor2tenso...@googlegroups.com.
> To post to this group, send email to tensor...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/tensor2tensor/04c8fff3-f11d-43df-97e4-162d49827bb6%40googlegroups.com.