reproducing paper results difficulty

[lima....@gmail.com]

Dear sir/mdm,

We're using your paper in our master AI at University of Amsterdam and we are having a hard time to reproduce the results reported in your paper. We are running the model as-is, without any changes but we have not managed to reproduce the reported figures. Is there anything special that you guys did in terms of tuning that we should be aware of? Below our results for the test dev split:

   

Answer

Supporting Facts

Joint

EM

F1

EM

F1

EM

F1

41.21

55.60

16.83

58.13

8.37

34.66

As a final trial I've increased the early stopping moment but I noticed that the model doesn't get any better.  Could you help us? Looking forward to hear from you. 

Regards,

Edwin Lima

[zhi...@google.com]

There are a few discussions here https://github.com/hotpotqa/hotpot/issues/6

"I would suggest trying different random seeds to study the effects of model variance. Some random seeds might work better."

Let me know if the problem still exists.

[Edwin Lima]

Hi Zhiliny,

I have tried a different approach which is decreasing the factor of change for the learning rate after every evaluation. So in your code it was 0.5 and I made it 0.9 as show in the image below, in other words I keep 90 percent of the previous learning rate instead of only 50%. 

See below the decay of the learning rate curve, orange is according to your code, blue is my change. The point were both curves cross is the moment where the training (early) stops.

The training as expected is taking longer but the training prelimanry results look very promising. Thanks for your reply.

--
You received this message because you are subscribed to a topic in the Google Groups "HotpotQA" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hotpotqa/hUVirC8e978/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hotpotqa+u...@googlegroups.com.
To post to this group, send email to hotp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hotpotqa/33d3b43b-2d15-4e23-9128-7d58892fdb9f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.