Submission format

Jaemin Cho

unread,

Nov 8, 2020, 4:48:29 PM11/8/20

to Visual Commonsense Reasoning

Hi Rowan,

According to submitting-to-the-leaderboard, it seems we need to submit

4 values for Q -> A task

and 16 values for QA -> R task (ex. rationale_conditioned_on_a0_0...)

per question.

But I guess only 4 out of 16 values, which correspond to predicted answers, are used when calculating QA->R accuracy. Can I just fill out the remaining 12 values as zeros?

Thanks,

Jaemin

Jaemin Cho

unread,

Nov 8, 2020, 4:56:30 PM11/8/20

to Visual Commonsense Reasoning

Oh, maybe I'm wrong. When evaluating QA->R task you might use the ground truth answer, not the answer predicted by the model.

Rowan Zellers

unread,

Nov 9, 2020, 4:59:45 PM11/9/20

to Visual Commonsense Reasoning

hi Jaemin! Exactly, you're free to fill in the remaining 12 values as zeros, but your performance will probably be worse on QA->R which conditions on ground truth answers.

thanks,

Rowan

Jaemin Cho

unread,

Nov 9, 2020, 5:02:42 PM11/9/20

to Visual Commonsense Reasoning

I see. Thanks for the clarification!

Reply all

Reply to author

Forward