Re: Fullwiki test has at least one unanswerable question

[Peng Qi]

Hi Joe,

The passages in the fullwiki json are provided only for development purposes, and if you are planning to benchmark in this setting, you would likely need to build your own retrieval system over the entire Wikipedia to come up with the passages to arrive at the final answer. See the paper for more details.

Hope this helps!

Best,

Peng

On Oct 20, 2023 at 3:02 AM -0700, Joseph Davidson <joseph....@goodai.com>, wrote:

In the fullwiki test set (http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_test_fullwiki_v1.json), question 1 in the JSON ( "What year did the British politician born in 1967 began to represent the Daventry in the UK House of Commons?" ) is unanswerable given the supporting text for that question. Indeed, checking manually, none of the individuals mentioned in the data have ever represented Daventry.

Is there intended to be some other knowledge base that the model will draw the correct answer from? Otherwise how can you expect the model to be able to answer this?

Thanks,

- Joe

--
You received this message because you are subscribed to the Google Groups "HotpotQA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hotpotqa+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hotpotqa/599d3b14-e676-4798-a290-c15c3fc5ee9an%40googlegroups.com.