Competition Server Live

[Pedro Rodriguez]

Hi All,

We have completed maintenance of the HCQA server and test data is uploaded. We made a few changes that are documented in the readme page (https://github.com/Pinafore/qb-api) and competition webpage https://sites.google.com/view/hcqa/dates, but are highlighted below:

• As noted before, if a question is in the test fold, you will only have one chance to submit an answer, any further requests will return a 400 error.
• Answers
  
• As described here (https://github.com/Pinafore/qb-api#note-on-answers), answers map to exactly wikipedia page title with whitespaces replaced by underscores. In the readme we provide a link to a file containing wikipedia page titles in the correct format. We made this change since in uploading the test data we realized that the dev data answers didn't match the non_naqt.db database file as a result of some work we did last spring on answer alignment. This should make it easier to use the dataset since you should be able to use the answer as is, without modification.
  
• Since we made this change just now, we are being more flexible on what we consider to be a correct answer. An answer will be correct if it is an exact match, exact match ignoring case, or exact match after converting unicode characters (like accented characters) to their closest ascii visual equivalent (accented e to regular e for example). You will still need to convert any whitespace to underscores. On October 25th we will also release a report for each team, and allow you to contest the correctness of your answers.
• For example, the title "Māori people" should be submitted as "Māori_people". However, we would also consider "Maori_people" correct since ā->a. We would also accept "maori_people" since it is a case insensitive match.
• We request that when you run your system against our server, please insert a delay between requests in your client code. Running without a delay for too long may result in a timed out request. An example of doing that is here: https://github.com/Pinafore/qb-api/blob/dbf3063ca34ab5ba73e2b25baf822657e06c1f5e/web/es_guesser/example.py#L49
  
• All prior history on the dev server was not transferred to the test server. You will need to visit qb.entilzha.io/register again to obtain a new API key and user ID and change your variables accordingly.
• We have run the dummy reference system, and elastic search reference system included in the repository code. Their results are clearly marked on the leaderboard.
• The submission deadline is extended to October 20th. Other dates have been updated on the webpage https://sites.google.com/view/hcqa/dates.

Please let us know if you have any questions, or are having trouble with anything.

Thanks again for competing, good luck, and see you at NIPS!

-- 
Pedro Rodriguez
PhD Candidate in Machine Learning and NLP | UMD | CU
pedrorodriguez.io | github.com/EntilZha