Couple of Questions

[Kyle Morrison]

A couple of questions/requests about the competition:

Would it be possible to release the max and min scores for each instance, to allow the competitors to accurately calculate their overall score during testing?

Does the server have any failure recovery functionality? Which is to say, if me or my program notice that my solver is having trouble on a particular instance or a particular trial, can I simply abort client-side and issue a new session request or round request from the Server? How would the server handle such requests?

Regards,

Kyle Morrison

[Scott Sanner]

Hi Kyle,

Good questions, thanks.

> Would it be possible to release the max and min scores for each instance, 

I will certainly release min scores and I might release max scores for a few of the smallest instances in each domain, but my main reason for not releasing *all* max scores is that I really want to avoid having competitors withdraw if they don't perceive being in first or second place on the test competition problems.  Also, I'm not actively enforcing the 12 hour time constraint on the test competition so some test competition results may be misleading.  

Lastly, I note the final competition will have four entirely new domains and overall some planners may do well in some domains and poorly in others... I would hate for a competitor to withdraw due to their test competition performance and later tell me they would have done really well on some of the final competition domains.

> Does the server have any failure recovery functionality?

Sungwook has setup the Server to run a separate thread for each instance request so you can simply abandon trials for an instance and establish a new connection to the same Server port for a new instance.

To avoid too many stalled threads, I do recommend sending a malformed (e.g., empty) XML message for any instance you want to terminate as the Server will respond by killing that thread.  (Try this and you'll see the Server states that it is terminating that instance.)

Regarding scoring and incomplete trials, every time a trial is completed for an instance, this is logged, so you'll get credit for any trials completed and otherwise minS for incomplete trials.  See the Competition Fomat section here for more details on scoring:

  http://users.cecs.anu.edu.au/~ssanner/IPPC_2011/

Note that you will never get a normalized score of less than 0 for any trials that you've completed.

Cheers,

Scott

[Kyle Morrison]

Hmm, perhaps I have misread the website.

When it says "normalized-score_{planner,instance} = max(0, sum_{trials 1..30} [raw-score_{planner,instance,trial} - minS_{instance}] / [30 * (maxS_{instance} - minS_{instance})] )"
I assumed the maxS and minS were theoretical scores for the problem domain, meaning no planner can do better than maxS or worse than minS. What you're saying implies that these values represent actual planner scores, meaning maxS and minS represent the scores of the planners that did the worst and best, respectively. Is this the case?

Also, thank you, that's good to know about the Server.

Kyle

[Scott Sanner]

Sorry, there's a lot of info on the competition website

  http://users.cecs.anu.edu.au/~ssanner/IPPC_2011/

which makes it a little hard to digest, but you'll find the definitions of minS_{instance} and maxS_{instance} 3-4 lines above the normalized-score definition cited below.

Cheers,

Scott