IPPC 2011 Results Available

[Scott Sanner]

Hi all,

Final results from the IPPC 2011 are now available:

  http://users.cecs.anu.edu.au/~ssanner/IPPC_2011/

Thanks to all who competed and congratulations to the winners:

  Boolean MDP Track

  * 1st place -- PROST: Thomas Keller, Patrick Eyerich

  * 2nd place -- Glutton: Andrey Kolobov, Peng Dai, Mausam, Dan Weld

  Boolean POMDP Track

  * Distinguished 1st place (best score) -- POMDPX: Kegui Wu, Wee Sun Lee

  * Tie for 1st place (within 95% stderr) -- KAIST: Dongho Kim, Kanghoon Lee, Kee-Eung Kim

The IPPC results presentation, log files, tabulated results, and planner descriptions are now posted on the above web page.

Cheers,

Scott

P.S. I have posted software links for POMDPX, Symbolic Perseus, and SPUDD.  For the others, please send me your software links if you have them -- it's free advertising for your work!

[Scott Sanner]

Hi all,

It has been pointed out to me that there was a normalization error in the results calculation: rather than normalizing by the max of the average over the last 30 runs for each planner/instance, I was previously using all runs (so the max was lower than it should have been in some of the cases... results tended to improve with later runs).  This has been fixed and a new data archive / results posted.

**Importantly, the relative ranking of the top two MDP and POMDP planners remains unchanged.**

There are two obvious differences though:

- There is a larger gap now b/w the top 2 POMDP planners although due to high stderr values, all analysis still shows overlapping error bounds.

- In the MDP domain the 3rd and 4th place planners swapped places (this change was not due to the renormalization, but rather an error in merging the results from two log files due to a mid-competition server change for the OSU Beaver runs).  

Cheers,

Scott