Warmup Competition
92 views
Skip to first unread message
ssanner
Mar 13, 2014, 10:44:42 AM3/13/14
to ippc-2014...@googlegroups.com
Dear IPPC 2014 Discrete Track MDP or POMDP Competitors,
It's nearing time for the warmup competition! Important points below with sign-up link at end:
(1) EC2 Setup: we will run the warmup competition on EC2. Marek has previously posted instructions for how to setup EC2 for this year:
Please post a reply to Marek's message if you have any trouble getting your EC2 account / instance setup.
(2) EC2 Cost: we have applied for an EC2 grant to provide free compute time to all competitors, but we will not hear if we are successful until mid-April (after the warm-up competition finishes).
For the warm-up competition, you could use a *free* micro-instance if you don't want to spend any money, but you will likely be very unhappy with it's performance. You'll need to use an m1.large instance for the final competition and we recommend using it for the warm-up competition to get accustomed to its performance characteristics. For now, you will have to pay for any non-free EC2 time out-of-pocket, but 5 hours of m1.large compute time costs about USD $1.25 and you'll only need 5 hours for the warm-up competition.
(3) Objective and Format: the goal of the warm-up competition is simply to test your client with our RDDLSim server infrastructure in the EC2 environment.
Results will not be released and there is no performance threshold that your planner must meet in order to compete in the final competition.
For simplicity, we will use exactly the final competition problems from the IPPC 2011... you can download them in either tgz or zip format:
There are two files of competition instances in the above archives
final_comp/mdp_instance_list.txt
final_comp/pomdp_instance_list.txt
that should be used *respectively* by the MDP track and POMDP track competitors. For the warmup competition, we will require you to execute on all instances (from the respective file for your track) with a suffix of either __1 or __5 (small and medium sizes).
This amounts to 8 domains, 2 instances per domain = 16 instances total (a slight departure from the 4 domains I mentioned previously).
The time limit per instance is 18 minutes (to complete 30 trials). Overall, the warm-up competition will take < 5 hours (16 instances X 18 minutes each).
You are welcome to compete on *other/all* instances from the IPPC 2011 if you want, but you should complete on at least the 16 required. We will send you the log file for your results after you have completed the warm-up competition.
(4) Signup: to signup for a warmup competition slot, please enter your name on the Doodle below indicating what day between April 7-18 you want to run your planner:
You are welcome to change the date, so for now, please enter your name and a nominal date so we can know who is running in the competition and get a rough idea of when.
Prior to each competitor's nominated date we will send the competitor the EC2 public hostname and a unique port during which they can connect to the server in order to complete their ~5 hours of warm-up runs.
If you have questions / concerns, please post a reply for discussion.
Cheers,
Scott
ssanner
Apr 7, 2014, 11:09:16 PM4/7/14
to ippc-2014...@googlegroups.com
Dear IPPC 2014 Competitors,
Please signup for IPPC 2014 if you have not done so already (see previous email in this thread for details).
We'll be flexible (so email us if you are having difficulties or need more time), but please note that completing the warmup competition is a prerequisite for competing in the final competition.
Thanks,
Scott, Marek, Jesse
Marek Grzes
Apr 10, 2014, 11:37:18 PM4/10/14
to ippc-2014...@googlegroups.com
Dear All,
Our EC2 instance is ready to go and I can start a dedicated competition server for your team. Please email to me the exact time and day when you'd like to run your warmup experiment. If possible, please specify the time and day for the Toronto time zone. Also, if possible it would be really convenient for me if your starting time wouldn't be before 7am and not later then midnight Toronto time. Hope you'll understand my preference :). Thanks to those who already expressed their preference using doodle - I will still need the exact time from you guys.
To sum up:
1) I would need from you the exact time and day when you'd like to start your experiment
2) In response, I am going to send you back the IP address of the server and the TCP port number that will be assigned to your team.
Looking forward to hearing from you all!
Cheers,
Marek
Our EC2 instance is ready to go and I can start a dedicated competition server for your team. Please email to me the exact time and day when you'd like to run your warmup experiment. If possible, please specify the time and day for the Toronto time zone. Also, if possible it would be really convenient for me if your starting time wouldn't be before 7am and not later then midnight Toronto time. Hope you'll understand my preference :). Thanks to those who already expressed their preference using doodle - I will still need the exact time from you guys.
To sum up:
1) I would need from you the exact time and day when you'd like to start your experiment
2) In response, I am going to send you back the IP address of the server and the TCP port number that will be assigned to your team.
Looking forward to hearing from you all!
Cheers,
Marek
--
You received this message because you are subscribed to the Google Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Thomas Keller
Apr 14, 2014, 4:20:51 AM4/14/14
to ippc-2014...@googlegroups.com, Florian Geißer
Dear Marek,
Florian and I would like to start the IPPC warmup competition at 8am
Toronto time. Will there be a list of instances available to simplify
sequential planner calls? If possible, it'd be neat if we were allowed
to use a simple text file that contains a line with the domain file name
and the associated instance file name for each problem (we can, of
course, write the file manually ourselves, I just want to make sure it's
OK to do so).
Also, we would be interested in have more than one configuration of the
PROST planner compete. In particular, we are working on a version with
automatically tuned parameters (this is in collaboration with Prof.
Frank Hutter, who is also in Freiburg now). Moreover, if you are
interested I'd submit a version that is similar to the winner of the
last IPPC (it is not identical since some bugs have been fixed for it as
well, but it would be the same parameter setting and thus quite
comparable). In both cases, I don't see a reason to run the warmup
competition, though, since they build on the same code and there
shouldn't be any problem that doesn't occur in our "main" configuration.
Please let us know if competing with two configurations is allowed, and
if you are interested in having the 2011 version in the competition.
Cheers,
Thomas & Florian
>> <mailto:ippc-2014-discr...@googlegroups.com>.
Florian and I would like to start the IPPC warmup competition at 8am
Toronto time. Will there be a list of instances available to simplify
sequential planner calls? If possible, it'd be neat if we were allowed
to use a simple text file that contains a line with the domain file name
and the associated instance file name for each problem (we can, of
course, write the file manually ourselves, I just want to make sure it's
OK to do so).
Also, we would be interested in have more than one configuration of the
PROST planner compete. In particular, we are working on a version with
automatically tuned parameters (this is in collaboration with Prof.
Frank Hutter, who is also in Freiburg now). Moreover, if you are
interested I'd submit a version that is similar to the winner of the
last IPPC (it is not identical since some bugs have been fixed for it as
well, but it would be the same parameter setting and thus quite
comparable). In both cases, I don't see a reason to run the warmup
competition, though, since they build on the same code and there
shouldn't be any problem that doesn't occur in our "main" configuration.
Please let us know if competing with two configurations is allowed, and
if you are interested in having the 2011 version in the competition.
Cheers,
Thomas & Florian
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ippc-2014-discr...@googlegroups.com
> <mailto:ippc-2014-discr...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ippc-2014-discr...@googlegroups.com
Marek Grzes
Apr 14, 2014, 11:59:02 PM4/14/14
to ippc-2014...@googlegroups.com
On 14/04/14 04:20, Thomas Keller wrote:
> In both cases, I don't see a reason to run the warmup competition
Hi Thomas,
> In both cases, I don't see a reason to run the warmup competition
The warmup competition is to test the entire infrastructure. We would
basically like to be sure that all the pieces work together, that the
EC2 instances work fine, that participants know how to use them, that we
know how to use them, that there are not issues in the RDDL simulator
(there were some extension this year etc.), and that our communication
with participants is effective. It is not that much about collecting
results (you can use a very limited version of your planner in the
warmup competition if you wish, it does not matter very much); instead
we are interested in seeing all the pieces working together.
Cheers,
Marek
Scott Sanner
Apr 15, 2014, 12:22:45 AM4/15/14
to ippc-2014...@googlegroups.com, Thomas Keller, Florian Geißer
Hi Thomas and Florian (and IPPC competitors),
In response to your email, two notes:
Instance lists: I attach the MDP and POMDP instance lists for the warmup competition (however note that Google Groups sometimes rejects these attachments). If you don't seen them attached, I have inlined their contents below. Similar instance lists will be provided *with* the problem archive distributed for the final competition. You're welcome to run other instances for the warmup competition, this is just the minimum set.
Multiple planners: For the final competition, re-running previous (corrected) IPPC 2011 baselines is great and multiple planners are OK if they are distinctly different (the variations you have suggested sound fine to me). I suggest capping the number of variants at 3, but if people object we can discuss on the mailing list. Of course what we don't want is for people to run 3 variants with slight differences in tuning parameters -- there should be something fundamentally different in each planner. We'll ask that all competitors register their intended planners in a post to this IPPC list and briefly describe differences between their planners before the competition. Deadline for this posting is TBD (sometime in May), but feel free to post your plans now if you want.
Cheers,
Scott
mdp_instance_list_1_and_5.txt
===
crossing_traffic_inst_mdp__1
crossing_traffic_inst_mdp__5
elevators_inst_mdp__1
elevators_inst_mdp__5
game_of_life_inst_mdp__1
game_of_life_inst_mdp__5
navigation_inst_mdp__1
navigation_inst_mdp__5
recon_inst_mdp__1
recon_inst_mdp__5
skill_teaching_inst_mdp__1
skill_teaching_inst_mdp__5
sysadmin_inst_mdp__1
sysadmin_inst_mdp__5
traffic_inst_mdp__1
traffic_inst_mdp__5
pomdp_instance_list_1_and_5.txt
===
crossing_traffic_inst_pomdp__1
crossing_traffic_inst_pomdp__5
elevators_inst_pomdp__1
elevators_inst_pomdp__5
game_of_life_inst_pomdp__1
game_of_life_inst_pomdp__5
navigation_inst_pomdp__1
navigation_inst_pomdp__5
recon_inst_pomdp__1
recon_inst_pomdp__5
skill_teaching_inst_pomdp__1
skill_teaching_inst_pomdp__5
sysadmin_inst_pomdp__1
sysadmin_inst_pomdp__5
traffic_inst_pomdp__1
traffic_inst_pomdp__5
<mailto:ippc-2014-discrete+unsub...@googlegroups.com>.an email to ippc-2014-discrete+unsub...@googlegroups.com
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to ippc-2014-discrete+unsub...@googlegroups.com<mailto:ippc-2014-discrete+unsub...@googlegroups.com>.
--
You received this message because you are subscribed to the Google Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discrete+unsub...@googlegroups.com.
Andrey Kolobov
Apr 15, 2014, 12:58:47 AM4/15/14
to ippc-2014...@googlegroups.com, Thomas Keller, Florian Geißer
Hi all,
In response to Scott's request for entry descriptions, my current plan is to run the Gourmand solver(its implementation and description are here: http://www.cs.washington.edu/ai/planning/gourmand.html) as the competition baseline. I've accumulated a few minor fixes to its implementation over the past two years, but algorithmically it is still as described in the paper at the link above.
Cheers,
Andrey
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.
Thomas Keller
Apr 15, 2014, 3:50:06 AM4/15/14
to ippc-2014...@googlegroups.com
Hi Scott,
I'd actually prefer a list of domain and instance file names, i.e. a
list of entries of the form:
elevators_mdp.rddl_prefix elevators_inst_mdp__1.rddl_prefix
elevators_mdp.rddl_prefix elevators_inst_mdp__5.rddl_prefix
game_of_life_mdp.rddl_prefix game_of_life_inst_mdp__1.rddl_prefix
...
Is it possible to provide such a list for the final competition? I'll
prepare it myself for the warmup run today.
Cheers
Thomas
On 04/15/2014 06:22 AM, Scott Sanner wrote:
> Hi Thomas and Florian (and IPPC competitors),
>
> In response to your email, two notes:
>
> *Instance lists: *I attach the MDP and POMDP instance lists for the
> ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
>
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
>
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
> <mailto:ippc-2014-discr...@googlegroups.com>.
I'd actually prefer a list of domain and instance file names, i.e. a
list of entries of the form:
elevators_mdp.rddl_prefix elevators_inst_mdp__1.rddl_prefix
elevators_mdp.rddl_prefix elevators_inst_mdp__5.rddl_prefix
game_of_life_mdp.rddl_prefix game_of_life_inst_mdp__1.rddl_prefix
...
Is it possible to provide such a list for the final competition? I'll
prepare it myself for the warmup run today.
Cheers
Thomas
On 04/15/2014 06:22 AM, Scott Sanner wrote:
> Hi Thomas and Florian (and IPPC competitors),
>
> In response to your email, two notes:
>
> warmup competition (however note that Google Groups sometimes rejects
> these attachments). If you don't seen them attached, I have inlined
> their contents below. Similar instance lists will be provided *with*
> the problem archive distributed for the final competition. You're
> welcome to run other instances for the warmup competition, this is just
> the minimum set.
>
> *Multiple planners: *For the final competition, re-running previous
> these attachments). If you don't seen them attached, I have inlined
> their contents below. Similar instance lists will be provided *with*
> the problem archive distributed for the final competition. You're
> welcome to run other instances for the warmup competition, this is just
> the minimum set.
>
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
>
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to ippc-2014-discre...@googlegroups.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
>
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to ippc-2014-discre...@googlegroups.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it,
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ippc-2014-discr...@googlegroups.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> <mailto:ippc-2014-discr...@googlegroups.com>.
Scott Sanner
Apr 15, 2014, 4:55:31 AM4/15/14
to ippc-2014...@googlegroups.com
Hi Thomas,
It is possible of course, but if individual planners need certain formats, I would prefer the competitors to generate such files on their own.
I planned to follow exactly the IPPC 2011 format which just provided instance names. I will follow the same naming convention as in IPPC 2011.
Cheers,
Scott
<mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from
it, send
<mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to ippc-2014-discrete+__unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to ippc-2014-discrete+unsub...@googlegroups.com
<mailto:ippc-2014-discrete+unsub...@googlegroups.com>.
--
You received this message because you are subscribed to the Google Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discrete+unsub...@googlegroups.com.
Thomas Keller
Apr 15, 2014, 4:59:46 AM4/15/14
to ippc-2014...@googlegroups.com
Hi Scott,
I don't mind preparing it myself, I just wanted to make sure it's OK to
do so and that I'll have the 5 minutes it takes. Thanks!
Cheers
Thomas
On 04/15/2014 10:55 AM, Scott Sanner wrote:
> Hi Thomas,
>
> > Is it possible to provide such a list for the final competition?
>
> It is possible of course, but if individual planners need certain
> formats, I would prefer the competitors to generate such files on their own.
>
> I planned to follow exactly the IPPC 2011 format which just provided
> instance names. I will follow the same naming convention as in IPPC 2011.
>
> Cheers,
> Scott
>
>
>
> On Tue, Apr 15, 2014 at 5:50 PM, Thomas Keller
> <tke...@informatik.uni-freiburg.de
> elevators_mdp.rddl_prefix elevators_inst_mdp__5.rddl___prefix
> game_of_life_mdp.rddl_prefix game_of_life_inst_mdp__1.rddl___prefix
> pomdp_instance_list_1_and_5.__txt
> <mailto:tke...@informatik.uni-freiburg.de>
> <mailto:tkeller@informatik.__uni-freiburg.de
> ippc-2014-discret...@googlegroups.com
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>
>
> <mailto:ippc-2014-discret...@googlegroups.com
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>>.
>
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>
>
> <mailto:ippc-2014-discret...@googlegroups.com
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>>.
>
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>.
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <mailto:ippc-2014-discr...@googlegroups.com>.
I don't mind preparing it myself, I just wanted to make sure it's OK to
do so and that I'll have the 5 minutes it takes. Thanks!
Cheers
Thomas
On 04/15/2014 10:55 AM, Scott Sanner wrote:
> Hi Thomas,
>
> > Is it possible to provide such a list for the final competition?
>
> It is possible of course, but if individual planners need certain
> formats, I would prefer the competitors to generate such files on their own.
>
> I planned to follow exactly the IPPC 2011 format which just provided
> instance names. I will follow the same naming convention as in IPPC 2011.
>
> Cheers,
> Scott
>
>
>
> On Tue, Apr 15, 2014 at 5:50 PM, Thomas Keller
> <tke...@informatik.uni-freiburg.de
> <mailto:tke...@informatik.uni-freiburg.de>> wrote:
>
> Hi Scott,
>
> I'd actually prefer a list of domain and instance file names, i.e. a
> list of entries of the form:
>
> elevators_mdp.rddl_prefix elevators_inst_mdp__1.rddl___prefix
>
> Hi Scott,
>
> I'd actually prefer a list of domain and instance file names, i.e. a
> list of entries of the form:
>
> elevators_mdp.rddl_prefix elevators_inst_mdp__5.rddl___prefix
> game_of_life_mdp.rddl_prefix game_of_life_inst_mdp__1.rddl___prefix
> ===
> crossing_traffic_inst_pomdp__1
> crossing_traffic_inst_pomdp__5
> elevators_inst_pomdp__1
> elevators_inst_pomdp__5
> game_of_life_inst_pomdp__1
> game_of_life_inst_pomdp__5
> navigation_inst_pomdp__1
> navigation_inst_pomdp__5
> recon_inst_pomdp__1
> recon_inst_pomdp__5
> skill_teaching_inst_pomdp__1
> skill_teaching_inst_pomdp__5
> sysadmin_inst_pomdp__1
> sysadmin_inst_pomdp__5
> traffic_inst_pomdp__1
> traffic_inst_pomdp__5
>
>
>
>
>
>
>
> On Mon, Apr 14, 2014 at 6:20 PM, Thomas Keller
> <tke...@informatik.uni-__freiburg.de
> crossing_traffic_inst_pomdp__1
> crossing_traffic_inst_pomdp__5
> elevators_inst_pomdp__1
> elevators_inst_pomdp__5
> game_of_life_inst_pomdp__1
> game_of_life_inst_pomdp__5
> navigation_inst_pomdp__1
> navigation_inst_pomdp__5
> recon_inst_pomdp__1
> recon_inst_pomdp__5
> skill_teaching_inst_pomdp__1
> skill_teaching_inst_pomdp__5
> sysadmin_inst_pomdp__1
> sysadmin_inst_pomdp__5
> traffic_inst_pomdp__1
> traffic_inst_pomdp__5
>
>
>
>
>
>
>
> On Mon, Apr 14, 2014 at 6:20 PM, Thomas Keller
> <mailto:tke...@informatik.uni-freiburg.de>
> <mailto:tkeller@informatik.__uni-freiburg.de
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>
>
> <mailto:ippc-2014-discret...@googlegroups.com
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>>.
>
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to
> the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving
> emails from
> it, send
> an email to
> ippc-2014-discret...@googlegroups.com
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to
> the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving
> emails from
> it, send
> an email to
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>
>
> <mailto:ippc-2014-discret...@googlegroups.com
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>>.
>
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the
> Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails
> from it,
> send an email to
> ippc-2014-discret...@googlegroups.com
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails
> from it,
> send an email to
> <mailto:ippc-2014-discrete%2B__uns...@googlegroups.com>
> <mailto:ippc-2014-discrete%__2Buns...@googlegroups.com
> <mailto:ippc-2014-discrete%252Buns...@googlegroups.com>__>.
> For more options, visit
> https://groups.google.com/d/____optout
> <https://groups.google.com/d/__optout>
> <https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>
> <mailto:ippc-2014-discre...@googlegroups.com
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to ippc-2014-discre...@googlegroups.com
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it,
> <mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ippc-2014-discr...@googlegroups.com
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "IPPC 2014 (DISCRETE TRACK)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> <mailto:ippc-2014-discr...@googlegroups.com>.
Scott Sanner
Apr 15, 2014, 5:16:05 AM4/15/14
to ippc-2014...@googlegroups.com
> I'll have the 5 minutes it takes
We're certainly happy to give competitors some reasonable preparation time between downloading the instances and beginning the competition.
Cheers,
Scott
On Tue, Apr 15, 2014 at 6:59 PM, Thomas Keller <tke...@informatik.uni-freiburg.de> wrote:
Hi Scott,
I don't mind preparing it myself, I just wanted to make sure it's OK to do so and that I'll have the 5 minutes it takes. Thanks!
Cheers
Thomas
On 04/15/2014 10:55 AM, Scott Sanner wrote:
Hi Thomas,
> Is it possible to provide such a list for the final competition?
It is possible of course, but if individual planners need certain
formats, I would prefer the competitors to generate such files on their own.
I planned to follow exactly the IPPC 2011 format which just provided
instance names. I will follow the same naming convention as in IPPC 2011.
Cheers,
Scott
On Tue, Apr 15, 2014 at 5:50 PM, Thomas Keller
<tke...@informatik.uni-freiburg.de
<mailto:tkeller@informatik.uni-freiburg.de>
<mailto:tkeller@informatik.__uni-freiburg.de
ippc-2014-discrete+____unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2B__unsub...@googlegroups.com>
<mailto:ippc-2014-discrete%__2Bunsu...@googlegroups.com
<mailto:ippc-2014-discrete%252Bunsubscribe@googlegroups.com>__>
<mailto:ippc-2014-discrete+____unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2B__unsub...@googlegroups.com>
<mailto:ippc-2014-discrete%__2Bunsu...@googlegroups.com
<mailto:ippc-2014-discrete%252Bunsubscribe@googlegroups.com>__>>.
For more options, visit
https://groups.google.com/d/____optout
<https://groups.google.com/d/__optout>
<https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>>.
--
You received this message because you are subscribed to
the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving
emails from
it, send
an email to
ippc-2014-discrete+____unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2B__unsub...@googlegroups.com>
<mailto:ippc-2014-discrete%__2Bunsu...@googlegroups.com
<mailto:ippc-2014-discrete%252Bunsubscribe@googlegroups.com>__>
<mailto:ippc-2014-discrete+____unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2B__unsub...@googlegroups.com>
<mailto:ippc-2014-discrete%__2Bunsu...@googlegroups.com
<mailto:ippc-2014-discrete%252Bunsubscribe@googlegroups.com>__>>.
For more options, visit
https://groups.google.com/d/____optout
<https://groups.google.com/d/__optout>
<https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>>.
--
You received this message because you are subscribed to the
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails
from it,
send an email to
For more options, visit
https://groups.google.com/d/____optout
<https://groups.google.com/d/__optout>
<https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from
it, send
<mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to ippc-2014-discrete+__unsub...@googlegroups.com
<mailto:ippc-2014-discrete%2Bunsu...@googlegroups.com>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to ippc-2014-discrete+unsub...@googlegroups.com
<mailto:ippc-2014-discrete+unsub...@googlegroups.com>.
--
You received this message because you are subscribed to the Google Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discrete+unsub...@googlegroups.com.
Andrey Kolobov
Apr 15, 2014, 5:24:29 AM4/15/14
to ippc-2014...@googlegroups.com, Scott Sanner
Hi Scott,
Another question about the rules... What happens if a participant initiates several sessions to solve a given problem, but fails to finish the last one within the allocated time?
Say, a planner initiates three sessions (one after another) to solve a given problem. It completes the first two sessions, 30 rounds each, but only finishes 20 out of 30 rounds of the third one before the allocated 1080 seconds run out. In this case, which of the rounds will count towards the planner's official result? Will you count the last completed session, or the last 30 rounds (in the example above -- 20 rounds from the last session + the last 10 rounds from the penultimate session)?
Thanks,
Andrey
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.
Nicolas Drougard
Apr 15, 2014, 12:19:11 PM4/15/14
to ippc-2014...@googlegroups.com, mgr...@cs.uwaterloo.ca
Dear organizers, The planner we intend to use for the competition is based on your translation of RDDL problems to the SPUDD format. Don't worry, we don't plan to run SPUDD :) The current code of rddlsim does not seem to be able to translate RDDL domains that would generate intermediate variables, typically (we guess) domains with non-boolean variables which would each one be translated into probabilistically-dependent boolean variables. Thus: - Will there be benchmarks with intermediate variables? - If so, are there any plans to provide a translation to SPUDD that works with intermediate variables? As a side note, we also intend to run 2 or 3 different planners for the competition, except if we observe in our own tests that one of them largely outperforms the others on all criteria.
Finally, 04/17/14 8am (Toronto time) will be ok for our warmup experiment.
Best, Nicolas and Florent.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discrete+unsub...@googlegroups.com.
Scott Sanner
Apr 15, 2014, 9:27:23 PM4/15/14
to Andrey Kolobov, ippc-2014...@googlegroups.com
Hi Andrey,
This year I will calculate (and report) the total time used by each planner on the last 30 trials of each instance.
If the time for 30 trials on any instance exceeds 18 minutes, this will be flagged.
If this does happen, I suppose the fair resolution will be to use only the last trials whose time sums to less than 18 minutes... the remaining trials will be treated as if they were not attempted at all.
Cheers,
Scott
Scott Sanner
Apr 15, 2014, 9:33:56 PM4/15/14
to ippc-2014...@googlegroups.com, Marek Grzes
Hi Nicolas and Florent,
- Will there be benchmarks with intermediate variables?
Unfortunately not since we are following the requirements of the 2011 competition.
- If so, are there any plans to provide a translation to SPUDD that
works with intermediate variables?
Even though it won't be needed for the 2014 discrete competition, this is a fairly straightforward extension (of both the SPUDD language and the translator)... I'm happy for someone to contribute this if you want to contact me separately.
Cheers,
Scott
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.
Andrey Kolobov
Apr 15, 2014, 9:40:22 PM4/15/14
to Scott Sanner, ippc-2014...@googlegroups.com
Hi Scott,
My question was a bit different. Suppose that a planner makes 3 attempts, 30 trials each, to solve a problem. The first two sessions complete on time, but the third one gets terminated after 20 trials because the planner finally runs out of time (to be clear -- the planner manages to complete a total of 80 trials within the allocated 1080 seconds). In this situation, do the last 30 trials out of 80 count towards the planner's result, even though those 30 come from two different sessions?
Thanks,
Andrey
Scott Sanner
Apr 15, 2014, 9:55:33 PM4/15/14
to Andrey Kolobov, ippc-2014...@googlegroups.com
> In this situation, do the last 30 trials out of 80 count towards the planner's result, even though those 30 come from two different sessions?
Yes, I think this is OK. As long as the 30 trial time limit is verified (and enforced) by the evaluation script, I don't see this as a problem.
Cheers,
Scott
Reply all
Reply to author
Forward
0 new messages