Competition rules: taking account of translation time?

[Florent Teichteil]

Hi all,

The translation of RDDL format to SPUDD format is fine for most domains
except a few like traffic where the entire time allocated for each
round is not sufficient to translate an instance to SPUDD format. In
this case, our planner cannot even begin to solve the problem. From the
5th instance of traffic and above, translating the RDDL instance into
SPUDD format takes about 20 minutes.

If one considers that translation to a given format is part of the
planning algorithm, which would mean that the complexity of an MDP
problem relies on its specific encoding or that this encoding is an
integral part of the problem, then we acknowledge that our planner is
not able to tackle the traffic domain because of prohibitive
translation time to SPUDD format. We are not necessarily against this
viewpoint and we would simply give up the competition without regret
in this case.

In the opposite viewpoint, the translation time should not be taken into
account in the evaluation metric because a planner should be only
evaluated on its ability to solve a problem that is already defined in
its input format.

On the other hand, if one thinks that the representation format is an
integral part of the problem, then she could also argue that each
planner should be only compared against other planners that accept the
same input language. To the best of our knowledge, it has not been
demonstrated that RDDL-based planners outperform SPUDD-based planners
when problems are directly provided in the SPUDD format. Note that the
same comment holds for problems encoded in the PPDDL format. In our
opinion, there is a trade-off, which is important in many real-world
applications, between the expressiveness power of a language and the
efficacy of the solving algorithm for a given language.

We think that both viewpoints are perfectly tenable: there are many
cases where having a very expressive and compact language is important,
but also cases for which restrictive languages with dedicated planners
is advantageous. We are fine with both viewpoints! But the consequences
regarding our participation is not the same...

So finally the deliberately controversial question is: is it a
competition on MDP solving in general, or on RDDL-like MDP solving? :-)

What do you think?

All the best,
Florent and Nicolas.

[Scott Sanner]

Hi Florent and Nicolas,

As for the IPPC 2011, I will provide the SPUDD translation... you won't need to do it yourself.  Does that resolve your problem?

Cheers,

Scott

--
You received this message because you are subscribed to the Google Groups "IPPC 2014 (DISCRETE TRACK)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Florent Teichteil]

Excellent!

Actually, we're just realizing that you've already given the information on the competition's website. Sorry about this...
I'm afraid we missed this information due to the stress of the warmup competition :)

By the way, we are now switching to SPUDD's parser -- we directly used the ADDs generated by rddlsim instead of parsing SPUDD files. Now that we've just come to our senses, we need to modify our planner so that it directly uses SPUDD's parser. It won't conceptually change our planner but it may definitely introduce bugs. Would you mind if we could have a new small "warmup" slot on Amazon to test our new infrastructure?

Thanks,
Florent and Nicolas.

To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discrete+unsub...@googlegroups.com.

[Scott Sanner]

> By the way, we are now switching to SPUDD's parser

OK, one note I seem to recall from the IPPC 2011 is that the official SPUDD format does not support the initial state section but the SPUDD team modified their parser to handle this.  This should be in the "spudd competition version" on the SPUDD page linked from the IPPC 2011.

> Would you mind if we could have a new small "warmup" slot on Amazon to test our new infrastructure?

I think this should be fine... just let Marek know when you want to start.

Cheers,

Scott

To unsubscribe from this group and stop receiving emails from it, send an email to ippc-2014-discr...@googlegroups.com.

[Marek Grzes]

On 18/04/14 10:50, Florent Teichteil wrote:
> Would you mind if we could have a new small "warmup" slot on Amazon to
> test our new infrastructure?

Dear All,

The warm-up competition is over and all interested teams finished their
tests successfully. I will keep our server running for some more time in
case some of you would need more time to test your software. Please let
me know when you think you would need our server.

We will post the results in a few days too.

Thank you all for your participation and your contribution!

Marek