Re: [rl-list] Reward Shaping

[Carlos Diuk]

Hi Fatemeh,

There seem to be two different problems. The fact that you get Inf values might indicate that there's simply a bug in your code: are you using a proper discount factor? If you're doing tabular Q-learning with reward discounting and a discount factor \gamma, and the maximum immediate reward in your environment is Rmax, the maximum value in your table should be Rmax/(1-\gamma). 

That being said, it's also possible that you're not doing shaping properly. Are you using potential-based shaping? 

http://ai.stanford.edu/~ang/papers/shaping-icml99.pdf

You might also want to look at:

http://www.aaai.org/Papers/JAIR/Vol19/JAIR-1907.pdf

http://paul.rutgers.edu/~jasmuth/pub/aaai08-shaping.pdf

Good luck!

--Carlos

On Thu, May 9, 2013 at 3:47 AM, Fatemeh Sarani Rad <f.sara...@gmail.com> wrote:

Dear all,

I work on Reward Shaping in MDP model. I chose GridWorld problem as a test bed and I changed Reward function based on environment [1]. I have problem about Inf values that appear in Q-value results. I would appreciate if someone has similar experiences about Reward Shaping and inform me.

 

[1] Norouzzadeh Sholeh,“Shaping Methods to Accelerate Reinforcement Learning: From Easy to Challenging Tasks”, A Thesis of Delft University of Technology, 2010.

 

Fatemeh Sarani Rad

Knowledge Engineering Lab

University of Economics Science

Tehran Iran

 

--
--
You received this message because you are subscribed to the "Reinforcement Learning Mailing List" group.
To post to this group, send email to rl-...@googlegroups.com
To unsubscribe from this group, send email to
rl-list-u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rl-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Reinforcement Learning Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rl-list+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Marek Grzes]

When it comes to the use of domain knowledge to speed up RL, reward
shaping has convenient theoretical properties. However, sometimes the
same knowledge can be used in an alternative way and lead to better
improvements, see, e.g.
http://www.ifaamas.org/Proceedings/aamas2010/pdf/01%20Full%20Papers/06_06_FP_0313.pdf
or
http://webloria.loria.fr/~maraya/html/wiki/lib/exe/fetch.php?media=araya12icml-extended.pdf
Overall, it seems to me that if there is no way to inject domain
knowledge directly into the learning algorithm, reward shaping is
probably the best solution.

Cheers,
Marek

On 14/05/13 08:28, Sam Devlin wrote:
> If you are interested in potential-based reward shaping (or anyone else
> reading), I would also recommend:
>
> An extension of Ng's original single agent proof to model-based RL:
> J. Asmuth, M. Littman, and R. Zinkov. Potential-based shaping in
> model-based reinforcement learning.
> In Proceedings of the Twenty-Third AAAI Conference on Artificial
> Intelligence, pages 604�609, 2008.
> Available here: http://www.aaai.org/Papers/AAAI/2008/AAAI08-096.pdf
>
> An extension to allow state and action based shaping:
> E. Wiewiora, G. Cottrell, and C. Elkan. Principled methods for advising
> reinforcement learning agents.
> In Proceedings of the Twentieth International Conference on Machine
> Learning, 2003.
> Available here: http://www.aaai.org/Papers/ICML/2003/ICML03-103.pdf
>
> Our extension to the multi-agent case:
> S. Devlin and D. Kudenko. Theoretical considerations of potential-based
> reward shaping for multi-agent systems.
> In Proceedings of The Tenth Annual International Conference on Autonomous
> Agents and Multiagent Systems, 2011.
> Available here:
> http://www.ifaamas.org/Proceedings/aamas2011/papers/D1_G45.pdf
>
> And the dynamic case, which proves the same guarantees can hold even if the
> potential function changes online:
> Devlin S., and Kudenko D.: *Dynamic Potential-Based Reward Shaping.*
> *In Proceedings of the 11th International Conference on Autonomous Agents
> and Multiagent Systems (AAMAS)*, 2012.
> Available here: http://www.ifaamas.org/Proceedings/aamas2012/papers/2C_3.pdf
>
> Kind regards,
> Sam.

>
> On Thursday, May 9, 2013 11:13:43 AM UTC+1, Carlos Diuk wrote:
>> Hi Fatemeh,
>>
>> There seem to be two different problems. The fact that you get Inf values
>> might indicate that there's simply a bug in your code: are you using a
>> proper discount factor? If you're doing tabular Q-learning with reward
>> discounting and a discount factor \gamma, and the maximum immediate reward
>> in your environment is Rmax, the maximum value in your table should be
>> Rmax/(1-\gamma).
>>
>> That being said, it's also possible that you're not doing shaping
>> properly. Are you using potential-based shaping?
>> http://ai.stanford.edu/~ang/papers/shaping-icml99.pdf
>>
>> You might also want to look at:
>> http://www.aaai.org/Papers/JAIR/Vol19/JAIR-1907.pdf
>> http://paul.rutgers.edu/~jasmuth/pub/aaai08-shaping.pdf
>>
>> Good luck!
>> --Carlos
>>
>>

>> On Thu, May 9, 2013 at 3:47 AM, Fatemeh Sarani Rad <f.sara...@gmail.com<javascript:>

>>> wrote:
>>> Dear all,
>>>
>>> I work on Reward Shaping in MDP model. I chose GridWorld problem as a
>>> test bed and I changed Reward function based on environment [1]. I have
>>> problem about Inf values that appear in Q-value results. I would appreciate

>>> if someone has similar experiences about Reward Shaping and inform me.***
>>> ***
>>>
>>> ** **
>>>
>>> [1] Norouzzadeh Sholeh,�Shaping Methods to Accelerate Reinforcement
>>> Learning: From Easy to Challenging Tasks�, A Thesis of Delft University
>>> of Technology, 2010.****
>>>
>>> ** **
>>>
>>> Fatemeh Sarani Rad****
>>>
>>> Knowledge Engineering Lab ****
>>>
>>> University of Economics Science****
>>>
>>> Tehran Iran****
>>>
>>> ** **

>>>
>>> --
>>> --
>>> You received this message because you are subscribed to the
>>> "Reinforcement Learning Mailing List" group.

>>> To post to this group, send email to rl-...@googlegroups.com<javascript:>

>>> To unsubscribe from this group, send email to

>>> rl-list-u...@googlegroups.com <javascript:>

>>> For more options, visit this group at
>>> http://groups.google.com/group/rl-list?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "Reinforcement Learning Mailing List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to rl-list+u...@googlegroups.com <javascript:>.