Forward and backward view of elegibility traces

[Davi Carnaúba]

With the assumption that a state is never revisited, can I say that the Forward and Backward view are equal even in the on-line case?

Best regards,

Davi

[Richard Sutton]

Davi, 

yes, i think so.  i am thinking that if a state is never revisited (in the tabular case without function approximation) then the on-line and off-line cases are the same by the end of an episode.  thus the forward and backward views will be equivalent in this case just as they are in the off-line case.

but note that now we have true online TD(lambda), which gives us the desired exact equivalence in the general linear case.

-rich

--
You received this message because you are subscribed to the "Reinforcement Learning Mailing List" group.
To post to this group, send email to rl-...@googlegroups.com
To unsubscribe from this group, send email to
rl-list-u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rl-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Reinforcement Learning Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rl-list+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Davi Vieira]

Thanks!

Best,

Davi

[Ashique Rupam Mahmood]

Hi Davi,

I believe you are referring to tabular TD(lambda). In that case, the claim is correct, as Rich mentioned. However, when we go beyond the tabular setting, for example in the linear function approximation setting, the online forward view and the backward view of TD(lambda) are not equivalent even in the case of no revisit. The algorithm that achieves this equivalence is called true online TD(lambda). Its equivalence holds whether a state is revisited or not.

Cheers,

Rupam