Step size Selection in Reinforcement Learning

[rajkumarmaity]

Hi,

I have seen in standard RL literatures that constant step-size is chosen in experimental  validation of RL algorithms. However, such algorithm's theoretical convergence is guaranteed only if step-size  satisfies Robbins-Monro conditions. Suppose I have an algorithm which converge with step-size satisfying these conditions (both theoretically and experimentally), but it does not converge with constant step-size (it should converge in distribution only theoretically), then how is the constant step-size method is justified for experimental scenarios ?

Thanks

[Sergio Valcarcel Macua]

Hi,

A number of theoretical results on almost sure convergence for
diminishing step-size can be extrapolated to convergence in probability
for sufficiently small constant step-size.

In other words, if diminishing step-size ensures (with probability one)
asymptotic convergence to the stationary point, then small step-size
usually ensures convergence to a ball around the stationary point.

My intuitive reason is that diminishing step-size averages the noise
completely so that you get to the stationary point, while constant
step-size only averages the noise up to the same order of magnitude of
the step-size.

Best
Sergio

> --
>