Rigorous formulation of RL for episodic MDP

[Sergio]

Hello,

I just realized about another subtle question.

By an episodic MDP I have understood so far that there is a non-empty
set of terminal (absorbing) states, and that the number of steps before
reaching this terminal state may be random, therefore, we can handle the
random-horizon MDP as a kind of infinite horizon problem and the fixed
point Bellman equation still makes sense.

The problem that I find is that in most of the examples I have seen, the
discount factor \gamma (i.e., complement to the termination probability)
is set to 1 for all the states, except the terminal ones.
Therefore, the matrix (I - \gamma P) could be singular! But that matrix
to be singular does not make sense!

Is that right?
Could you suggest me any rigorous treatment on episodic MDP?

Thank you very much!
Best,
Sergio








--
"Live quietly in the moment and see the beauty of all before you. The future will take care of itself......"
� Paramahansa Yogananda (Autobiography of a Yogi)