GenerativeModels.jl
66 views
Skip to first unread message
Zachary Sunberg
Jan 26, 2016, 7:04:51 PM1/26/16
to julia-pomdp-users
Hi all,
In my work with POMDPs.jl, I have been working mostly with problems that are described by generative models. While the POMDPs.jl interface is certainly able to represent these problems, I have found that writing generative models using transition(), observation(), reward(), and rand() is awkward, time consuming, sometimes error prone, and potentially costly in terms of performance. It can easily take minutes or more to explain to someone how to implement a problem in this fashion, let alone debug an actual implementation, and it is possible that someone glancing at the POMDPs.jl documentation would conclude that their problem is not compatible with POMDPs.jl because it is difficult or impossible to express transition and observation distributions.
Moreover, solvers that only require a generative model such as MCVI and MCTS and its derivatives have to use several lines of code to simply generate the next state, observation, and reward. This decreases readability, and may impact performance or make it more time-consuming to write high performance code.
I think that these issues could be mitigated by introducing a new package called GenerativeModels.jl.
Before describing the package, let us define a few terms:
- PDF problem = a problem where the transition and observation probability distributions can be written explicitly
- GM problem = a problem that can be defined with a generative model (this includes all PDF problems)
- GM-only problem = a problem for which the transition and observation pdfs cannot be written explicitly; only a generative model is available
- PDF solver = a solver that uses explicit probability distributions when solving (SARSOP, for example, unless I am mistaken)
- GM solver = a solver that only requires a generative model (MCTS, POMCP, MCVI, DESPOT(?), etc.)
The main member of GenerativeModels.jl would be the following function:
(sp, o, r) = generate(p::POMDP, s::State, a::Action, rng::AbstractRNG)*
This function could be used in GM solvers instead of transition(), observation(), reward(), and rand(). GM problem writers would also have the option of simply writing the generate() function for their problem instead of transition(), observation(), reward(), and rand(). That way, GM problem writers and GM solver writers could `short circuit` their way out of ever having to deal with transition(), observation(), reward(), and rand().
Of course if a GM-only problem creator writes generate() instead of transition(), etc., she will not be able to use any PDF solvers. But, this is not a problem because GM-only problems are fundamentally incompatible with PDF solvers.
The package would also provide a default implementation of generate() for problems that have transition(), observation(), reward(), and rand() written for them so that all PDF problems can be solved by all GM solvers.** Of course PDF problem writers have the option of implementing both generate() and the other functions.
Edward proposed something like this earlier (Issue #19), and we decided to keep it out of the interface. I think now that the interface has matured and we have some experience writing problems and solvers it is time to think about adding this to the POMDPs ecosystem via this new package.
What do you all think? Should I move forward in creating this package?
- Zach
* (or (sp, r) = generate(p::POMDP, s::State, a::Action, rng::AbstractRNG) for MDPs). The function would also have the trailing optional arguments that we typically have in our interface functions for preallocation purposes.
** This may require the creation of a type e.g. GMWrapper with allocated distributions since it is inefficient to allocate distributions every time generate() is called.
In my work with POMDPs.jl, I have been working mostly with problems that are described by generative models. While the POMDPs.jl interface is certainly able to represent these problems, I have found that writing generative models using transition(), observation(), reward(), and rand() is awkward, time consuming, sometimes error prone, and potentially costly in terms of performance. It can easily take minutes or more to explain to someone how to implement a problem in this fashion, let alone debug an actual implementation, and it is possible that someone glancing at the POMDPs.jl documentation would conclude that their problem is not compatible with POMDPs.jl because it is difficult or impossible to express transition and observation distributions.
Moreover, solvers that only require a generative model such as MCVI and MCTS and its derivatives have to use several lines of code to simply generate the next state, observation, and reward. This decreases readability, and may impact performance or make it more time-consuming to write high performance code.
I think that these issues could be mitigated by introducing a new package called GenerativeModels.jl.
Before describing the package, let us define a few terms:
- PDF problem = a problem where the transition and observation probability distributions can be written explicitly
- GM problem = a problem that can be defined with a generative model (this includes all PDF problems)
- GM-only problem = a problem for which the transition and observation pdfs cannot be written explicitly; only a generative model is available
- PDF solver = a solver that uses explicit probability distributions when solving (SARSOP, for example, unless I am mistaken)
- GM solver = a solver that only requires a generative model (MCTS, POMCP, MCVI, DESPOT(?), etc.)
The main member of GenerativeModels.jl would be the following function:
(sp, o, r) = generate(p::POMDP, s::State, a::Action, rng::AbstractRNG)*
This function could be used in GM solvers instead of transition(), observation(), reward(), and rand(). GM problem writers would also have the option of simply writing the generate() function for their problem instead of transition(), observation(), reward(), and rand(). That way, GM problem writers and GM solver writers could `short circuit` their way out of ever having to deal with transition(), observation(), reward(), and rand().
Of course if a GM-only problem creator writes generate() instead of transition(), etc., she will not be able to use any PDF solvers. But, this is not a problem because GM-only problems are fundamentally incompatible with PDF solvers.
The package would also provide a default implementation of generate() for problems that have transition(), observation(), reward(), and rand() written for them so that all PDF problems can be solved by all GM solvers.** Of course PDF problem writers have the option of implementing both generate() and the other functions.
Edward proposed something like this earlier (Issue #19), and we decided to keep it out of the interface. I think now that the interface has matured and we have some experience writing problems and solvers it is time to think about adding this to the POMDPs ecosystem via this new package.
What do you all think? Should I move forward in creating this package?
- Zach
* (or (sp, r) = generate(p::POMDP, s::State, a::Action, rng::AbstractRNG) for MDPs). The function would also have the trailing optional arguments that we typically have in our interface functions for preallocation purposes.
** This may require the creation of a type e.g. GMWrapper with allocated distributions since it is inefficient to allocate distributions every time generate() is called.
Mykel Kochenderfer
Jan 26, 2016, 7:31:34 PM1/26/16
to julia-pomdp-users
I love the idea. POMDPs.jl would be the base layer, but your new package would make things easier. Kind of like PGF is the base, and TikZ is on top of it and makes things easier.
ebal...@stanford.edu
Jan 26, 2016, 8:59:15 PM1/26/16
to julia-pomdp-users
I am all for this, of course. I agree that doing generative models our current way may become cumbersome at some point. I put a convenience function in DESPOT that wraps transition(), observation(), reward(), rand(), etc and called it step(), but calling it generate() is fine too.
Reply all
Reply to author
Forward
0 new messages