Using a simple particle filter from ParticleFilters.jl for a custom policy

55 views

Skip to first unread message

jacques.d...@gmail.com

unread,

Dec 7, 2017, 9:49:44 PM12/7/17

to julia-pomdp-users

Hi,

I am trying to implement a custom policy for my problem - I am calling this DemandResponse. I am using the heuristics.jl files in Powseeker and LaserTag problems as starting points. Once I have defined my policy, I try to use a simple particle filter as a belief updater, as in the Gallery example for the Powseeker problem. This will be used when simulating my policy.

The issue I am having is below. Essentially when trying to update the belief, the code is looking for an "action" method that operates on a ParticleCollection{DemandResponse.FacilityState} (FacilityState is the immutable I am using as a state variable). What I don't understand is that in the Powseeker problem implementation, such a function does not seem to be implemented (as shown by calling methods(action)), and yet the code runs. Obviously, I am missing something here, but it seems that in the Powseeker problem the action method is defined for a policy and SkierState, and then somehow automatically extended to a collection of SkierStates, whereas in my problem the action method is defined for a policy and FacilityState, but somehow does not extend automatically.

Any insights you may have into this behavior would be very helpful!

Error when running DemandResponse problem

$ julia

_ _ _(_)_ | A fresh approach to technical computing

(_) | (_) (_) | Documentation: https://docs.julialang.org

_ _ _| |_ __ _ | Type "?help" for help.

| | | | | | |/ _` | |

| | |_| | | | (_| | | Version 0.6.0 (2017-06-19 13:05 UTC)

_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release

|__/ | x86_64-apple-darwin13.4.0

julia> include(normpath(joinpath(pwd(),"DemandResponse.jl")))

DemandResponse

julia> using DemandResponse

julia> using POMDPToolbox

julia> using POMDPs

julia> using QMDP

julia> using SARSOP

julia> using ParticleFilters

julia> using BasicPOMCP

julia> p = DemandResponsePOMDP()

DemandResponse.DemandResponsePOMDP

perf_vals: Array{Float64}((10,)) [-100.0, -95.0, -90.0, -5.0, 0.0, 10.0, 15.0, 20.0, 20.0, 30.0]

T: Int64 30

maxSOC: Int64 10

julia> sim = HistoryRecorder(max_steps=30, rng=MersenneTwister(1), show_progress=true)

POMDPToolbox.HistoryRecorder(MersenneTwister(UInt32[0x00000001], Base.dSFMT.DSFMT_state(Int32[1749029653, 1072851681, 1610647787, 1072862326, 1841712345, 1073426746, -198061126, 1073322060, -156153802, 1073567984 … 1977574422, 1073209915, 278919868, 1072835605, 1290372147, 18858467, 1815133874, -1716870370, 382, 0]), [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 382), false, true, Nullable{Any}(), Nullable{Any}(), Nullable{Any}(30), Nullable{Integer}(), Any[], Any[], Any[], Any[], Float64[], Nullable{Exception}(), Nullable{Any}())

julia> pol4 = customPol()

DemandResponse.customPol()

julia> filter = SIRParticleFilter(p, 10_000, rng=MersenneTwister(7))

ParticleFilters.SimpleParticleFilter{DemandResponse.FacilityState,ParticleFilters.LowVarianceResampler,MersenneTwister}(DemandResponse.DemandResponsePOMDP

perf_vals: Array{Float64}((10,)) [-100.0, -95.0, -90.0, -5.0, 0.0, 10.0, 15.0, 20.0, 20.0, 30.0]

T: Int64 30

maxSOC: Int64 10

, ParticleFilters.LowVarianceResampler(10000), MersenneTwister(UInt32[0x00000007], Base.dSFMT.DSFMT_state(Int32[-999861353, 1073303822, 1577438924, 1073033480, -422137667, 1073598072, -1196125069, 1073464250, -454864994, 1072957280 … 596292345, 1073060802, 524882967, 1073401286, -862338375, -1617381160, 2103711644, 725776379, 382, 0]), [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 382), DemandResponse.FacilityState[], Float64[])

julia> hist = simulate(sim, p, pol4, filter)

ERROR: MethodError: no method matching action(::DemandResponse.customPol, ::ParticleFilters.ParticleCollection{DemandResponse.FacilityState})

Closest candidates are:

action(::POMDPToolbox.StochasticPolicy, ::Any) at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:18

action(::POMDPToolbox.CategoricalTabularPolicy, ::Any) at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:46

action(::POMDPToolbox.EpsGreedyPolicy, ::Any) at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:61

...

Stacktrace:

[1] simulate(::POMDPToolbox.HistoryRecorder, ::DemandResponse.DemandResponsePOMDP, ::DemandResponse.customPol, ::ParticleFilters.SimpleParticleFilter{DemandResponse.FacilityState,ParticleFilters.LowVarianceResampler,MersenneTwister}, ::DemandResponse.FSInit, ::DemandResponse.FacilityState) at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/simulators/history_recorder.jl:138

[2] simulate(::POMDPToolbox.HistoryRecorder, ::DemandResponse.DemandResponsePOMDP, ::DemandResponse.customPol, ::ParticleFilters.SimpleParticleFilter{DemandResponse.FacilityState,ParticleFilters.LowVarianceResampler,MersenneTwister}) at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/simulators/history_recorder.jl:76

Output from running similar code on Powseeker problem:

$ julia

_ _ _(_)_ | A fresh approach to technical computing

(_) | (_) (_) | Documentation: https://docs.julialang.org

_ _ _| |_ __ _ | Type "?help" for help.

| | | | | | |/ _` | |

| | |_| | | | (_| | | Version 0.6.0 (2017-06-19 13:05 UTC)

_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release

|__/ | x86_64-apple-darwin13.4.0

julia> using POMDPToolbox

julia> using POMDPs

julia> include("Powseeker.jl")

WARNING: deprecated syntax "typealias PowseekerProblem Union{PowseekerMDP,PowseekerPOMDP}" at /Users/jacquesdechalendar/Box Sync/Stanford Ed/AA228/project/gallery/Powseeker.jl/src/Powseeker.jl:94.

Use "const PowseekerProblem = Union{PowseekerMDP,PowseekerPOMDP}" instead.

Powseeker

julia> using Powseeker

julia> pomdp = PowseekerPOMDP()

Powseeker.PowseekerPOMDP{Powseeker.#grad_scaled_peaks2}

mdp: Powseeker.PowseekerMDP{Powseeker.#grad_scaled_peaks2}

dist_std_frac: Float64 0.3

grad_std: Float64 0.2

gps_std: Float64 50.0

compass_std: Float64 0.17453292519943295

julia> filter = SIRParticleFilter(pomdp, 10_000, rng=MersenneTwister(7))

ERROR: UndefVarError: SIRParticleFilter not defined

julia> policy = RandomlyCheckGPS(Topout(pomdp, 0.1), 0.2, MersenneTwister(12))

Powseeker.RandomlyCheckGPS{Powseeker.Topout{Powseeker.#grad_scaled_peaks2},MersenneTwister}(Powseeker.Topout{Powseeker.#grad_scaled_peaks2}(Powseeker.PowseekerMDP{Powseeker.#grad_scaled_peaks2}

topology: Powseeker.scaled_peaks2 (function of type Powseeker.#scaled_peaks2)

gradient: Powseeker.grad_scaled_peaks2 (function of type Powseeker.#grad_scaled_peaks2)

duration: Int64 60

dt: Float64 2.0

step_len: Float64 60.0

force: Float64 4.0

terminal_vel: Float64 50.0

max_flat_speed: Float64 4.0

psi_std: Float64 0.008726646259971648

vel_std: Float64 0.05

xlim: Tuple{Float64,Float64}

ylim: Tuple{Float64,Float64}

, 0.1), 0.2, MersenneTwister(UInt32[0x0000000c], Base.dSFMT.DSFMT_state(Int32[589013436, 1072892337, 1395473285, 1073017187, -346481041, 1072887813, 23263411, 1072776800, -1379277159, 1073038715 … -173461052, 1072708370, 859747322, 1072998707, -624312252, -2047062377, 1852138555, -1338826024, 382, 0]), [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 382))

julia>

julia> using ParticleFilters

julia> filter = SIRParticleFilter(pomdp, 10_000, rng=MersenneTwister(7))

ParticleFilters.SimpleParticleFilter{Powseeker.SkierState,ParticleFilters.LowVarianceResampler,MersenneTwister}(Powseeker.PowseekerPOMDP{Powseeker.#grad_scaled_peaks2}

mdp: Powseeker.PowseekerMDP{Powseeker.#grad_scaled_peaks2}

dist_std_frac: Float64 0.3

grad_std: Float64 0.2

gps_std: Float64 50.0

compass_std: Float64 0.17453292519943295

julia> sim = HistoryRecorder(rng=MersenneTwister(94), max_steps=60, show_progress=true)

POMDPToolbox.HistoryRecorder(MersenneTwister(UInt32[0x0000005e], Base.dSFMT.DSFMT_state(Int32[654715864, 1073461659, -1154176253, 1073038371, -30180749, 1072778016, -255639611, 1073641004, 1361453792, 1073343733 … 2098551036, 1073590072, -361676843, 1073366619, 559736570, 871537037, 2078045984, -1765932767, 382, 0]), [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 382), false, true, Nullable{Any}(), Nullable{Any}(), Nullable{Any}(60), Nullable{Integer}(), Any[], Any[], Any[], Any[], Float64[], Nullable{Exception}(), Nullable{Any}())

julia> hist = simulate(sim, pomdp, policy, filter)

julia> methods(action)

# 17 methods for generic function "action":

action(policy::POMDPToolbox.StochasticPolicy, b::Void) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:22

action(policy::POMDPToolbox.StochasticPolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:18

action(policy::POMDPToolbox.CategoricalTabularPolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:46

action(policy::POMDPToolbox.EpsGreedyPolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/stochastic.jl:61

action(p::POMDPToolbox.FunctionPolicy, x) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/function.jl:25

action(p::POMDPToolbox.FunctionPolicy, x, a) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/function.jl:26

action(p::POMDPToolbox.VectorPolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/vector.jl:12

action(p::POMDPToolbox.VectorPolicy, s, a) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/vector.jl:13

action(p::POMDPToolbox.ValuePolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/vector.jl:47

action(p::POMDPToolbox.ValuePolicy, s, a) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/vector.jl:48

action(policy::POMDPToolbox.RandomPolicy, b::Void) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/random.jl:22

action(policy::POMDPToolbox.RandomPolicy, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/random.jl:18

action(p::POMDPToolbox.PolicyWrapper, s) in POMDPToolbox at /Users/jacquesdechalendar/.julia/v0.6/POMDPToolbox/src/policies/utility_wrapper.jl:44

action(p::Powseeker.Downhill, s::Powseeker.SkierState) in Powseeker at /Users/jacquesdechalendar/Box Sync/Stanford Ed/AA228/project/gallery/Powseeker.jl/src/heuristics.jl:7

action(p::Powseeker.Topout, s::Powseeker.SkierState) in Powseeker at /Users/jacquesdechalendar/Box Sync/Stanford Ed/AA228/project/gallery/Powseeker.jl/src/heuristics.jl:19

action(p::Powseeker.SkiOver, s::Powseeker.SkierState) in Powseeker at /Users/jacquesdechalendar/Box Sync/Stanford Ed/AA228/project/gallery/Powseeker.jl/src/heuristics.jl:34

action(p::Powseeker.RandomlyCheckGPS, b) in Powseeker at /Users/jacquesdechalendar/Box Sync/Stanford Ed/AA228/project/gallery/Powseeker.jl/src/heuristics.jl:53

julia>

Zachary Sunberg

unread,

Dec 8, 2017, 2:04:12 AM12/8/17

to julia-pomdp-users

In the Powseeker.jl test/heuristics.jl script, the problem is just a Powseeker*MDP* not a POMDP, so the policies operate on SkierStates. You would have to use the RandomlyCheckGPS, which has an action function that will work on a ParticleCollection if you wanted to run it on a POMDP.

It should be pretty easy to extend a policy that works on a state to a ParticleCollection. You can call rand or mode (or mean if the proper operators are defined for your states) on the ParticleCollection to get a state, or you can get a vector of states in the particle collection with particles(b). Note all the particles in a ParticleCollection have the same weight.

Reply all

Reply to author

Forward

0 new messages