Optimal Control using Dynamic Programming

[Narayani Vedam]

Hi,

  I am new to Julia. I need to solve an optimal control problem using dynamic programming. Are there pre-defined functions/packages that I could use?

Thanks.

[Uwe Fechner]

Hello,
as far as I know, there is no ready-made package for optimal control.
There is one package on control design:
https://github.com/JuliaControl/Control.jl

And there are a lot of packages for solving optimization problems.

Could you be a little bit more specific about your problem?

How do you want to describe the system, that you want to control?
Is it a linear system?

Uwe

[Tony Kelman]

The closest pre-made package for doing dynamic programming for optimal control would likely be one of the quantitative economics packages. If you have discrete decisions or want to grid your input space, it should be fairly straightforward to write out a cost-to-go matrix and apply the Bellman equation iteratively.

[Narayani Vedam]

The system is non-linear. I have a cost function, that I need to minimize to find the optimal control to move from one state to another. There are a few constraints that the states should satisfy even.

[NV]

Hi,

   I am trying to control the velocity of a system optimally. I have a cost function (DP formulation) to minimize subject to a few constraints.

Thanks

[Tony Kelman]

I'm biased, but for control problems with constraints I would pose it as a model predictive control problem and use JuMP.jl with the Ipopt.jl optimization solver. If you prefer to construct a cost to go map and do dynamic programming then I can't point you to any Julia code that I know of that solves your exact problem, though there may be some similar matlab code posted somewhere that you could translate. Without a complete mathematical or code description of your problem you're being too vague for this forum to help much.

[Narayani Vedam]

Thank you for your inputs. My concern was about calculating the cost to go map. I have discretized the state space and have 61 possible states in each stages and there are 20 such stages.

That roughly amounts to 61*61 values for one stage. I am overwhelmed by the amount of computations that this would mean and how best I could initialize a suitable array and index it for later use.

[Tony Kelman]

If you want an uninitialized 61-by-61 array of double precision floating point numbers you can do Array(61, 61)

If you want the array initialized to zero you can call zeros(61, 61)

A few for loops should accomplish what you're after here. Depending how expenaive your cost function is to evaluate this doesn't sound very big at all. How many state and input dimensions do you have?

[Narayani Vedam]

I have 3 states and two inputs. State 1 is [10:1:20], state 2 is [85:0.2:95] and state 3 is [1:1:12]. The two inputs are [0 1] and [-1 0].

[cdm]

you may find the Julia interface to apmonitor useful:

https://groups.google.com/forum/#!searchin/julia-opt/apmonitor/julia-opt/y5sc_xtLJfk/PKNBik_A82AJ

good luck,

cdm

On Thursday, September 24, 2015 at 10:25:19 PM UTC-7, NV wrote:

[Alex Dowling]

I'm also bias to recommend formulating a nonlinear programming problem and using JuMP with IPOPT. This approach has been successfully demonstrated in the chemical engineering controls/systems literature on some rather nasty large nonlinear systems.

Here is a sample problem for nonlinear control of a chemical reactor using the discretized NLP approach in GAMS:

http://cepac.cheme.cmu.edu/pasi2011/library/biegler/hicks.pdf

[Maxim Egorov]

If you formulate the problem as a Markov decision process, you can use POMDPs.jl which supports value iteration. The POMDPs.jl framework requires a functional problem definition, and some examples can be found in POMDPModels.jl.