Packing and unpacking parameters

[Yuuki Soho]

It's a bit of a stupid question, but I don't really know how to deal with this efficiently.

So, in many application I have some model with parameters, and I want to the able to change the number of parameters, or they order easily.

For passing parameters to functions I want to pack them into a vector p, such that I don't have huge function definition, but inside

the function's body I'd prefer to have all the parameters given by their name, so I can use them in equations (instead of using p[1], p[2], ...).

I can write two functions p = pack(a,b,c) and (a,b,c) = unpack(p) but that's pretty restrictive because if you add or remove a parameters, I have to change all 

my function calls and definition. If I add another model I also need to write another pack and unpack pairs.

Is there an better approach to do this in Julia ? I was thinking maybe doing a macro @unpack p that would spawn all the variables needed, but I'm not

sure that's the right way to do it.

[Ivar Nesje]

Do you mean like

a, b, c = 1:3

string((1:3)...)

Ivar

[Patrick O'Leary]

Have you looked into using keyword arguments? http://julia.readthedocs.org/en/latest/manual/functions/#keyword-arguments

[Yuuki Soho]

Basically I want to define only once the name of my parameters, and never write them down in function's arguments or declare them in function's body,

such that I can add, remove, rename, reorder parameters without having to do anything, and still have all my parameters available in my functions body.

I was thinking of something like this: I define a list of variables name n = ["a","b","c"] in my main function

and then having a macro @unpack n p that would generate:

a = p[1]

b = p[2]

c = p[2]

Such that I can do something like :

function F1(n,p)

@unpack n p

return a*cos(b-c)

end 

Does that make any sense, or I am missing something obvious ? 

[Cristóvão Duarte Sousa]

How amazing! I have been about to post the very same question to this list! So, I wouldn't say it's a stupid question at all :)

I believe this kind of packing/unpacking of several heterogeneous (in structure, not in type) parameters appear a lot in modeling and optimization.

This probably is a question transversal to many (all) programming languages, but after porting some code of mine from Python to Julia this issue is now hitting me as the major bottleneck.

I need to pack several parameters which are scalars, vectors and symmetric matrices of fixed size.

My first attempt was to define a generic packing function to map from a Dict of symbol=>parameter to the "vector form" which used a pack function dispatched on the parameters type. The vector form was exposed externally to the optimization routines and the Dict form was used internally in the model. The problem of that format approach was that Julia base types were not expressive enough, for example, to know the size of a Vector parameter or to know that a Matrix was symmetric and thus only a set of its unique elements must be packed to the vector form.

I then thought about creating my own Types, but that would represent a huge task not worth it. If it were possible to delegate methods from a type to one of its fields (https://github.com/JuliaLang/julia/pull/3292) that would be a much  easier task:

    immutable SymmetricMatrix{T}

        a::Matrix{T}

    end

    @somehow_delegate_methods SymmetricMatrix --> SymmetricMatrix.a

I ended up defining a 'format' array made of (Symbol, Symbol) pairs were the first symbol is the parameter name and the second symbol is just a label indicating the type of the parameter over which some unpack/pack are dispatched (if interested see https://github.com/cdsousa/Robotics.jl/blob/master/src/dynparams.jl).

Anyway I still have a feeling that this is suboptimal, and I wonder what are the common practices in other languages.

I even did a rough search on StackOverflow but found nothing so far...

[Mike Innes]

RE some kind of `@unpack` macro: have a look at this question on Stack Overflow and its answers:

http://stackoverflow.com/questions/9345056/in-clojure-how-to-destructure-all-the-keys-of-a-map/11182432#11182432

Interestingly enough it is possible to an extent, but for many reasons ends up being a bad idea.

What you probably want instead is the Dictionary type - see below. Using it, you can write params[:a], params[:b] etc.

http://docs.julialang.org/en/release-0.2/stdlib/base/#indexable-collections

That said, it would be nice to have some kind of map destructuring macro similar to Clojure's. Hopefully the pattern matching libraries will eventually include something along those lines.

Hope this helps,

Mike

[Toivo Henningsson]

Maybe I don't understand the problem well enough, but I think that an immutable type could work pretty well. You will have to qualify your references to the members, but you only have to declare the names and types once. I think it was discussed to create default constructors with keyword arguments that correspond to the field names but I'm not sure if we have it yet. Then, you could refer to the parameters by name everywhere.

[Cristóvão Duarte Sousa]

By reading your question once again, I realised that it may be not quite like mine.

If you are ok with using p.a, p.b, p.c, ... then the custom composite type is the solution.

A more flexible approach (as I do) is the dictionary with symbols as keys which would then be called by p[:a], p[:b], p[:c]...

It is expected that in future the dot operator is going to be overloadable (https://github.com/JuliaLang/julia/issues/1974), allowing the creation of dictionary types whose symbol keys can be accessed as p.a, p.b, p.c...

If you strictly want to call parameters by a, b, c... then maybe some macro can solve it, about this I cannot help.

On the other hand, I really need p to be a Vector, so it can be seen as such by optimisation and linear algebra routines. In that case some packing/unpacking functions need to be called and probably some copies will always have to happen. What I'm looking for is way to have generalized and yet well performing packing/unpacking functions, flexible enough to allow easy addition/reorder of parameters.

Please let me spoil a little the OP, and ask if someone can point some insights on my problem.

On Friday, March 7, 2014 2:15:58 PM UTC, Yuuki Soho wrote:

[andrew cooke]

are you guys looking for something like attach in R?

(it's not clear to me what you want, although a reference to python sounds like it might have been stickng values into an object's dict).

andrew

[James Porter]

I've struggled with this exact problem in python in the past (e.g. https://stackoverflow.com/questions/16182898/unpacking-parameters-for-a-simulation). It's exacerbated by the fact that interfaces to solvers, optimizers, etc. often require the parameters be passed in as a vector, so using dictionary won't help. Dictionary access in tight loops also has obviously bad performance implications. My intuition is that a macro is the way out, something like:

@unpack params a1 a2 a3 b1 b2 C  # etc . . .

as you suggest. The tricky bit is implementing it in a way that avoids unnecessary memory allocation. I'm not sure how good the LLVM optimization passes are but I suspect that doing it as you suggested previously (e.g. translating the above into a1 = params[1], a2 = params[2], etc.) will result in unnecessarily copying things onto the stack from the params array.

—James

On Friday, March 7, 2014 8:15:58 AM UTC-6, Yuuki Soho wrote:

[John Myles White]

For now, I suspect the easiest way to do this is to switch back and forth between immutable types and vectors by using linear indexing in the fields of a type, then linear indexing within each field. Here’s a specific example of how a generic function might work:

immutable Foo

a::Vector{Float64}

b::Matrix{Float64}

end

f = Foo(ones(3), 2 * ones(4, 4))

v = Array(Float64, 3 + 4 * 4)

i = 0

for fieldindex in 1:2

field = f.(fieldindex)

for fieldelement in field

i += 1

v[i] = fieldelement

end

end

Writing a generic function to do this isn’t much more work.

 — John

[Cristóvão Duarte Sousa]

Great approach John! I was not aware of the linear indexing of type fields, that opens a lot of possibilities. Thanks

[Yuuki Soho]

I'm trying to write an unpack macro, just to learn a bit about meta-programming (I get it's probably not the best idea,

but sometimes you learn a lot doing stupid things), but I have to say than even after reading three times the doc,

I have no idea how to do it. I wanted to do something like that:

n = [:a,:b]

p = [1 2]

@unpack n p

being transformed to:

a = p[1]

b = p[2]

But that doesn't seem to be possible because the unpack macro just get the expressions "n" and "p", so there's not much you can do there. 

It seems I need an unpack function first to create the correct expression:

@setvariables unpack(n,p)

Where unpack unpack(n,p) generate the expression "a b 1 2", any ideas how I can do that ?

[John Myles White]

unpack can just look at its inputs and build up Expr() objects piece-by-piece. Here’s an example of building up some indexing code from your first example:

function makeindex(name::Symbol, indices::Vector{Int})
n = length(indices)
res = Array(Expr, n)
for i in 1:n
res[i] = Expr(:ref, name, indices[i])
end
return res
end

julia> makeindex(:p, [1, 2])
2-element Array{Expr,1}:
:(p[1])
:(p[2])

But, as you’ve said, this is a pretty strange use of metaprogramming since you’re trying to work with values, even though macros should mostly operate on syntax. In addition, you seem to want to work with the value of p and the symbolic name p at the same time, which is particularly complicated.

One way to think about macros that helped me a lot is to envision them all taking place during a single compile-time pass through your codebase. As such, they should never depend on the value of anything that isn’t statically available before your program runs. This restriction isn’t actually a real one, but it may help push you in the right direction.

— John

[Yuuki Soho]

Is it possible to do something like defining a piece of code, and then just insert it when needed ? Something like:

@insertcodehere somecode

[John Myles White]

I think that depends on what you mean. Here's an example of a "complex" macro expansion that's just shorthand for a longer block of code.

macro foo()
quote
println("I'm the first line")
println("I'm the second line")
println("I'm the third and final line")
end
end

julia> @foo
I'm the first line
I'm the second line
I'm the third and final line

-- John

[Mike Innes]

I'm trying to write an unpack macro, just to learn a bit about meta-programming (I get it's probably not the best idea,

but sometimes you learn a lot doing stupid things)

 Absolutely agreed! But bear in mind that this may be a hard macro to write in the general case. Nevertheless, I can give you some pointers.

Firstly, @unpack n p isn't going to work in the way you want, because the value of n will not be known (in principle) until runtime. It is possible - the way to do it is to construct and evaluate a let binding, but doing this every time the code runs will be extremely slow. Interpreted languages like R sometimes offer these kinds of features because they avoid the compiler overhead. You could also call eval(n) within the macro but that's very brittle.

On the other hand, as long as you aren't going to change the value of n at run time, you can reference it from within the macro itself - something like (not tested)

const n = [:a, :b, :c]

macro unpack(p)

  Expr(:block, [:($(n[i]) = $p[$i]) for i = 1:length(n)])

end

@unpack [1,2,3] # Now a = 1, b = 2, etc.

The problem with this, then, is generalising it; you don't want to define this macro for every set of parameters, after all. But if you want to avoid that, you'll need to write a macro that writes a macro - I'll leave that as an exercise for the reader.

[Yuuki Soho]

That was a slightly stupid question John, I should have thought about it 2 minutes:)

I was hopping to do something like that, but it doesn't work because of the hygiene I guess.

macro unpack()

quote

a = p[1]

b = p[2]

end 

end

macroexpand(:(@unpack))

#184#a = p[1] # line 1: #185#b = p[2]

This is what I get with yours Mike:

@unpack [1,2,3]

3-element Array{Expr,1}: :(a = [1,2,3][1]) :(b = [1,2,3][2]) :(c = [1,2,3][3])

Thanks for the answers!

[John Myles White]

Stupid seems a little harsh. Maybe just a bit vague. :)

 -- John

[Mike Innes]

macro unpack(p)

  Expr(:block, [:($(n[i]) = $p[$i]) for i = 1:length(n)]...) |> esc

end

Ok, that should work better. To avoid the hygiene pass you can wrap everything in esc; for example

macro unpack()

  quote

    a = p[1]

    b = p[2]

  end |> esc

end

[Billou Bielour]

Awesome, it works now:

const pnames = [:p1, :p2, :d1, :d2, :K1, :K2, :d3, :d4, :s1, :s2, :g1, :g2]

macro unpack(ex)
  Expr(:block, [:($(pnames[i]) = $ex[$i]) for i = 1:length(pnames) ]...) |> esc
end

macro pack(ex)
    N = length(pnames)
    Expr(:block, [

:($ex = zeros($N,1)),
 [:( $ex[$i] = $(pnames[i]) ) for i = 1:length(pnames) ]
]...) |> esc

end

I can use it like that:

    dt = 1e-3
    p1 = 24.0
    p2 = 20.0

    d1 = 1.0
    d2 = 0.5

    K1 = 2.00
    K2 = -1.0

    d3= 4.
    d4= 4.
    s1=0.1
    s2=0.2

    g1=0.5
    g2=0.5

    Nt = 10^5

    @pack p

    th, phi, idx_th, idx_phi = doSimulation1(Nt,dt,p)

and then in doSimulation1:

function doSimulation1(Nt::Int,dt::Float64,p)

    @unpack p

    w1 = dt*2*pi/p1
    w2 = dt*2*pi/p2

    ...

It works only for scalars, and it's not really useful for functions that get called a lot because it allocates memory, but it's still pretty neat.