Help with writing a macro to for fetch, process, store pattern

[Andrew Simper]

A lot of the time it is good to copy a structure to local variables and then process on those for efficiency before storing the local values back to a structure. To help out with this I'm trying to write  a macro, so this is what I would like the end result to be:

Point

    x::Float64

    y::Float64

end

function process (p::Point)

    local x = p.x;

    local y = p.y;

    # do some processing on x and y

    p.x = x

    p.y = y

end

and I would like write a macro that does this so the end code would like like:

function process (p::Point)

    @fetch p

    # do some processing on x and y

    @store p

end

So far I've got this working at the REPL using:

point = Point (1, 2)

map (eval, ([:($name = point.$name) for name in names(point)]))

println("x=$(x) y=$(y)")

which prints out: x=1.0 y=2.0

Can someone please help out turning this into a macro?

[Jameson Nash]

A macro can't do this because a macro can't see types. But I don't need a macro to get an even more efficient version of your attempted optimization. I just add the immutable keyword:

immutable Point

    x::Float64

    y::Float64

end

[Andrew Simper]

Yep, thanks for pointing that out :) But I'm not sure that helps with what I'm saying, since inside a function I won't be able to access the variables x and y directly without using dot notation:

function distance (point::Point)

    # now you have to use point.x and point.y which is quite obfuscated eg:

    sqrt (point.x*point.x + point.y*point.y)

end

when really I would like to be able to type ( https://groups.google.com/forum/#!topic/julia-users/UvBff9QVKaA ):

function distance (point::point)

    using point

    sqrt (x*x + y*y)

end

or

function distance (point::Point)

    @fetch point

    sqrt (x*x + y*y)

end

For the above cases this isn't a big deal, but I have algorithms that require 30 to 100 so state variables like this the entire algorithm starts becoming unreadable since all the eye can see is point.this and point.that not just this and that. 

[Ismael VC]

How about this?

julia> function distance(point::Point)c

         x = point.x

         y = point.y

         sqrt(x*x + y*y)

       end

[Andrew Simper]

Yep, an automated way of doing that is exactly what I want. Here is an example structure that I want to fetch every member of before doing intensive numeric processing for an iterative solver:

type CircuitModel

    v1::Float64

    v5::Float64

    v7::Float64

    v2::Float64

    v3::Float64

    v4::Float64

    v9::Float64

    v12::Float64

    v15::Float64

    v18::Float64

    v19::Float64

    v20::Float64

    v21::Float64

    v8::Float64

    v22::Float64

    v23::Float64

    v24::Float64

    i1::Float64

    i2::Float64

    i3::Float64

    i4::Float64

    i5::Float64

    i6::Float64

    i7::Float64

    ic1eq::Float64

    ic2eq::Float64

    ic3eq::Float64

    ic4eq::Float64

    ic5eq::Float64

    ic6eq::Float64

    ic7eq::Float64

    ic8eq::Float64

    sr::Float64

    srinv::Float64

    pi::Float64

    gmin::Float64

    is1::Float64

    nvtf1::Float64

    is2::Float64

    nvtf2::Float64

    nvtinvf1::Float64

    vcritf1::Float64

    nvtinvf2::Float64

    vcritf2::Float64

    gc1::Float64

    gr1::Float64

    gr2::Float64

    itxr2::Float64

    gc2::Float64

    gc3::Float64

    gc4::Float64

    gc5::Float64

    gr7::Float64

    gc6::Float64

    gr3::Float64

    itxr3::Float64

    gc7::Float64

    gr4::Float64

    gc8::Float64

    gr5::Float64

    vpos::Float64

    vneg::Float64

    gin::Float64

    gininv::Float64

    vposcap::Float64

    vnegcap::Float64

    ginotacore::Float64

    ginotares::Float64

    ginotacoreinv::Float64

    ginotaresinv::Float64

    vc3lo::Float64

    vc3hi::Float64

    a4a4c::Float64

    a5a5c::Float64

    a6a6c::Float64

    a14a14c::Float64

    a16a16c::Float64

    a17a17c::Float64

    a17a17nrmc::Float64

    a12a17c::Float64

    a16a16nrmc::Float64

    a15a16c::Float64

    a14a14nrmc::Float64

    a13a14c::Float64

    a6a14c::Float64

    a13a6c::Float64

    a5a5nrmc::Float64

    a4a5c::Float64

    a2a5c::Float64

    a2a4c::Float64

    a4a4nrmc::Float64

    a1a4c::Float64

    v5c::Float64

    v7c::Float64

end

function process1 (state::CircuitModel, input::Float64)

    @fetch state

    # loop over an algorithm using all the named items in state to solve for the next input changing the state

   @store state

end

function process1 (state::CircuitModel, input::Float64)

    @fetch state

    # loop over an algorithm using all the named items in state to solve for the next input changing the state

   @store state

end

function process2 (state::CircuitModel, input::Float64)

    @fetch state

    # loop over an algorithm using all the named items in state to solve for the next input changing the state

   @store state

end

function process3 (state::CircuitModel, input::Float64)

    @fetch state

    # loop over an algorithm using all the named items in state to solve for the next input changing the state

   @store state

end

So I don't want to have actual duplicate code doing all the copying of the state to local variables since then if I add a new variable to the type then I have to update the copy to local variables in all 3 functions, so this makes things a bit brittle.

A better option would be to support a "using" statement, along the lines of:

function process4 (state::CircuitModel, input::Float64)

    using state;

    # loop over an algorithm using all the named items in state to solve for the next input changing the state

end

But I was hoping I could write a macro as a workaround for now, but it seems macros can't handle what I want.

On Monday, June 9, 2014 7:55:54 AM UTC+8, Ismael VC wrote:

How about this?

julia> function distance(point::Point)

[Nathaniel Nicandro]

You can use the esc function that can introduce variables in the calling environment

julia> type Point

           x

           y

       end

julia> macro fetch(p)

           variables = quote

               x = $p.x

               y = $p.y

           end

           return esc(variables)

       end

julia> p = Point(5, 10)

Point(5,10)

julia> x

ERROR: x not defined

julia> y

ERROR: y not defined

julia> @fetch p;

julia> x

5

julia> y

10

[Andrew Simper]

Yes thanks for pointing that out, but if you look at this thread I already got some help with the |> esc for this type of thing over on another thread https://groups.google.com/forum/#!topic/julia-users/UvBff9QVKaA , but I started this thread specifically to find out how to get write a macro using names(obj) to do the work instead of having to write a new fetch macro manually every time I create a new type. 

[Stefan Karpinski]

Any time there's an eval in what your macro emits, something's probably wrong. In this case, however, you can't really avoid it – largely because you can't do this correctly. The trouble is that the type of the object you want to bind the fields of is a runtime property, but you want to bind variables based on it – which is a compile-time property.

On Mon, Jun 9, 2014 at 12:55 PM, Nathaniel Nicandro <nathanie...@gmail.com> wrote:

Oh, yes I see that now. Then what you could do is quote the call to names() in your macro expression so the names call is executed in the calling environment. 

macro fetch(p)

    return quote

        local vars = names($p)

        local ex = Expr(:block)

        append!(ex.args, [:($v = $p.$v) for v in vars])

        eval(ex)

    end

end

This is similar to what yo have done at the REPL but it quotes the names() call so it can get evaluated in the macro calling environment instead of within the macro itself. ex =  Expr(:block) is just another quote block with a list of expressions to evaluate in ex.args. Note that there is no need to use esc() here because the eval() call happens in the calling environment.

[Jameson Nash]

> something's probably wrong.

Like this macro assuming you want to disable optimization of your code and only work in the global scope without modules.

I maintain my recommendation of using an immutable type (which allows llvm to do some additional optimizations) and making a single local variable with a shorter name for typing repeatedly. 

The only feature somewhat unique to Visual Basic that I liked is the with block , which is similar to what you are asking. I would love to have this syntax in Julia too, but I'm not sure it is possible/a good idea. Since it is just a source transform, it should avoid Stefan's note about the compile/runtime distinction error. 

[yu...@altern.org]

I think that adapting the macro at the end of this thread: https://groups.google.com/forum/#!searchin/julia-users/unpack/julia-users/IQS2mT1ITwU/gEyj6JNJsuAJ

and getting the fields of your type by using names(CircuitModel) it should work. I'll give it a try tomorrow if I have the time.

[Nathaniel Nicandro]

Yes I realize now that I jumped the gun on my post. It would definitely be very useful if you could reduce this code

x = point.x
y =

point.y
# do stuff on x and y
point.x = x
point.y = y

by something like

@fetch_and_store(Point, :foo, quote
   #do stuff on x and y
end)

p = Point(1, 0)
foo(p)

Where @fetch_and_store creates a function foo(point::Point) that handles all of the setup and storing of variables, but this still requires the type information of Point.  

[Andrew Simper]

Hi Stefan,

I agree I would rather use an immutable type, what I am trying to work around is the lack of support for some kind of "using" language feature similar to how namespaces work but for instances of a user defined composite type, that would be ideal. Then I can avoid all the obfuscating dot notation to access variables.

type Typename

    a

    b

end

# this would be ideal:

function calculate (value::Typename)

    using value

    a = 1 # complicated algorithm

    b = 2 # complicated algorithm

    return value

end

# warning / error doing this:

function calculateBad1 (value::Typename)

    a::Float64

    using values

    a = 1 # complicated algorithm

    b = 2 # complicated algorithm

    return value

end

# warning / error doing this:

function calculateBad2 (value1::Typename, value2::Typename)

    using value1

    using value2

    a = 1 # complicated algorithm

    b = 2 # complicated algorithm

    return value1

end

[Andrew Simper]

To be more specific I suppose I should call the variable being passed to the function "this", so what I am after is good way to write functions that process on and update state, which can either be immutable or not depending on what is most efficient for the specific operations:

immutable State

    a

    b

    # ....

    z

end

function process (this::State, input)

    using this

    a = 1 # complicated computation that depends on a, b, c, ... z and input

    b = 2 # complicated computation that depends on a, b, c, ... z and input

    # ...

    z = 26 # complicated computation that depends on a, b, c, ... z and input

    return this

end

This is not just about object oriented like programming, but it is a common pattern that algorithms share over and over again, so it would be nice to have a concise (end computationally efficient) way to express this in Julia.

[Yuuki Soho]

 This should do what you want:

const circuitModelNames = names(CircuitModel)

macro fetch(ex)

  Expr(:block, [ :($(circuitModelNames[i]) = $ex.$(circuitModelNames[i]) ) for i = 1:length(circuitModelNames) ]...) |> esc

end

julia> macroexpand (:(@fetch model))

:(begin 

        v1 = model.v1

        v5 = model.v5

        v7 = model.v7

        v2 = model.v2

        v3 = model.v3

        v4 = model.v4

        v9 = model.v9

        v12 = model.v12

        v15 = model.v15

        v18 = model.v18

        v19 = model.v19

        v20 = model.v20

        v21 = model.v21

        v8 = model.v8

        v22 = model.v22

        v23 = model.v23

        v24 = model.v24

        i1 = model.i1

        i2 = model.i2

        i3 = model.i3

        i4 = model.i4

        i5 = model.i5

        i6 = model.i6

        i7 = model.i7

        ic1eq = model.ic1eq

        ic2eq = model.ic2eq

        ic3eq = model.ic3eq

        ic4eq = model.ic4eq

        ic5eq = model.ic5eq

        ic6eq = model.ic6eq

        ic7eq = model.ic7eq

        ic8eq = model.ic8eq

        sr = model.sr

        srinv = model.srinv

        pi = model.pi

        gmin = model.gmin

        is1 = model.is1

        nvtf1 = model.nvtf1

        is2 = model.is2

        nvtf2 = model.nvtf2

        nvtinvf1 = model.nvtinvf1

        vcritf1 = model.vcritf1

        nvtinvf2 = model.nvtinvf2

        vcritf2 = model.vcritf2

        gc1 = model.gc1

        gr1 = model.gr1

        gr2 = model.gr2

        itxr2 = model.itxr2

        gc2 = model.gc2

        gc3 = model.gc3

        gc4 = model.gc4

        gc5 = model.gc5

        gr7 = model.gr7

        gc6 = model.gc6

        gr3 = model.gr3

        itxr3 = model.itxr3

        gc7 = model.gc7

        gr4 = model.gr4

        gc8 = model.gc8

        gr5 = model.gr5

        vpos = model.vpos

        vneg = model.vneg

        gin = model.gin

        gininv = model.gininv

        vposcap = model.vposcap

        vnegcap = model.vnegcap

        ginotacore = model.ginotacore

        ginotares = model.ginotares

        ginotacoreinv = model.ginotacoreinv

        ginotaresinv = model.ginotaresinv

        vc3lo = model.vc3lo

        vc3hi = model.vc3hi

        a4a4c = model.a4a4c

        a5a5c = model.a5a5c

        a6a6c = model.a6a6c

        a14a14c = model.a14a14c

        a16a16c = model.a16a16c

        a17a17c = model.a17a17c

        a17a17nrmc = model.a17a17nrmc

        a12a17c = model.a12a17c

        a16a16nrmc = model.a16a16nrmc

        a15a16c = model.a15a16c

        a14a14nrmc = model.a14a14nrmc

        a13a14c = model.a13a14c

        a6a14c = model.a6a14c

        a13a6c = model.a13a6c

        a5a5nrmc = model.a5a5nrmc

        a4a5c = model.a4a5c

        a2a5c = model.a2a5c

        a2a4c = model.a2a4c

        a4a4nrmc = model.a4a4nrmc

        a1a4c = model.a1a4c

        v5c = model.v5c

        v7c = model.v7c

    end)

[Nathaniel Nicandro]

Also you can get around having to specify a variable to hold the names of the type by adding an extra parameter to the macro

macro fetch(name, typ)

    _typ = eval(typ)

    r = Expr(:block)

    r.args = [:($n = $name.$n) for n in _typ.names]

    return esc(r)

end

macroexpand(:(@fetch model CircuitModel))

[Andrew Simper]

That's what I was after thanks Natnaiel! So I can now at least get a macro to expand to a function which I can store per type, and call that at the start of process to fetch the class scope names to local variables :) 

This is a great workaround until there is a neater way to get the names of a function arguments into the local scope.