eval in current scope

[Marius Millea]

Hi, is there a way to "eval" something in the current scope? My problem is the following, I've written a macro that, inside the returned expression, builds an expression which I need to eval. It looks like this,

macro foo()

    quote

        ex = ...

        eval_in_current_scope(ex)

    end

end

Now, you might say I'm using macros wrong and I should just be doing,

macro foo()

    ex = ...

end

 

but in this case when I build "ex", it needs to occur at runtime since it depends on some things only available then. So is there any way to go about this? Thanks. 

[Marius Millea]

And just to be clear, by "current scope" here I mean the scope of where the code from this macro is getting "pasted", not the macro scope. 

[Steven G. Johnson]

On Tuesday, September 27, 2016 at 5:28:40 AM UTC-4, Marius Millea wrote:

Hi, is there a way to "eval" something in the current scope?

No.  I think you are fundamentally misunderstanding what macros do.  Macros are functions evaluated at parse-time.  The runtime scope doesn't even exist when the macro is called.

[Marius Millea]

Macros are functions evaluated at parse-time.  The runtime scope doesn't even exist when the macro is called.

 

That's right, the answer may well have nothing to do with marcos (maybe I obscured the question by even mentioning them in an attempt to give bigger context to what I'm trying to accomplish). 

I guess it really boils to just "is there a way to eval something in the current scope". Not knowing much about the internals of all of this, given that "eval" does exactly that in the global scope, I guess it wouldn't seem like such a stretch that something exists for the current scope. For example, in Python it exists, 

In [1]: def f(x):
   ...:     return eval("x+1")


In [2]: f(3)
Out[2]: 4

But perhaps the JIT requirements make it impossible in Julia? 

julia> function f(x)
          eval(:(x+1))
       end
f (generic function with 1 method)


julia> f(3)
ERROR: UndefVarError: x not defined
 in eval(::Module, ::Any) at ./boot.jl:234
 in f(::Int64) at ./REPL[1]:2

[Yichao Yu]

On Tue, Sep 27, 2016 at 7:47 AM, Marius Millea <marius...@gmail.com> wrote:
>> Macros are functions evaluated at parse-time. The runtime scope doesn't
>> even exist when the macro is called.
>
>
> That's right, the answer may well have nothing to do with marcos (maybe I
> obscured the question by even mentioning them in an attempt to give bigger
> context to what I'm trying to accomplish).

No.

>
> I guess it really boils to just "is there a way to eval something in the
> current scope". Not knowing much about the internals of all of this, given
> that "eval" does exactly that in the global scope, I guess it wouldn't seem
> like such a stretch that something exists for the current scope. For
> example, in Python it exists,
>
> In [1]: def f(x):
> ...: return eval("x+1")
>
>
> In [2]: f(3)
> Out[2]: 4
>
>
>
> But perhaps the JIT requirements make it impossible in Julia?

It's not the JIT requirement, rather the performance trade off

[Jussi Piitulainen]

You might be able to wrap your expression so as to create a function instead, and call the function with the values of the variables that the actual expression depends on. In Python, because I haven't learned to construct expressions in Julia yet and don't have the time to learn it now:

def f(x): return eval("lambda x: x + 1")(x)

[Marius Millea]

Aha, interesting! I think that might work, let me see if it actually works in my real case... Fyi, in Julia it might look like this:

julia> function f(x)

           eval(:(x->x+1))(x)

       end

f (generic function with 1 method)

julia> f(3)

4

Coincidentally as I've been digging I think that's the same solution suggested here https://github.com/JuliaLang/julia/issues/2386#issuecomment-13966397

Marius

[Adrian Salceanu]

Yichao, not long ago I had a similar question (it was in the context of building a templating system), you were very kind and point me in the right direction. So I kept digging. 

Marius, sorry for piggy backing on your question, but I'm curious about some code that seems to do the trick. I'm curious if I'm missing anything. 

# A.jl
push!(LOAD_PATH, ".")

module A
using B

function a()
  @injectvar
  println("in a()")
  @show external_var
end

a()

try
  println("in A")
  @show external_var
catch ex
  @show ex
end

end

#B.jl
module B
export @injectvar

macro injectvar()
  esc(:(external_var = 5))
end

end

If I run A.jl I get: 

in a()
external_var = 5
in A
ex = UndefVarError(:external_var)

So external_var is defined in A.a() but not in A. 

# C.jl
push!(LOAD_PATH, ".")

module C
using A

A.a()

try
  println("in C")
  @show external_var
catch ex
  @show ex
end

end

Similarely, if I run C.jl I get

in a()
external_var = 5
in A
ex = UndefVarError(:external_var)
in a()
external_var = 5
in C
ex = UndefVarError(:external_var)

external_var is only available inside the function. 

[Fábio Cardeal]

The lambda example works, but couldn't you do:

function f(x)
  eval(:($x + 1))
end

?

You can still do that in a macro:

macro m()
  quote
    ex = :($x + 1)
    eval(ex)
  end
end

function f(x)
  @m
end

f(1) == 2

What would be a situation where this wouldn't work?

[Stefan Karpinski]

If you need to generate code where the outer function has access to some part of a local scope, you can generate the entire function and then call it:

julia> ex = :(x + 1)

:(x + 1)

julia> @eval function f(x)

           y = $ex

           return y^2

       end

f (generic function with 1 method)

julia> f(2)

9

But note that if you want some piece of f to be "pasted" from the user and have access to certain parts of the local state of f, it's probably a much better design to let the user pass in a function which f calls, passing the function the state that it should have access to:

julia> f(2)

9

julia> function g(h, x)

           y = h(x)

           return y^2

       end

g (generic function with 1 method)

julia> g(x->x+1, 2)

9

It's three more characters to type, but the way you call g isn't coupled to the local variable names that g happens to use (x), and you can pass named functions like `abs` to g more easily. It also prevents someone from passing in a bit of code that completely breaks the caller in unexpected ways.

[Steven G. Johnson]

On Tuesday, September 27, 2016 at 10:27:59 AM UTC-4, Stefan Karpinski wrote:

But note that if you want some piece of f to be "pasted" from the user and have access to certain parts of the local state of f, it's probably a much better design to let the user pass in a function which f calls, passing the function the state that it should have access to:

Right; using "eval" in a function is almost always a mistake, an indication that you should really be using a higher-order function.

[Marius Millea]

I think there's at least once scenario where eval-in-a-macro is not a mistake, mainly when you want to generate some code that depends on 1) some passed in expression and 2) something which can only be known at runtime. Here's my example:

The macro (@self) which I'm writing takes a type name and a function definition, and gives the function a "self" argument of that type and rewrites all occurrences of the type's fields, X, to self.X. Effectively it takes this:

type MyType
    x
end

@self MyType function inc()
    x += 1
end

and spits out:

function inc(self::MyType)
    self.x += 1
end

(if this sounds familiar, its because I've discussed it here before, which spun off this, that I'm currently working on tweaking)

To do this my code needs to modify the function expression, but this modification depends on fieldnames(MyType), which can only be known at runtime. Hence what I'm doing is, 

macro self(typ,func)

    function modify_func(fieldnames)
        # in here I have access to `func` expression *and* `fieldnames(typ)` as evaluated at runtime
        # return modified func expression
    end

    quote
        $(esc(:eval))($modify_func(fieldnames($(esc(typ)))))
    end
 
end

I don't see a cleaner way of doing this, but I'm happy to take suggestions. 

(Btw: my original question was w.r.t. that last "eval', which makes it so that currently this doesn't work on function closures. I'm still processing the several suggestions in this context...)

[Cedric St-Jean]

Would eval'ing the type inside the macro work? This shows [:x, :y]

macro type_fields(typ)

    fields = fieldnames(eval(typ))

    @show fields

end

type A

    x

    y

end

@type_fields A

You can use that to generate the right expansion for that type.

[Mauro]

You could do it like so:

A macro which defines a type-specific version @self_MyType of your @self
macro at the definition of the type:

@with_self type MyType
x
end

Then use that generated macro when defining the function:

@self_MyType function inc()
x += 1
end

This avoids any at little extra cost. The only down side is that you
cannot easily handle SomeoneElsesType. I do something similar in
https://github.com/mauro3/Parameters.jl/blob/cb147097de1cad1dd5c089243743f21a576a4a32/src/Parameters.jl#L381
where I construct a type specific macro: @unpack_MyType.

(And on a side note, and I think repeating myself from an earlier
thread, why don't you use Parameters.jl's @unpack instead? Consider
this:

type MyType
delta
end

# somewhere else, maybe far away, define:
@self MyType function f(x)
sin(pi*x + delta)
end

# ...
# some time far in the future add a field to MyType:
type MyType
delta
pi
end

# and now your function f breaks silently!

Being explicit with what you unpack from MyType, using e.g. @unpack, avoids this.
)

On Fri, 2016-09-30 at 00:53, Marius Millea <marius...@gmail.com> wrote:
> I think there's at least once scenario where eval-in-a-macro is not a
> mistake, mainly when you want to generate some code that depends on 1) some
> passed in expression and 2) something which can only be known at runtime.
> Here's my example:
>
> The macro (@self) which I'm writing takes a type name and a function
> definition, and gives the function a "self" argument of that type and
> rewrites all occurrences of the type's fields, X, to self.X. Effectively it
> takes this:
>
> type MyType
> x
> end
>
> @self MyType function inc()
> x += 1
> end
>
> and spits out:
>
> function inc(self::MyType)
> self.x += 1
> end
>
> (if this sounds familiar, its because I've discussed it here before, which

> spun off this <https://github.com/fcard/SelfFunctions.jl>, that I'm

> currently working on tweaking)
>
>
> To do this my code needs to modify the function expression, but this

> modification depends on fieldnames(MyType), which can *only* be known at

> runtime. Hence what I'm doing is,
>
> macro self(typ,func)
>
> function modify_func(fieldnames)
> # in here I have access to `func` expression *and*
> `fieldnames(typ)` as evaluated at runtime
> # return modified func expression
> end
>
> quote
> $(esc(:eval))($modify_func(fieldnames($(esc(typ)))))
> end
>
> end
>
> I don't see a cleaner way of doing this, but I'm happy to take suggestions.
>
> (Btw: my original question was w.r.t. that last "eval', which makes it so
> that currently this doesn't work on function closures. I'm still processing
> the several suggestions in this context...)
>
>
> On Tuesday, September 27, 2016 at 5:12:44 PM UTC+2, Steven G. Johnson wrote:
>>
>> On Tuesday, September 27, 2016 at 10:27:59 AM UTC-4, Stefan Karpinski
>> wrote:
>>>
>>> But note that if you want some piece of f to be "pasted" from the user
>>> and have access to certain parts of the local state of f, it's probably a
>>> much better design to let the user pass in a function which f calls,
>>> passing the function the state that it should have access to:
>>>
>>
>> Right; using "eval" in a function is almost always a mistake, an
>> indication that you should really be using a higher-order function.

>> <https://en.wikipedia.org/wiki/Higher-order_function>
>>

[Mauro]

On Fri, 2016-09-30 at 00:53, Marius Millea <marius...@gmail.com> wrote:

> I think there's at least once scenario where eval-in-a-macro is not a
> mistake, mainly when you want to generate some code that depends on 1) some
> passed in expression and 2) something which can only be known at runtime.

I think one of the problems with macro-evals is that (at least the
advanced user) will expect a macro to do a pure syntax transform,
independent of the state of the runtime. Anything else would be
surprising. Maybe macros which call `eval` should have names with an
`!` to advertise their non-standard-ness?

> Here's my example:
>
> The macro (@self) which I'm writing takes a type name and a function
> definition, and gives the function a "self" argument of that type and
> rewrites all occurrences of the type's fields, X, to self.X. Effectively it
> takes this:
>
> type MyType
> x
> end
>
> @self MyType function inc()
> x += 1
> end
>
> and spits out:
>
> function inc(self::MyType)
> self.x += 1
> end
>
> (if this sounds familiar, its because I've discussed it here before, which

> spun off this <https://github.com/fcard/SelfFunctions.jl>, that I'm

> currently working on tweaking)
>
>
> To do this my code needs to modify the function expression, but this

> modification depends on fieldnames(MyType), which can *only* be known at

> runtime. Hence what I'm doing is,
>
> macro self(typ,func)
>
> function modify_func(fieldnames)
> # in here I have access to `func` expression *and*
> `fieldnames(typ)` as evaluated at runtime
> # return modified func expression
> end
>
> quote
> $(esc(:eval))($modify_func(fieldnames($(esc(typ)))))
> end
>
> end
>
> I don't see a cleaner way of doing this, but I'm happy to take suggestions.
>
> (Btw: my original question was w.r.t. that last "eval', which makes it so
> that currently this doesn't work on function closures. I'm still processing
> the several suggestions in this context...)
>
>
> On Tuesday, September 27, 2016 at 5:12:44 PM UTC+2, Steven G. Johnson wrote:
>>
>> On Tuesday, September 27, 2016 at 10:27:59 AM UTC-4, Stefan Karpinski
>> wrote:
>>>
>>> But note that if you want some piece of f to be "pasted" from the user
>>> and have access to certain parts of the local state of f, it's probably a
>>> much better design to let the user pass in a function which f calls,
>>> passing the function the state that it should have access to:
>>>
>>
>> Right; using "eval" in a function is almost always a mistake, an
>> indication that you should really be using a higher-order function.

>> <https://en.wikipedia.org/wiki/Higher-order_function>
>>

[Marius Millea]

Would eval'ing the type inside the macro work? This shows [:x, :y]

This only works if A and type_fields are defined in the same module though. Although to be honest it surprised me a bit that it works at all, I guess the type definitions are evaluated prior to macro expansions? 

A macro which defines a type-specific version @self_MyType of your @self 
macro at the definition of the type: 

Yea, the solutions both me and fcard coded up originally involved having to call a macro on the type definition, this is precisely what I'm trying to get rid of right now. The reason for not using @unpack is just that its more verbose than this solution (at the price of the type redefinition thing, but for me its a fine tradeoff). It *really* like getting to write super concise functions which read just like the math they represent, nothing extra distracting, e.g. from my actual code:

"""Hubble constant at redshift z"""
@self Params Hubble(z) = Hfac*sqrt(ρx_over_ωx*((ωc+ωb)*(1+z)^3 + ωk*(1+z)^2 + ωΛ) + ργ(z) + ρν(z))


"""Optical depth between two redshifts given a free electron fraction history Xe"""
@self Params function τ(Xe::Function, z1, z2)
    σT*(ωb*ρx_over_ωx)/mH*(1-Yp) * quad(z->Xe(z)/Hubble(z)*(1+z)^2, z1, z2)
end

 

[Cedric St-Jean]

This only works if A and type_fields are defined in the same module though. Although to be honest it surprised me a bit that it works at all, I guess the type definitions are evaluated prior to macro expansions? 

Good point. 

You can use a generated function then:

using MacroTools

macro withself(fdef)

    @assert @capture(fdef, fname_() = expr_) # could accept other arguments

    :(@generated function $(esc(fname))(self)

        expr = $(Expr(:quote, expr))   # grab the `expr` from the macro body

        # Replace this with recursive traversal. For demo, I'm assuming a single function call

        @assert @capture(expr, f_(args__))

        :($f($([arg in fieldnames(self) ? Expr(:., :self, QuoteNode(arg)) : arg for arg in args]...)))

    end)

end

type A

    ii

    oo

end

a = 20

@withself foo() = ii + oo + a

foo(A(33, 44))  # 97

Since generated functions are passed the _type_ of their arguments, which is what you were looking for. It's an abuse of Julia's metaprogramming facilities in strict Lisp tradition ;)

[Cedric St-Jean]

Looks like I was accidentally on 0.4, and hygiene rules changed in 0.5. This works:

using MacroTools

macro withself(fdef)

    @assert @capture(fdef, fname_() = expr_)

    esc(:(@generated function $(fname)(self)

        expr = $(Expr(:quote, expr))

        # Replace this with recursive traversal. For demo, I'm assuming a single function call

        @assert @capture(expr, f_(args__))

        :(($f)($([arg in fieldnames(self) ? Expr(:., :self, QuoteNode(arg)) : arg for arg in args]...)))

    end))

end

type A

    ii

    oo

end

a = 20

@withself foo() = ii + oo + a

foo(A(33, 44))  # 97