Converting string to module type/Programmatically using modules

[Joshua Adelman]

I'm attempting to do some dynamic module loading and subsequent processing and can't quite figure something out. I think I have a reasonable (albeit maybe not idiomatic) mechanism for dynamically importing a set of modules. See my stackoverflow question and the subsequent self answer:

http://stackoverflow.com/q/27696356/392949

My next question is, after I've dynamically/programmatically imported a bunch of modules, is there a way of iterating over that set of modules? Specifically, if I store the module names as strings (e.g. ["mod00", "mod01", "mod02"]) and each module contains some function `func` that isn't exported, how would I call `mod00.func()` if I only know the string "mod00"? I think this comes down to converting the string to a Module type, but I'm not sure and haven't come up with something workable after a few passes through the docs.

Any suggestions would be very much appreciated.

Josh

[Isaiah Norton]

Regarding your SO question, the suggested way to do this is with a macro, but (as you observed) it is a bit tricky. Whenever I try to do this I have to remind myself how it works by using parse and dump to see how the expression should look:

```

julia> parse("import foo")

:(import foo)

julia> dump(x)

Expr 

  head: Symbol import

  args: Array(Any,(1,))

    1: Symbol foo

  typ: Any

```

So then I can duplicate that and eval it, for example:

```

julia> eval(Expr(:import, :Color))

```

module contains some function `func` that isn't exported, how would I call `mod00.func()` if I only know the string "mod00"

```

julia> module mod00

       func() = println("hello")

       end

julia> eval(symbol("mod00")).(symbol("func"))()

hello

```

I'm not sure what the overall goal is here, but just beware that eval should only be used like this for testing purposes or else your code will be slow.

 

[Joshua Adelman]

Thanks Isaiah. That helps a lot and I think I'm starting to understand what those calls are doing. I'm coming from a python background, so I don't have much experience with metaprogramming. I'm prototyping port of a complex system that I had previously written in python/cython to Julia to see how Julia performs.

The basic idea is to dynamically import a bunch of modules that each contain some metadata and an implementation of a feature to transform some data. The metadata will be parsed and inserted into a composite type that also contains a ::Function field that will hold the implementing function. A collection of these composite types will then be systematically run against some large set of data. 

The dynamic import is to provide something like a plugin system and also to control the startup cost by only importing what is needed for a particular calculation, rather than every available module. 

Thanks again for your help. I'm having a lot (too much?) fun with Julia thus far.

Josh

[Ismael VC]

I get a different expression from parsing foo:

julia> versioninfo()
Julia Version 0.3.3
Commit b24213b (2014-11-23 20:19 UTC)
Platform Info:
  System: Linux (i686-pc-linux-gnu)
  CPU: Intel(R) Atom(TM) CPU N570   @ 1.66GHz
  WORD_SIZE: 32
  BLAS: libblas
  LAPACK: liblapack
  LIBM: libm
  LLVM: libLLVM-3.3

julia> parse("import foo")

:($(Expr(:import, :foo)))

julia> parse("import foo, bar, baz")
:($(Expr(:toplevel, :($(Expr(:import, :foo))), :($(Expr(:import, :bar))), :($(Expr(:import, :baz))))))

This is returning the needed expression, but it's returning it like a function instead of executing it:

julia> macro dynamic_import(modules)
           quote
               ex = Expr(:toplevel)
               names = map(m -> symbol(split(m, '.')[1]), $modules)
               for name in names
                   push!(ex.args, Expr(:import, name))
               end
               return ex
           end
       end

julia> @dynamic_import ["Newton.jl", "MyTest.jl"]
:($(Expr(:toplevel, :($(Expr(:import, :Newton))), :($(Expr(:import, :MyTest))))))

I thought I almost got this! :D

[Ismael VC]

Shouldn't the macro evaluate it automatically? I think we need way more metaprogramming documentation.

julia> @dynamic_import ["Newton.jl", "MyTest.jl"]
:($(Expr(:toplevel, :($(Expr(:import, :Newton))), :($(Expr(:import, :MyTest))))))

julia> eval(ans)

julia> Newton
Newton

julia> MyTest
MyTest

El lunes, 29 de diciembre de 2014 21:45:45 UTC-6, Joshua Adelman escribió:

[Michael Hatherly]

You don’t need the quote ... end block since you’re creating the expression manually using Expr objects.
Removing that and changing $modules to modules.args should work alright I think.

— Mike

​

[Ismael VC]

Thank you Michael I got it!

julia> macro dynamic_import(modules)
           ex = Expr(:toplevel)
           names = map(m -> symbol(split(m, '.')[1]), modules.args)

           for name in names
               push!(ex.args, Expr(:import, name))
           end
           return ex
       end

julia> @dynamic_import ["Newton.jl", "MyTest.jl"]

julia> Newton
Newton

julia> MyTest
MyTest

[Michael Hatherly]

No prob.

— Mike

​

[Ismael VC]

Joshua 've updated your question on Stack Overflow: http://bit.ly/1wz5n5U

You could also generalize this line:

module_files = filter(r"^mod[0-9][0-9].jl$", readdir())

By abstracting it into a function:

julia> module_files(dir=".") = filter(r"^mod[0-9][0-9].jl$", readdir(dir))
module_files (generic function with 2 methods)

So you may use it like this:

julia> @dynamic_import module_files()


Cheers!
Ismael VC 

El lunes, 29 de diciembre de 2014 21:45:45 UTC-6, Joshua Adelman escribió: