Using ModX: Can scope avoid collisions & improve readability?

[ma.laf...@gmail.com]

Hi Julia developers,

I am not sure where I should post this, so here goes...

I am really impressed with the Julia language.  It does many things well.  But there is one aspect of C++ that I miss dearly:

C++ provides "using namespace X" to "make available" the contents of X to the current scope.  This even works on un-named scopes within a function:

• See https://github.com/ma-laforge/testcases/blob/master/namespace_test/namespace_test.jl

In Julia, the closest functional equivalent appears to be the "using" directive.  However, my experiments show that "using" really only works at the module-level:

• See https://github.com/ma-laforge/testcases/blob/master/namespace_test/namespace_test.cpp

Admittedly, the overall module system in Julia is superior to C++ namespaces.  I simply wish the using directive could be applied to arbitrary scopes.

Scope-level "using" directives are particularly useful when the multi-dispatch system cannot resolve ambiguities.  For example, an "Imperial" module might provide "c" in feet/s.

Effect on coding style
Without scope-level "using" directives, it seems impractical to rely on module hierarchy to limit the name length of "factory" elements (those unresolvable by Julia's multi-dispatch system):

• See https://github.com/ma-laforge/testcases/blob/master/namespace_test/arm_objects.jl

In C++, it is very natural and easy to use compact names for these factory elements (ex: classes/constructors/constants...):

• See https://github.com/ma-laforge/testcases/blob/master/namespace_test/arm_objects.cpp

Questions

1. Is there a subtlety of Julia's use model that I might be misunderstanding here?
2. If not: would it be feasible to implement scope-level "using" in Julia?
   

Note
I did notice others out there posting issues regarding namespaces and clobbering.  One that stands out right now is a post by Edmund:

• https://groups.google.com/d/msg/julia-users/T5YYKQzl-Cc/oJVwnMz9zroJ

It appears these posts have died down lately.  Indeed, Julia's current module system does provide reasonably adequate options.  I just wanted to suggest a way to improve it slightly.

[Patrick O'Leary]

On Saturday, February 28, 2015 at 11:06:38 AM UTC-6, MA Laforge wrote:

C++ provides "using namespace X" to "make available" the contents of X to the current scope.  This even works on un-named scopes within a function:

(etc.)

I know this is something that's come up before, and I think rejected--I think because it causes conflicts with multiple dispatch? But I can't seem to find the thread(s).

I can't create a hard conflict in a quick mental search for an example, but I can create some level of confusion:

module Foo
    bar::Int = 1
end

module Baz
    bar::Float64 = 27.3
    with module Foo # not current Julia syntax
        bar #which bar is this?
    end
end

[Josh Langsfeld]

It's discussed in the FAQ so there must be a good reason for it, though no rationale is mentioned.

http://docs.julialang.org/en/latest/manual/faq/#can-i-use-using-or-import-inside-a-function

[ele...@gmail.com]

The C++ "using namespace" and Julia "using module" are not quite the same thing.  The C++ namespace is a top level entity that is created and initialized as part of the C++ startup code and the "using" just makes the names of these global entities available within the scope.   The Julia "using" imports, and thus creates, the module for the first time, and Julia modules can contain statements that run at initialization, whereas C++ namespaces can only contain declarations.  

But if the Julia "using" is within the function, when should the actual import and initialize and execute the statements be done?  Every time the function is called is very expensive.  Hoist the import out of the function to the top level, but doing this naively (ie putting the hoisted import just before the function) will then affect all code following the function  Have a "pseudo" top- level visible only to the function adds a complete new complication to the name management code.  But then what happens if the module is imported into two functions, how is its initialization managed? Or the top level and the function?

So for now the "simpler is better" approach is to have the user manage importing at the top level only.

Cheers

Lex

[MA Laforge]

Your comment sounds alot like what Stefan said:
https://groups.google.com/d/msg/julia-users/UvBff9QVKaA/P10-LRLezCUJ

I admit I don't fully appreciate why this is a *technical* problem.  Most scoping rules would dictate that you should be referring to the *most local* version of the value.  In your case bar would come from module Foo.

On the other hand, I can envisage a scary situation arising by adding variables to the Foo module *after* the code for Baz has been completed & verified.  To avoid this potentially disastrous event: the Julia compiler might have to err out.

On Monday, March 2, 2015 at 3:00:21 PM UTC-5, Patrick O'Leary wrote:

[MA Laforge]

Indeed.  I think that is the problem: Julia's "using" statement is tightly coupled to the module import functionality.

From my understanding, the C++ "using namespace N1::N2::N3" directive simply informs the compiler to use shorthand notation for all variables/functions present in namespace N1::N2::N3.  You do not actually "Import" anything new.  In fact, with C++: the contents of that module are usually compiled independently in another library file anyways.  I don't think the using directive affects the linker either (at least not directly).

[Patrick O'Leary]

On Monday, March 2, 2015 at 7:45:02 PM UTC-6, MA Laforge wrote:

Your comment sounds alot like what Stefan said:
https://groups.google.com/d/msg/julia-users/UvBff9QVKaA/P10-LRLezCUJ

I admit I don't fully appreciate why this is a *technical* problem.  Most scoping rules would dictate that you should be referring to the *most local* version of the value.  In your case bar would come from module Foo.

One counterargument to that is that it makes the most local thing nonlocal--it's an unexported part of Foo. The export list is in part a declaration of "I'm okay with these things sharing namespace if you want," and if you did `export bar` from Foo, then `import Foo`, you'd get what you wanted.

(There's a side discussion about things happening inside functions; I'm not concerned about that here because that can only make things more complicated.)

[Josh Langsfeld]

I'm curious about that workaround suggested in the FAQ, where you wrap the function inside its own module. What is happening under the hood there? Does it reinitialize the desired module entirely inside of the wrapper module or does it just make a reference to some other compiled and initialized top-level area? This is assuming the module has already been built elsewhere and you just want easy local access to the exported symbols.

On Monday, March 2, 2015 at 5:59:30 PM UTC-5, ele...@gmail.com wrote:

[MA Laforge]

Josh:
I do not fully appreciate the details of how modules get imported either.  Here is what I can gather:

module moda
    export modvar, setmodvar
    modvar = 1
    setmodvar(x) = (global modvar=x)
end

module modb
    export modvar, setmodvar
    modvar = 2
    setmodvar(x) = (global modvar=x)
end

module modc
    using moda
    import modb
    println("using moda")
    @show modvar, moda.modvar, modb.modvar
    @show setmodvar(5)
    @show modvar, moda.modvar, modb.modvar
    @show modb.setmodvar(3)
    @show modvar, moda.modvar, modb.modvar
end

module modd
    import moda
    using modb
    println("using modb")
    @show modvar, moda.modvar, modb.modvar
    @show setmodvar(8)
    @show modvar, moda.modvar, modb.modvar
end

This gives the following results:

using moda
(modvar,moda.modvar,modb.modvar) => (1,1,2)
setmodvar(5) => 5
(modvar,moda.modvar,modb.modvar) => (5,5,2)
modb.setmodvar(3) => 3
(modvar,moda.modvar,modb.modvar) => (5,5,3)
using modb
(modvar,moda.modvar,modb.modvar) => (3,5,3)
setmodvar(8) => 8
(modvar,moda.modvar,modb.modvar) => (8,5,8)

So it looks like it is the same code in the background (at least the variables appear to be in a common address space).

So... it looks like import merely "makes visible"  the code at a module level...
And "using" first "imports", then "provides a shortcut" to the exported variables/functions...

But as far as I can tell, compilation is done "on demand".  I believe that the "real" compile step is done when we call a particular version of the code.  Compilation is actually a separate step from "using" and "import"

[MA Laforge]

Indeed this appears to be the intent: "export" tells the user what you want to be his/her interface.

But I have also noticed myself omitting "exports" in order to avoid collisions with elements that cannot be resolved by multi-dispatch (like type constructors):

module Electronics
type Circuit ... end #Name is very common... don't "export"
end

module RailSystem
type Circuit ... end #Name is very common... don't "export"
end

using Electronics
using RailSystem #No collision with Circuit: because it is not exported

eckt = Electronics.Circuit("rectifier")
railckt = RailSystem.Circuit("Line A")

Otherwise, I would have to use more verbose names (which just seems silly to me):

Electronics.Circuit -> Electronics.ElectronicCircuit #Not a common name: should be ok to export
RailSystem.Circuit -> RailSystem.RailCircuit #Not a common name: should be ok to export

...And maybe that's simply how modules should be defined: Never export elements that cannot be resolved by multi-dispatch.

[Josh Langsfeld]

So then your results would indicate that putting 'using MyModule' inside a function could actually bring the exported names into only the local scope if the module had already been constructed in the global scope. I know I've seen some people discuss name pollution by 'using' too many modules. It might be part of a wider solution to enable local scope 'using', either with a new keyword or just having 'using' check if the module already exists.

[ele...@gmail.com]

This is essentially the method used by Python (as I believe I understand it, import is the worst documented thing in Python).  Python puts a reference to all modules into sys.modules, no matter where they are imported.  And sys.modules is always the first thing searched before path.  So all modules are imported and initialized once (excepting users explicitly deleteing or reloading).  An no matter where the imports happen the first will actually load and initialize the module and all the others will just use that copy.  But since modules are referenced by sys.modules, not the top level namespace, there is no pollution until a user explicitly imports into that namespace.

[MA Laforge]

ele...: I did not know about this sys.modules thing.  Good to know.

Josh & ele...:
Agreed.  I don't think pollution happens until the "using modX" call happens.  So, I don't really understand why we can't have scope-level "using" commands.

And yes, maybe Julia *should* use a different keyword than "using" when applied to arbitrary scopes (as opposed to when we use it at the module-level).

...So linking back to Patrick's comment:
The "export" list sort of tells the module user what functions are meant to be used.  It just happens that Julia's implementation of "using" pulls in all "export"-ed elements into the current module namespace.  This works extremely well with the multi-dispatch engine.

However, I can also see a place for a slightly weaker directive: Let's call it "provide" for now.  The "provide" directive could be used to tell the user what other elements (like constructors) are "supported" by the module.  The "provide"-ed elements would not be pulled in by the "using" statement.  Only "export"-ed values do that.  Instead, "provide"-ed elements would *only* be pulled-in when a user calls "using >>namespace<< modX" (or some other keyword).

This would allow developers to use shorter names.  Names that would otherwise collide if they were to be "export"-ed:

module Electroncis
    type Circuit
        name::String
        elemlist::Array{CktElem}
        ...
    end

    abstract IndepCktElem
    abstract drivePattern
    type Vsrc <: IndepCktElem ...; end
    type SineWave <: drivePattern ...; end

    connect(c::Circuit, x::CktElem, n::NodeList) = ...
    drive(c::Circuit, x::IndepCktElem, n::NodeList, p::drivePattern) = ...
    ...

    export connect, drive  #No problem: Can be handled by multi-dispatch
    provide Circuit        #Dangerous: Cannot always be resolved by multi-dispatch
    provide Vsrc, SineWave #Also a little dangerous
end

Now, in the user's module, you can do the following:

using Electronics #Brings in connect & drive, as usual

#Eventually gets a little verbose when dealing with elements that cannot easily
#be resolved by multi-dispatch ("provide"-ed elements):
function make_circuit()
    c = Electronics.Circuit("Myckt")
    drive(c, Electronics.Vsrc(), [:SUPPLY, :GND], Electronics.SineWave(1e9))
    return c
end

#But this function *could* be made more succinct:
function make_circuit2()
    using namespace Electronics #Also brings in all other "legitimate" elements
    c = Circuit("Myckt") #More succinct constructor call

    #Things gets even nicer with more complex expressions:
    drive(c, Vsrc(), [:SUPPLY, :GND], SineWave(1e9))
    return c
end