module system merged

[Jeff Bezanson]

Hello all,

I am pleased to announce that the initial version of our module system
is now part of the master branch. It does not yet do everything we
want, but it is good enough to start trying out. We are likely to
encounter a few bugs and possibly performance regressions over the
next couple days.
Here is a description of what we have:

Modules in julia are separate global variable workspaces. They are
delimited syntactically, inside "module Name ... end". The style is not
to indent the body of the module, since that would typically lead to
whole files being indented. One can have multiple files per module, and
multiple modules per file:

module Foo

include("file1.jl")
include("file2.jl")

end

At this time, files and file names are totally unrelated to modules.
Modules are associated only with module expressions.

Including the same code in different modules provides mixin-like behavior.
One could use this to run the same code with different base definitions,
for example testing code by running it with "safe" versions of some
operators:

module Normal
import Base.*
include("mycode.jl")
end

module Testing
include("checked_operators.jl")
include("mycode.jl")
end

The "import Base.*" statement makes all exported identifiers in the Base
module available in the current module.

There are four important standard modules: Root, Core, Base, and Main.

Root is a hidden module containing the loaded top-level modules. You
never need to use it directly; "import" always begins its name lookup
in the Root module. "import Foo" gives you access to the "Foo" identifier
in the Root module, i.e. a reference to the Foo module. You can then
access identifiers in that module as "Foo.bar".

Core contains all identifiers considered "built in" to the language, i.e.
part of the core language and not libraries. Every module implicitly
specifies "import Core.*", since you can't do anything without those
definitions.

Base is the standard library (the contents of base/). This is not imported
by default, so most modules will want to start with "import Base.*".

Main serves as the current open module when julia starts and you run a
script or begin typing at the prompt. whos() lists identifiers in Main.
Main has a special behavior: when a "module" expression is encountered,
the resulting module is stored in Root. In all other cases a module is
stored within the current module, creating submodules. Main imports Base.*.

The forms of "import" implemented so far are "import Foo", which imports
a single module, "import Foo.*" as discussed, and "import Foo.a" which
imports a single identifier from a module. Modules must explicitly
list exported identifiers using "export a, b, c, ...".
base/sysimg.jl contains the export list for Base. This must now be
updated when public identifiers are added or removed from Base.
base/boot.jl lists the exports for Core. This is not part of Base, but
stored in base/ for convenience.

Currently "export" only affects which symbols can be imported into
other modules. All symbols can be accessed if they are qualified, e.g.
Base._jl_something_private. I was going to "fix" this, but I think it's
handy (especially for debugging) and we might just keep it this way.


Method definition and assignment
--------------------------------

Assignment of globals always takes place in the current module.
The syntax "M.x = y" does not work to assign a global in another module.
Top-level assignments will also overwrite any imported bindings.
If a function is imported, however, defining methods for it will add
methods to the existing function. This can be overridden by declaring
"global f" to overwrite the imported binding with a local one. Example:

module Foo
import Base.sin
sin(x::T) = 42 # method added to Base.sin
end

module Bar
import Base.*
global sin
sin(x::T) = 42 # a new sin function with one definition
end


Macros
------

The module system brings a shiny new macro expander with some degree of
support for hygiene (loosely inspired by syntactic closures). One big
change is that manual gensyms are no longer required --- identifiers
will automatically be renamed to gensyms according to certain rules.
You still manipulate and generate the same expression objects as before,
and it is still possible to capture variables if you really want to.

But, writing macros now requires awareness of two environments: the
macro definition environment (where the "macro foo()" body occurs), and
the macro use environment (where "@foo" occurs). The macro expander
performs an extra pass on the result of a macro to resolve symbols.
Symbols are classified as either "local-ish" or "global-ish". A symbol
is local-ish if it is assigned to (and not declared global), declared
local, or used as a formal argument name. Otherwise it is global-ish.
Local-ish symbols are renamed to gensyms, and global-ish symbols are
resolved in the *macro definition* environment. For example, if a macro
in Base generates "x=2;sin(x)" you will get essentially
"#1=2;Base.sin(#1)".

Of course, certain expressions need to be evaluated in the macro use
environment. This is done by "escaping" them using the function esc().
esc(ex) wraps its argument in a special expression that protects it from
the macro expander's name-resolving shenanigans. When macro arguments
are interpolated into results, they are basically always wrapped in esc():

macro foo(e)
quote
sin($esc(e))
end
end

"@foo y" will use the caller's "y", but the macro def environment's "sin".

The same function can be used to insert captured identifiers:
:($esc(:x) = 1) will assign 1 to whatever "x" exists at the macro use site.

Macros for generating library definitions work unmodified, since the macro def
environment is the same as the macro use environment. Other macros can
be ported by sprinkling "esc" appropriately.

[Tim Holy]

Woo-hoo! In conjunction with the forthcoming package system, this will surely
make it easier to develop and use a growing number of add-ons to Julia.

Thanks, and congrats! It's a big change with a lot of code behind it.
--Tim

[Harlan Harris]

This is great! A few questions right off the bat...

On Tue, Jul 24, 2012 at 7:00 PM, Jeff Bezanson <jeff.b...@gmail.com> wrote:

Base is the standard library (the contents of base/). This is not imported
by default, so most modules will want to start with "import Base.*".

Are there situations where a package developer wouldn't want to import Base.*?
 

Main serves as the current open module when julia starts and you run a
script or begin typing at the prompt. whos() lists identifiers in Main.
Main has a special behavior: when a "module" expression is encountered,
the resulting module is stored in Root. In all other cases a module is
stored within the current module, creating submodules. Main imports Base.*.

To clarify, Main is the primary namespace that a script works in. When your script or a user in the REPL load()s a file with a module definition in it, that module becomes a sub-namespace of Root, which can then be imported with "import X" (would "import Root.X" work too?). Is that right?

The forms of "import" implemented so far are "import Foo", which imports
a single module, "import Foo.*" as discussed, and "import Foo.a" which
imports a single identifier from a module. Modules must explicitly
list exported identifiers using "export a, b, c, ...".

You mention submodules above. So if "module Foo" has a nested "module Bar", you'd import with "import Foo.Bar" or "import Foo.Bar.*" or "import Foo.Bar.Baz"?

Currently "export" only affects which symbols can be imported into
other modules. All symbols can be accessed if they are qualified, e.g.
Base._jl_something_private. I was going to "fix" this, but I think it's
handy (especially for debugging) and we might just keep it this way.

I like this. It's Pythonic in a good way -- no shenanigans enforcing privacy.

It's interesting that you chose an explicit export statement, versus something like the Python rule that symbols starting with _ are not exported by "import Foo.*". I don't have a strong opinion, but curious if there's a rationale.

Does the export statement have to be in any particular place in the module definition? Is there a convention?

Assignment of globals always takes place in the current module.
The syntax "M.x = y" does not work to assign a global in another module.

If a module wanted a global variable associated with it, it'd create a setter function and then export that function?
 

But, writing macros now requires awareness of two environments: the
macro definition environment (where the "macro foo()" body occurs), and
the macro use environment (where "@foo" occurs). The macro expander
performs an extra pass on the result of a macro to resolve symbols.

I assume that older macros that used gensym will still work? It doesn't seem like this is a breaking change for those macros, but I might not be thinking it through...

Are there any modules, aside from the main four, in the codebase now? I don't see any in extras...

Again, thanks!

 -Harlan

[Stefan Karpinski]

On Tue, Jul 24, 2012 at 9:27 PM, Harlan Harris <har...@harris.name> wrote:

This is great! A few questions right off the bat...

Base is the standard library (the contents of base/). This is not imported
by default, so most modules will want to start with "import Base.*".

Are there situations where a package developer wouldn't want to import Base.*?

If you wanted to define your own arithmetic operations, etc., then you might want to do that. Most of the time you'll want to import Base.* though.

 

Main serves as the current open module when julia starts and you run a
script or begin typing at the prompt. whos() lists identifiers in Main.
Main has a special behavior: when a "module" expression is encountered,
the resulting module is stored in Root. In all other cases a module is
stored within the current module, creating submodules. Main imports Base.*.

To clarify, Main is the primary namespace that a script works in. When your script or a user in the REPL load()s a file with a module definition in it, that module becomes a sub-namespace of Root, which can then be imported with "import X" (would "import Root.X" work too?). Is that right?

Yes, that's right. Note that just creating a module does not automatically create a binding for it. For example:

julia> module Foo

         export bar

         bar = 1

       end

julia> Foo

Foo not defined

julia> bar

bar not defined

julia> import Foo

julia> Foo

Module(Foo)

julia> bar

bar not defined

julia> import Foo.*

julia> bar

1

The forms of "import" implemented so far are "import Foo", which imports
a single module, "import Foo.*" as discussed, and "import Foo.a" which
imports a single identifier from a module. Modules must explicitly
list exported identifiers using "export a, b, c, ...".

You mention submodules above. So if "module Foo" has a nested "module Bar", you'd import with "import Foo.Bar" or "import Foo.Bar.*" or "import Foo.Bar.Baz"?

Yup. All of those will work:

julia> module Foo

         export Bar

         module Bar

           export baz

           baz = 1

         end

       end

julia> Foo.Bar

Foo not defined

julia> import Foo.Bar

julia> Bar

Module(Bar)

julia> Bar.baz

1

 

Currently "export" only affects which symbols can be imported into
other modules. All symbols can be accessed if they are qualified, e.g.
Base._jl_something_private. I was going to "fix" this, but I think it's
handy (especially for debugging) and we might just keep it this way.

I like this. It's Pythonic in a good way -- no shenanigans enforcing privacy.

We *may* get stricter at some point for the sake of allowing additional compiler optimizations, so one shouldn't rely on accessing the unexported guts of a module, but yeah, we're not in the business of fussily trying to keep people from doing things (unless it would sabotage performance). Btw, Jeff just dubbed this "the we are all consenting adults here" attitude.

It's interesting that you chose an explicit export statement, versus something like the Python rule that symbols starting with _ are not exported by "import Foo.*". I don't have a strong opinion, but curious if there's a rationale.

Does the export statement have to be in any particular place in the module definition? Is there a convention?

That had not occurred to us, dude. Not being a Python programmer, I wasn't aware of that convention (or is it enforced?). It's a pretty reasonable way to do things and I would consider it, but just like Julia isn't a dot-oriented language (except when it comes to operators), it's also not an underscore-oriented language. Explicit export just seemed more fitting and it's nice to have an upfront list of everything that's exported by a module. Kind of automatic documentation and just like many things in Julia halfway to the static languages.

The export statement can go anywhere, but I'm inclined to put it at the top of the module. However, there are situations, such as when a module is defined in parts which are then included where you might want each part to declare its own exports. Something like this:

module Fuzz

  include("blurm.jl")

  include("fonz.jl")

  include("qux.jl")

end

Assignment of globals always takes place in the current module.
The syntax "M.x = y" does not work to assign a global in another module.

If a module wanted a global variable associated with it, it'd create a setter function and then export that function?

Yep. Global state is discouraged.

 

But, writing macros now requires awareness of two environments: the
macro definition environment (where the "macro foo()" body occurs), and
the macro use environment (where "@foo" occurs). The macro expander
performs an extra pass on the result of a macro to resolve symbols.

I assume that older macros that used gensym will still work? It doesn't seem like this is a breaking change for those macros, but I might not be thinking it through...

Nope. Macros behave very differently now. We seriously need to update the metaprogramming page now...

Are there any modules, aside from the main four, in the codebase now? I don't see any in extras...

No, those are the only ones now.

[Jeff Bezanson]

Just to clarify, I did not invent the "we are all consenting adults
here" attitude; that is a python idea.

Using gensyms in macros will still work, but you also need esc() in
most cases to make macros work right.

> --
>
>
>

[Stefan Karpinski]

On Wed, Jul 25, 2012 at 12:40 AM, Stefan Karpinski <ste...@karpinski.org> wrote:

 

Btw, Jeff just dubbed this "the we are all consenting adults here" attitude.

I just learned that this is not a Jeff Bezanson original quip. Apparently it's a common saying in the Python world. I, on the other hand, come from a Perl background which is so much less, um, vanilla, that this phrase would just get you quizzical looks, followed by everyone going back to writing input filters that allow you to write your Perl programs in Elvish (see the excellent Lingua::Sarati, Lingua::Tengwar, and Lingua::Cirth modules).

[Jeffrey Sarnoff]

Perl -- I had not noticed ./base/intervention.jl

[Stefan Karpinski]

I'm mostly reformed. I have an occasional reflex to want to automatically assign things to _ but I've mostly suppressed that instinct.

--
 
 
 

[Fernando Perez]

On Tue, Jul 24, 2012 at 10:04 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> On Wed, Jul 25, 2012 at 12:40 AM, Stefan Karpinski <ste...@karpinski.org>
> wrote:
>>
>>
>>
>> Btw, Jeff just dubbed this "the we are all consenting adults here"
>> attitude.
>
>
> I just learned that this is not a Jeff Bezanson original quip. Apparently
> it's a common saying in the Python world.

Yup, common enough that I'm pretty sure I answered to someone's
question at Scipy'12 (talk or tutorial, can't remember) just by
quoting that, and pretty much everyone knows what is meant.

I have to say, big +1 to you guys for following that attitude.
Nothing that pisses me off more that bondage-and-discipline
languages... And congrats on the module system landing, this is a big
milestone for Julia!

I also wanted to publicly thank Jeff and Stefan for coming to
SciPy'12. It gave me an opportunity to quiz them over beers instead of
having to actually read the docs, but more importantly, it provided
multiple very useful discussions and what I hope will be the start of
fruitful cross-language collaboration and more idea transfer. Your
talk was excellent, I just hope the videos go up soon!

Cheers,

f

[Stefan Karpinski]

On Wed, Jul 25, 2012 at 1:21 AM, Fernando Perez <fpere...@gmail.com> wrote:

Yup, common enough that I'm pretty sure I answered to someone's
question at Scipy'12 (talk or tutorial, can't remember) just by
quoting that, and pretty much everyone knows what is meant.

Now that you mention it, I actually think I dimly recall that.

 

I have to say, big +1 to you guys for following that attitude.
Nothing that pisses me off more that bondage-and-discipline
languages... And congrats on the module system landing, this is a big
milestone for Julia!

Thanks. It is huge. Jeff's been hard at work for a long while on this. I think it turned out to be far harder than anticipated.

 

I also wanted to publicly thank Jeff and Stefan for coming to
SciPy'12. It gave me an opportunity to quiz them over beers instead of
having to actually read the docs, but more importantly, it provided
multiple very useful discussions and what I hope will be the start of
fruitful cross-language collaboration and more idea transfer.  Your
talk was excellent, I just hope the videos go up soon!

It was very fruitful — and fun. Our first Python collaboration move will be iPython Notebook support :-)

[Fernando Perez]

On Tue, Jul 24, 2012 at 10:29 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> It was very fruitful — and fun. Our first Python collaboration move will be
> iPython Notebook support :-)

You already speak like a wise man :)

f

[Jeffrey Sarnoff]

   ".. most of the time you will want to import Base.* "

  There are not many circumstances where a Julia developer (and vanishingly few where a user) would choose not to import
  e.g. error.jl expr.jl grisu.jl inference.jl promotion.jl ...

  I do not see the balance in causing to be written "import Base.*" into so very, very many as yet unstarted source files.
  Would you consider instead requiring that the developer write "nonimportable Base" (phrased as you like)?

[Patrick O'Leary]

On Wednesday, July 25, 2012 5:23:51 AM UTC-5, Jeffrey Sarnoff wrote:

   ".. most of the time you will want to import Base.* "

  There are not many circumstances where a Julia developer (and vanishingly few where a user) would choose not to import
  e.g. error.jl expr.jl grisu.jl inference.jl promotion.jl ...

  I do not see the balance in causing to be written "import Base.*" into so very, very many as yet unstarted source files.
  Would you consider instead requiring that the developer write "nonimportable Base" (phrased as you like)?

Like GHC's {-# LANGUAGE NoImplicitPrelude #-} pragma?

[Jeffrey Sarnoff]

Right, and as base has many files, the directory specific NoImplictBase and also the file specific NoImplictBase.<filename>

[David Campbell]

> That had not occurred to us, dude. Not being a Python programmer, I wasn't
> aware of that convention (or is it enforced?). It's a pretty reasonable way
> to do things and I would consider it, but just like Julia isn't a
> dot-oriented language (except when it comes to operators), it's also not an
> underscore-oriented language. Explicit export just seemed more fitting and
> it's nice to have an upfront list of everything that's exported by a module.
> Kind of automatic documentation and just like many things in Julia halfway
> to the static languages.

Just my 2c here. Having an explicit list of exports at the top of the
file makes the code less maintainable and readable. If you are going
to automatically generate documentation for a module, then the
generator will know what is exported and what is not and will generate
the correct documentation. If you are reading through the source code,
having a list of exports will mean having to jump back and forth to
see whether something is exported or not, rather than being able to
tell right away. Having items that may be exported in one module but
not in another could make things really confusing.



--
David Campbell

[John Cowan]

On Wed, Jul 25, 2012 at 3:05 PM, David Campbell <dcamp...@gmail.com> wrote:

>> That had not occurred to us, dude. Not being a Python programmer, I wasn't
>> aware of that convention (or is it enforced?). It's a pretty reasonable way
>> to do things and I would consider it, but just like Julia isn't a
>> dot-oriented language (except when it comes to operators), it's also not an
>> underscore-oriented language. Explicit export just seemed more fitting and
>> it's nice to have an upfront list of everything that's exported by a module.

I'd say to use explicit export declarations, but allow them to appear
anywhere in the module. So people who want them all at the top can do
so, and people who want each export declaration next to what it
exports can do that too.

--
GMail doesn't have rotating .sigs, but you can see mine at
http://www.ccil.org/~cowan/signatures

[Stefan Karpinski]

On Wed, Jul 25, 2012 at 4:16 PM, John Cowan <johnw...@gmail.com> wrote:

I'd say to use explicit export declarations, but allow them to appear
anywhere in the module.  So people who want them all at the top can do
so, and people who want each export declaration next to what it
exports can do that too.

That's exactly how it works.

[David Campbell]

The freedom to place an export declaration wherever one pleases does
not help the situation; it makes it worse, since there are more places
that must be searched to figure out whether something is exported.

--
David Campbell

[John Cowan]

On Wed, Jul 25, 2012 at 4:58 PM, David Campbell <dcamp...@gmail.com> wrote:

> The freedom to place an export declaration wherever one pleases does
> not help the situation; it makes it worse, since there are more places
> that must be searched to figure out whether something is exported.

grep or / or C-s or Ctrl-F is your friend here.

[David Campbell]

On Wed, Jul 25, 2012 at 6:03 PM, John Cowan <johnw...@gmail.com> wrote:
> On Wed, Jul 25, 2012 at 4:58 PM, David Campbell <dcamp...@gmail.com> wrote:
>
>> The freedom to place an export declaration wherever one pleases does
>> not help the situation; it makes it worse, since there are more places
>> that must be searched to figure out whether something is exported.
>
> grep or / or C-s or Ctrl-F is your friend here.
>

grepping still requires more time and effort than not grepping. What
are the benefits of having to list everything that is exported over
using a naming convention?

--
David Campbell

[John Cowan]

On Wed, Jul 25, 2012 at 6:25 PM, David Campbell <dcamp...@gmail.com> wrote:

> grepping still requires more time and effort than not grepping. What
> are the benefits of having to list everything that is exported over
> using a naming convention?

_The _fact _that _almost _all _the _identifiers _in _your _module
_except the relatively small public API _wind _up _looking _like
_this. __Ugh.

[Stefan Karpinski]

In general, I don't care for making how you spell an identifier semantically significant. Ruby does this with uppercase/lowercase and it has been a real curse for internationalization. Leading underscore is different because it's something that can be checked without getting into Unicode issues. However, I agree with John that it's ugly. I really don't think the issue of knowing whether something is exported or not when you're looking at the implementation of a module is significant. What you want to be easy is seeing what's exported *without* looking at the implementation. What's the point of an abstraction if I have to look it its internals? That's why explicit exports are nice.

There's a subtler issue here which is that it's perfectly possible to dynamically generate more globals in a module that begin with _ whereas it won't be possible to dynamically generate more export statements (we can easily prevent that). So export statements are much more amenable to statically knowing what a module might export *without* having to actually execute it. That's a nice feature, imo.

--

[Harlan Harris]

I'm trying to get extras/options.jl to work again -- it has a bunch of macros that don't quite work under the new rules -- and have some questions...

First, I assume it's appropriate for options to be a module? (Whether it's packaged with core julia or not, in the long run, will presumably depend on whether and when named function arguments happen (which would mostly supercede it).)

Second, I seem to be getting a collision between "module Options" and "type Options". It obviously works if I change the name of the module, but it's not clear to me that this should be necessary.

Third, I'm confused about the need for esc(). What I'm seeing via trial-and-error isn't matching my (clearly wrong) readings of what Jeff wrote. If I have a macro that looks like this (in part):

macro defaults(opts, ex...)
    for i = 1:length(ex)
         thisex = ex[i]
...

The above works as expected, which is weird, because I would have thought that it would require esc(ex). Can someone explain why I don't need to escape this?

Similarly, I'm having trouble getting access to a field of a macro argument:

macro defaults(opts, ex...)
    quote
        x = $esc(opts).key2index.vals[y]
    end
...

I've tried various approaches, and when I run it like:

@defaults foo a=5 b="cat"

where foo has that structure, I either get "type Expr has no field key2index" and "syntax error: error expanding macro default", or else the expansion works and I get "opts not defined" at evaluation time. Within a quote expression that will be returned to be evaluated, how do I write expressions that access macro argument fields?

I'm clearly missing something about the new macro system here...

Thanks!

 -Harlan

[Jeff Bezanson]

Try ($esc(ex)). Extra parens are often the answer. I might change the parsing here at some point; not sure.

--