Some constructive criticism & requests

[Steven Sagaert]

Hi,

I've been programming a recommender system in Julia for a couple of weeks and overall I like much of Julia and the choices that have been made but I've also found some things that annoy me and that I'd like to see changed. I figure that at 0.2 it's still a very young language with a small user base & ecosystem so I might actually have a chance of some of my requests actually make it into a future release.

* naming convention of the standard lib. For types Julia uses camelcase starting with uppercase (like Java for Classes) but for functions it's all lowercase with sometimes an underscore for separation. Having done a lot of Java programming this feels a bit weird to me. Why mix two different systems (camelcase & underscores?) Why not go for Java's system: camelcase (shorter than underscores) : function would start with lowercase (like Java methods) and types start with uppercase. I could live with all lowercase for functions but then I would like to see a consistent use of underscore for word separation. In the standard lib this is very inconsistent (many functions names have no separation where it should: e.g. haskey) and I find this very annoying since you never really know how a function name is written unless you look it up. In the absence of any code completion in the current development tool this little thing quickly becomes a source of irritation.

* modules: I find the current way how modules are used in client code confusing & lacking. Why two systems (using vs import)? using is simple enough but then you have no control over which functions are imported into your client namespace leading to possible name clashes. Import seems confusing (import <module> vs import <module.function>) & when importing function you can import any (disregarding the export list) and hence breaking encapsulation? 

I 've also found that export list of functions is annoying (it feels a bit like header files in C++):  it's easy to forget to add your functions to it especially when you are reworking code and since there's no compiler that complains you only find out about it at runtime with a cryptic error message.  

There is no version data in the source code for a module. This could be really useful voor specifying dependencies with version numbers (with a package manager that support that) to get exactly what you need to avoid "dependency hell".

Here is what I propose:

Basically I think the Java/C# system works pretty well so this solution is modeled after it.

- use only one keyword/mechanism for importing modules (if we ignore include from the discussion). 

- annotate functions in the module with public/private keywords to mark if they are exported or not. If you want to save keystrokes one use the convention that no keyword=private so only public would be needed then. This pretty much garantees that you'll never forget to export your functions.

- on the client side only public functions can be imported & used (encapsulation is enforced). If you really want to override encapsulation (e.g. for debugger tool/I.D.E., other frameworks (think Spring in Java) this can be done via reflection/metaprogramming but not regular programming. 

- import <module> imports all public functions in the client namespace. import <module.function> imports just this function in the namespace. In case of name clashes, the names are fully written in the client code: <module.function> but they still need to be imported.

- specify a version number next to the module name analogous to the C# mechanism & make the package manager aware of version number so that in dependencies can be precisely specified (similar to Maven,Gradle,...). This will save a lot of headaches once the library ecosystem starts to explode. R has this problem but now that it have thousands of packages already it's very hard to fix it anymore.

* packages should not be just a linear list of names (like in R which sucks big time if there a thousands) but be hierarchically organised in broad categories (like in Java,C#) and this organisation is reflected as directories in the filesystem. Julia standard libs should start with julia and external lib with the name of the firm/organisation/other meaningful name and have namesegments with a separator (e.g. a dot). For example julia.core,julia.net, julia.graphics,julia.gui,julia.db,julia.io,... Logically organising the packages like this will pay off in the long run when there are many.

* When an exception is thrown, a full stacktrace should be printed starting at the place (module name, line number, function name) where it is thrown instead of printing the line number where it is caught which is pretty much useless. This would be a big improvement for debugging!

* exception handling: catch clause is too primitive: since you cannot specify the type of the exception it catches all exeptions even if you do not want it to and hence then you have to test the type(s) with a bunch of if/else statements and rethrow the exception types that you do not want to catch. Java/C# is much better/compact here: just specify the exception type (or a list of types e.g. separated by ,) after catch and then only that type is caught.

* I find myself writing a lot of type annotation in my Julia code. Basically all function arguments & local vars that do not have an initial value get a type. I do this for self documentation & for helping the compiler generate efficient code. So my code is not that different than it would be from a statically typed language. Unfortunately type inconsistencies are only discovered at runtime and once you start specifying types in Julia (instead of being fully dynamic) it can be a slippery slope because it tends to go viral so that you need to narrow the types of other variables as well (e.g. String vs ASCIIString). I figure: if most of my code has type information anyway (& the rest can be easily inferred from type inference) why not go all the way and have the option of an ahead of time statically compiler that would flag type errors at compile time and that would generate a binary executable? I know this would make Julia a statically type language (or at least optionally like in Groovy) and some might just hate that idea thinking mostly of C or Java  with little flexibillity but what I'm thinking of is more like F# or Scala: 

AOT compiled statically typed languages with type inference that also have a shell that behaves like a shell of a dynamic language. Given that Julia source code is anyway compiled (JITted) to native code using explicit type information if it is available I feel this is not such a big step and is a logical progression. By fully compiling ahead of time there might also be more time for more global optimizations (whole program optimization). If it would be statically typed (but with the flexibility of a dynamic language) it would also allow to build way more powerful I.D.E.s e.g. code completion, refactoring,...

Sincerely,

Steven Sagaert  

[Jacob Quinn]

Thoughts inline:

On Wed, Aug 13, 2014 at 7:25 AM, Steven Sagaert <steven....@gmail.com> wrote:

Hi,

I've been programming a recommender system in Julia for a couple of weeks and overall I like much of Julia and the choices that have been made but I've also found some things that annoy me and that I'd like to see changed. I figure that at 0.2 it's still a very young language with a small user base & ecosystem so I might actually have a chance of some of my requests actually make it into a future release.

* naming convention of the standard lib. For types Julia uses camelcase starting with uppercase (like Java for Classes) but for functions it's all lowercase with sometimes an underscore for separation. Having done a lot of Java programming this feels a bit weird to me. Why mix two different systems (camelcase & underscores?) Why not go for Java's system: camelcase (shorter than underscores) : function would start with lowercase (like Java methods) and types start with uppercase. I could live with all lowercase for functions but then I would like to see a consistent use of underscore for word separation. In the standard lib this is very inconsistent (many functions names have no separation where it should: e.g. haskey) and I find this very annoying since you never really know how a function name is written unless you look it up. In the absence of any code completion in the current development tool this little thing quickly becomes a source of irritation.

The trend has actually been moving swiftly away from any method names with underscores, (many have been removed or renamed). If there are still ones (in 0.3, which is about to be released officially any day) that you find irksome, please file an issue and I imagine there will be a good effort to rename or remove it. Apart from that, I personally really like the convention of camel case for types and all lowercase for methods. Also take a look at Sublime-IJulia or the Julia LightTable plugin, or other options for code completion.

 

* modules: I find the current way how modules are used in client code confusing & lacking. Why two systems (using vs import)? using is simple enough but then you have no control over which functions are imported into your client namespace leading to possible name clashes. Import seems confusing (import <module> vs import <module.function>) & when importing function you can import any (disregarding the export list) and hence breaking encapsulation? 

 

There are actually good reasons for both `using` and `import`, but it's definitely a point of difference between pretty much every language out there in how these things are handled. The docs on modules were recently updated to help clarify, so hopefully that helps, otherwise, issues on how to clarify the docs is always welcome.

 

I 've also found that export list of functions is annoying (it feels a bit like header files in C++):  it's easy to forget to add your functions to it especially when you are reworking code and since there's no compiler that complains you only find out about it at runtime with a cryptic error message.  

I tend to agree with this. I actually wrote some code at some point that would load a module and return a list of every function/variable/type that was defined and whether it was exported or not to handle this situation. It may very well be a good idea to allow function/type/global variable definitions to have `export` placed before them so that it doesn't have to be in two places.

 

There is no version data in the source code for a module. This could be really useful voor specifying dependencies with version numbers (with a package manager that support that) to get exactly what you need to avoid "dependency hell".

Take a look at the package manager docs here to learn about package version conventions used in Julia. This is actually very-well thought out already and the package ecosystem has thrived. The answer to your comments here is that versions are tied specifically to modules, but to "packages" which may be a single module or many modules as defined by the package author.

 

Here is what I propose:

Basically I think the Java/C# system works pretty well so this solution is modeled after it.

- use only one keyword/mechanism for importing modules (if we ignore include from the discussion). 

- annotate functions in the module with public/private keywords to mark if they are exported or not. If you want to save keystrokes one use the convention that no keyword=private so only public would be needed then. This pretty much garantees that you'll never forget to export your functions.

- on the client side only public functions can be imported & used (encapsulation is enforced). If you really want to override encapsulation (e.g. for debugger tool/I.D.E., other frameworks (think Spring in Java) this can be done via reflection/metaprogramming but not regular programming. 

A common approach in Julia has been one of "consenting adults", meaning that overly strict restrictions and blocks are avoided in place of common style and convention. If a package doesn't export methods/types, then I should expect them to be stable or exist as part of the public API and thus should avoid using them (particularly since Julia and most packages are in a rapid pace of development). Personally, I think it would just be annoying if I were able to figure out non-exported functions (through reflection/metaprogramming), but had to jump through hoops.

 

- import <module> imports all public functions in the client namespace. import <module.function> imports just this function in the namespace. In case of name clashes, the names are fully written in the client code: <module.function> but they still need to be imported.

- specify a version number next to the module name analogous to the C# mechanism & make the package manager aware of version number so that in dependencies can be precisely specified (similar to Maven,Gradle,...). This will save a lot of headaches once the library ecosystem starts to explode. R has this problem but now that it have thousands of packages already it's very hard to fix it anymore.

* packages should not be just a linear list of names (like in R which sucks big time if there a thousands) but be hierarchically organised in broad categories (like in Java,C#) and this organisation is reflected as directories in the filesystem. Julia standard libs should start with julia and external lib with the name of the firm/organisation/other meaningful name and have namesegments with a separator (e.g. a dot). For example julia.core,julia.net, julia.graphics,julia.gui,julia.db,julia.io,... Logically organising the packages like this will pay off in the long run when there are many.

There's been some discussion around adding a KEYWORDS file to packages to aid in discoverability, as well as the idea of doing something hierarchical. I don't think there's one, easy solution that would fit all cases, so I would guess that the KEYWORDS idea will be implemented pretty soon to aid in discoverability; with possible another site in the future that tackles some kind of curated hierarchy of sorts. 

With regards to folder/directory organization, I personally love the fact that Julia treats me like an adult and lets me organize my package however the heck I want. Being forced to deal with several layers of nesting in folders (Java) or include weird shell/import files in each folder (python) is just annoying and finicky; not to mention the inherent problems with file systems in general (case sensitive vs. not, etc.). No, I think Julia totally wins here by allowing you all the freedom you want in organizing your package without being finicky.

 

* When an exception is thrown, a full stacktrace should be printed starting at the place (module name, line number, function name) where it is thrown instead of printing the line number where it is caught which is pretty much useless. This would be a big improvement for debugging!

* exception handling: catch clause is too primitive: since you cannot specify the type of the exception it catches all exeptions even if you do not want it to and hence then you have to test the type(s) with a bunch of if/else statements and rethrow the exception types that you do not want to catch. Java/C# is much better/compact here: just specify the exception type (or a list of types e.g. separated by ,) after catch and then only that type is caught.

Please join a few active discussions on how to improve error handling in Julia, your input would be much appreciated and it's a hot topic right now! See issues here and here.

 

* I find myself writing a lot of type annotation in my Julia code. Basically all function arguments & local vars that do not have an initial value get a type. I do this for self documentation & for helping the compiler generate efficient code. So my code is not that different than it would be from a statically typed language. Unfortunately type inconsistencies are only discovered at runtime and once you start specifying types in Julia (instead of being fully dynamic) it can be a slippery slope because it tends to go viral so that you need to narrow the types of other variables as well (e.g. String vs ASCIIString). I figure: if most of my code has type information anyway (& the rest can be easily inferred from type inference) why not go all the way and have the option of an ahead of time statically compiler that would flag type errors at compile time and that would generate a binary executable? I know this would make Julia a statically type language (or at least optionally like in Groovy) and some might just hate that idea thinking mostly of C or Java  with little flexibillity but what I'm thinking of is more like F# or Scala: 

AOT compiled statically typed languages with type inference that also have a shell that behaves like a shell of a dynamic language. Given that Julia source code is anyway compiled (JITted) to native code using explicit type information if it is available I feel this is not such a big step and is a logical progression. By fully compiling ahead of time there might also be more time for more global optimizations (whole program optimization). If it would be statically typed (but with the flexibility of a dynamic language) it would also allow to build way more powerful I.D.E.s e.g. code completion, refactoring,...

You're in luck since static compilation is on it's way. 

 

Sincerely,

Steven Sagaert  

[Jeff Bezanson]

Thanks for taking the time to write all of this, it's helpful.

I agree being able to write "export function foo(x) ... end" would be
good. Having to write things in 2 places is always annoying.

> * When an exception is thrown, a full stacktrace should be printed starting at the place (module name, line number, function name) where it is thrown instead of printing the line number where it is caught which is pretty much useless. This would be a big improvement for debugging!

This one confused me: we do print stack traces from where errors are
thrown, e.g.

julia> function foo(x)
x += 1
error("oops")
end
foo (generic function with 1 method)

julia> foo(1)
ERROR: oops
in foo at none:3

If this were showing where the error was caught it would be somewhere
in the REPL code. Maybe post an example of the behavior you saw?

On Wed, Aug 13, 2014 at 7:25 AM, Steven Sagaert
<steven....@gmail.com> wrote:

[Stefan Karpinski]

I think we should do away with import altogether and just have

using Foo
using Foo: bar

The first would load and create a binding for Foo, make it's exports available as "soft" bindings (what using does now). The second would also load and create a binding for Foo, and make bar available as a "hard" binding (what import does now).

[John Myles White]

+1 for that

-- John

[Miles Lubin]

I've found "import Foo" valuable for increasing the readability of code that depends on many modules, because it forces you to qualify identifiers. You still can do this with "using", but I'd prefer to have this be enforced by the compiler.

[John Myles White]

Can't you do that with require("Foo")?

 -- John

[Miles Lubin]

That would work, although it's much uglier and doesn't resolve the potential confusion from users who don't know what it does exactly.

[Steve Kelly]

I think the promise of multiple dispatch is that we can avoid namespacing since it can become implicit in the type annotation. This presents challenges in creating cohesive design, but the net benefits are much simpler and clear code for library users. I like to think of methods using multiple dispatch as verbs. Stefan already outlined this concept here. https://groups.google.com/forum/#!searchin/julia-dev/static$20compile/julia-dev/dJ8RhCxaacc/zNpv6453RqgJ

[Stefan Karpinski]

You can still do `using Foo` and then write Foo.bar to explicitly qualify names. We could also allow some syntax that would only bind Foo. Possibilities are:

using Foo: ;

using: Foo

The first one is the explicit import syntax with no explicit imports. The second one works on the premise that the names to left of the colon are a path to find something while the names to the right of the colon are what to import. In this case, the path to find Foo is empty since being a top-level module, it's at the root.

[Stefan Karpinski]

I would also argue that we should limit things that can go on the right-hand-side of the `using Foo: bar` syntax to names that are exported from Foo. You would still be able to access unexported Foo.baz, but importing it with using seems like asking for issues.

[Tobi]

I would like to throw in example:

Lets consider `Gtk` having a type `Window`. The issue now is that the term `Window` is pretty generic and might lead to a name clash. For this reason the type is called `GtkWindow` which makes the type unique but the methods can be used without the `Gtk.` prefix which would be required when using `import` instead of `using`.

One alternative would be to not export all the types but only the methods. Then one would use `Gtk.Window` but still use the methods without the namespace. But this would be a little strange to have a non-exported "public" type. Maybe the namespace concept and the visibility (i.e. public vs. private) are not orthogonal enough in Julia?

My general feeling is that namespaces for methods are not really needed (because of multiple dispatch) but for types they are useful and the above example illustrates that its a little hard to separate these two currently.

P.S.: There is Gtk.ShortNames but the above was mostly to give an example.

[Stefan Karpinski]

Yes, that's how I chose to make the Pkg interface work since it wanted a lot of generic names (functions in that case, actually). With my proposal, you could still get that effect by doing

using Gtk

Gtk.Window

If there was no other package exporting the name Window, then you could just write Window. With the proposal in #4345, if another package exported the name Window, then you would no longer be able to write just Window and would have to write Gtk.Window.

[Tobi]

Yes #4345 is a good proposal. And Pkg is indeed a pretty interesting example because all the functions are global and do not belong to some type. So I assume that you simply do not export the methods within the Pkg module. But is `export` really the right term for what is happening? I mean "export" implies for me "public interface" but here we have no export but a public interface...

[Stefan Karpinski]

I agree that "export" is not a great term. Pkg exports a bunch of things, but this is really just to a) document the public interface and b) make it easy to do `importall Pkg` and paste code from the Pkg module into the REPL and have it work without having to qualify all the names.

[Neal Becker]

Steve Kelly wrote:

> I think the promise of multiple dispatch is that we can avoid namespacing
> since it can become implicit in the type annotation. This presents
> challenges in creating cohesive design, but the net benefits are much
> simpler and clear code for library users. I like to think of methods using
> multiple dispatch as verbs. Stefan already outlined this concept here.
> https://groups.google.com/forum/#!searchin/julia-dev/static$20compile/julia-dev/dJ8RhCxaacc/zNpv6453RqgJ
>

...

Really? I don't see it. polymorphic functions can take quite general parameter
types. There's nothing preventing a clash as far as I can see.

Why shouldn't package A define a function 'sum', and package B also define a
function 'sum'? Maybe they even do 2 very different things.

[Stefan Karpinski]

The point is that they can share the same generic function if what they do is conceptually the same – even if they are implemented in radically different ways. This cuts down on clashing over the names of functions since much of the time you can share a function instead. With single dispatch you can also share "verbs" like this, but in Java and C++ at least, you have to anticipate what functions different types might want to share and put them in a common base class (or interface). With external generic functions, that's not true – you can come up with new verbs independent of someone else defining the types. This lets people share common types more easily, which cuts down on clashes over noun names. In short, multiple dispatch makes it more likely that a lot of different packages can share a "render" function and share a "Window" type, but of which reduce the need for namespace granularity. Our graphics packages don't actually do this very well yet, but they hopefully will in time.

[Iain Dunning]

Well-written post. 

I think a lot of the things you mention are preferences based on your past experience, I can't really comment on most of that as its mostly opinion/personal preference - I like most of things you dislike, so, not much point in exploring that further.

Other more actionable things:

In the absence of any code completion in the current development tool this little thing quickly becomes a source of irritation.

The Julia REPL has code completion, and there are plugins for many editors - what development tool do you use?

* modules: I find the current way how modules are used in client code confusing & lacking.

 

using: bring all exported names into the namespace

import: bring only what you specifically import

The updated manual (which you might not have seen if you've only looked at the 0.2 release manual) explains this very well.

The export thing is valid, as is the need to "reexport" things from nested modules. Needs work!

There is no version data in the source code for a module

- specify a version number next to the module name analogous to the C# mechanism & make the package manager aware of version number so that in dependencies can be precisely specified 

 

All packages have version numbers? I don't get this one... We have quite good dependency handling, few rough edges but the core is pretty solid (and working well in production environments, apart from more human issues like people stopping support for old Julia version early). Maybe I'm missing something?

on the client side only public functions can be imported & used (encapsulation is enforced)

Well-discussed above, but this goes against a core Julia philosophy - "I like being treated like an adult." Documentation/naming is the solution to this. I've never found this policy to cause any problem in Julia code.

 I find myself writing a lot of type annotation in my Julia code.

I've seen people do this, and I'm not sure why. It doesn't usually help performance at all... its odd behaviour. I also see people putting `local` in their code and things like that.

I only really use type annotation on function signatures to be defensive. It doesn't actually improve performance though. This is an education thing, I suppose.

* packages should not be just a linear list of names (like in R which sucks big time if there a thousands) but be hierarchically organised in broad categories (like in Java,C#) and this organisation is reflected as directories in the filesystem. Julia standard libs should start with julia and external lib with the name of the firm/organisation/other meaningful name and have namesegments with a separator (e.g. a dot). For example julia.core,julia.net, julia.graphics,julia.gui,julia.db,julia.io,... Logically organising the packages like this will pay off in the long run when there are many.

I agree that organization is good - we already have that in the human and technical senses of the word, e.g. 

http://juliaopt.org

http://juliastats.github.io/

http://juliadiff.org

The main benefit of this is interoperability of similar packages, and I think we've demonstrated that this is possible.

I don't see any benefit to reflecting this in the file system thing, or what problem it solves. Java does this, but I've never been sure why. Certainly Python, for example, does not seem to have hit any problems with its approach.

I hope you stick with Julia! Some of the discussion in this thread is really great already, fresh users willing to share their opinions really helps kick the tires on things we just get accustomed to.

[TR NS]

I would like to discuss the module issue because, as someone seriously considering making Julia my primary language for, possibly, the rest of my life, I obviously won't be happy unless the system rocks hotcakes. 

The problems with the current system starts with the simple fact that it is terribly confusing. I have read the docs a half dozen times now, and I still can't tell you the difference between `import` and `using` other than the fact that I can't rewrite a function unless I use `import`.

The next problem, and really this is the biggy, is that the current system is almost antithetical to encapsulation, which is the whole point of having modules in the first place. It basically calls for dumping everything into one pot and cries "multiple dispatch" as a rallying justification. But multiple dispatch, while it helps, is not really sufficient b/c it assumes every module's functions will take different types when in reality most take the same common types we all use. So there is still far too much potential for name clash. I think an analogy would be helpful here. In OOP we have inheritance, and sometimes inheritance is the right approach, but all good OO programmers know the better approach is usually composition. With composition we create a handle by which we can access certain functionality, keeping it separate from other functionalities. A good module system would support that kind of approach just as much, if not more than, an "inheritance" approach.

On top of all that, we also have `include` and `require` mechanisms, which is yet another layer of confusion. Include itself is a poor way to handle loading b/c it tends to create a inverted relationship between the components of a program. Think of PHP's use of $header and $footer in every page, when what is way better are layouts with a single central $yield for the body. It's the same basic problem here. What is needed is a modular composition system, not a mimicking of cut-and-paste via code. As for require, the fact that module names clash, well that just sucks in and of itself.

So what I would like to see is something of a hybrid between Ruby and Node.js approaches. Ruby has something they call "open classes". The idea is that if you require a file that has a module defined, say `Foo`, and `Foo` happens to already be defined, the definitions just merge. It will throw out a warning if two methods clash (the later of which will override the former) but that's it. It makes it so much easier to decompose one's program as best suits the program itself. For Node.js the ability to load in code and give it a name, e.g. `foo = require('Foo')` means name clashes are not even a concern. 

To combine these two concepts, `require` can be used to make modules available, merging as described above, but no actual code invades the scope (the exception might be toplevel code, but I am not so sure that's a good idea).

    require "foo"

This makes `Foo` available to we could now do:

    Foo.bar(...)

Or we can `use` the available modules, bring their functions in, either whole clothe:

    use Foo

Or piecemeal of any *exported* functions.

    use Foo(bar, baz)

So now our code and have:

    bar(...)

But we we can also define a compositional handle to use instead.

    use Foo as f

    use Foo(bar, baz) as f

Then in our code we can use it as:

    f.bar(...)

The exact syntax not withstanding, that is all that is needed for the whole system.

[Steven Sagaert]

that's exactly what I mean't in my post (except I used the keyword 'import' like in Java. This is also how it works in Java and this has worked out pretty well since the 1995.

So +1 on that. 

[TR NS]

Sorry for the noise, I just posted a reply to this topic and it appears to have vanished into thin air. Is it possible it got tied up in the moderation queue? If so, please do me a favor and see it through. Thanks.

[Stefan Karpinski]

On Aug 13, 2014, at 3:40 PM, TR NS <tran...@gmail.com> wrote:
>
> Sorry for the noise, I just posted a reply to this topic and it appears to have vanished into thin air. Is it possible it got tied up in the moderation queue? If so, please do me a favor and see it through. Thanks.

Sorry about that – first time posts are moderated since otherwise we get recruiter spam on the mailing lists.

[Iain Dunning]

TR NS, I feel like that is what we already have. I think we can disregard `require`, I don't feel like its ever needed. Let me annotate your post

To combine these two concepts, `require` can be used to make modules available, merging as described above, but no actual code invades the scope (the exception might be toplevel code, but I am not so sure that's a good idea).
    require "foo"

This is, right now, `import Foo`, which brings `Foo` and nothing else into the scope.

This makes `Foo` available to we could now do:
    Foo.bar(...)

This is how it works. 

Or we can `use` the available modules, bring their functions in, either whole clothe:
    use Foo

This is `using` now 

Or piecemeal of any *exported* functions.
    use Foo(bar, baz)

This is `import Foo: bar, baz` now

So now our code and have:
    bar(...)
But we we can also define a compositional handle to use instead.
    use Foo as f
    use Foo(bar, baz) as f

We don't have this yet, but there is an open issue for it. 

Then in our code we can use it as:
    f.bar(...)
The exact syntax not withstanding, that is all that is needed for the whole system.

So apart from aliasing, we are already there.

[Trans]

HI Ian, 

Maybe it is more there than I realized, but there are also some differences.

So first, as I explained about reading the docs and still being confused, I was mistaken about `import Foo` rather it is `importall Foo` that does this. So there is yet another function to add to the confusion. ;-)

The difference with `import` and `require` is that require is file based, which makes it clear what file is being loaded. If we depend on `import` for this, there needs to be enforced correspondence between module names and file names. Considering files can have multiple modules defined in them, I think it is much less confusing to use `require`, keeping the loading of code completely separate from `using` it in modules.

Also, there is the difference concerning "open modules". For example:

  # foo.jl

  module Foo

    function x ...

  end

  # foo/bar.jl

  module Foo

    function y ...

    module Bar

      function bq ...

    end

  end

  # main.jl

  require 'foo'

  require 'foo/bar'

Would just work. `Foo.x`, `Foo.y` and `Foo.Bar.bq` would all be available for use.

Besides that and the aliasing, as you mentioned, then yeah, I suppose it is close, but there are too many functions which mess it all up. All we need are the two: `require` and `use` (or `using`)  ... although maybe `import` is a better term than `use` to contrast with `export`. Well, in any case the functionality would be the same.

Hey, btw, what do you all think of Go's approach to exporting functions by capitalizing them?

[Stefan Karpinski]

On Wed, Aug 13, 2014 at 5:35 PM, Trans <tran...@gmail.com> wrote:

HI Ian, 

Maybe it is more there than I realized, but there are also some differences.

So first, as I explained about reading the docs and still being confused, I was mistaken about `import Foo` rather it is `importall Foo` that does this. So there is yet another function to add to the confusion. ;-)

This is definitely an area that needs simplification. It has evolved over several years and there are some leftover bits and pieces that aren't really needed anymore. For example, there's no reason to use `require` these days – we should unexport it and deprecate it. The `importall` function is only meant to be used out of laziness when you don't want to mess around with specific imports while developing.

The only functions that are really relevant are include, import and using. I've proposed merging import and using above, which would bring it down to two ways of loading code – include and import. These are quite different and both useful. The include function loads a file as if it were included in the current file. This is a simple mechanism for splitting a file up into multiple files, which one sometimes wants to do. It also allows the same code to sed in different contexts with different meanings. The import and using keywords are both used to acquire package resources. They load a package if necessary and make it and its contents available. A package is only ever loaded once by this mechanism; after the first time, all these keywords do is manipulate bindings. The only difference between import and using (and importall) is what they do with bindings.

 

The difference with `import` and `require` is that require is file based, which makes it clear what file is being loaded. If we depend on `import` for this, there needs to be enforced correspondence between module names and file names. Considering files can have multiple modules defined in them, I think it is much less confusing to use `require`, keeping the loading of code completely separate from `using` it in modules.

Also, there is the difference concerning "open modules". For example:

  # foo.jl

  module Foo

    function x ...

  end

  # foo/bar.jl

  module Foo

    function y ...

    module Bar

      function bq ...

    end

  end

  # main.jl

  require 'foo'

  require 'foo/bar'

Would just work. `Foo.x`, `Foo.y` and `Foo.Bar.bq` would all be available for use.

This is very Rubyesque – I'm assuming that's where your inspiration is coming from. Using path-based loading is ok in Ruby. The convention that paths are lowercase while classes are uppercase can be annoying at times (how exactly is the lowercased version of the camelcase class name you want is spelled – is there an underscore or not?). But mostly it's pretty good. And this is exactly how things used to work in Julia. That's why require exists. But it doesn't work well in Julia.

First, why does it work well in Ruby? In Ruby classes are namespaces. Defining a Ruby class is enough to start using it. Given the Foo class, you can construct a Foo object by doing `foo = Foo.new()`. Then you can use functionality of the Foo class by writing foo.bq(). The only thing that Ruby's require keyword needs to do is ensure that each path is only loaded a single time.

Why does this not work well in Julia? There aren't any classes. The functions you want to use on types aren't "inside" the objects they apply to – they are separate, external entities. When you require("foo"), you still don't have access to any of the things that are defined in Foo – you need to have bindings to those things also. Thus, inevitably, the code looks like this:

require("Foo")

using Foo

This redundant code was rampant for a while. And it was really annoying. Why would I require "Foo" if I didn't want to use it? Likewise, if I write `using Foo` and there's nothing called Foo yet, it's not a wild stretch of the imagination that I want to require whatever provides Foo – typically a file named Foo.jl. Why can't this just do both for me at once: load the code if necessary *and* create the bindings I want. This is precisely what `using` now does. If you write `using Foo` and Foo does not exist, Julia looks in the usual places (Pkg.dir() and LOAD_PATH) for Foo.jl and loads it, with the expectation that it will define a module named Foo.

The ability to reopen classes in Ruby is arguably the most heavily abused feature in the language. "Monkey patching" – the colloquial term for opening a class up and altering it – is one of the biggest problems with the Ruby ecosystem. It's such a big problem that Ruby 2.0 attempted to add "refinements" – i.e. dynamically scoped monkey patching (it sounds much less, um, refined when you put it that way, right?). I say attempted, because the feature is so complex that 2.0 didn't end up fully implementing it.

 

Hey, btw, what do you all think of Go's approach to exporting functions by capitalizing them?

I do not like it. Do all public names have to begin with an uppercase ASCII character? What about people coding in languages where there is no such thing as uppercase – do they still need to start everything with an English letter? What about operators? If you're allowed to start public names with characters in other alphabets, does the parser assume US English locale? Would being in a different locale change the meaning of code? Unicode capitalization is a horrorshow. Dealing with it in strings is bad enough, I really don't want it anywhere near the parser.

[John Myles White]

For what it's worth, I think importall makes code much harder to read. We used to depend on it in DataFrames, but we started to do explicit method-addition with Base.foo = bar. I've found the code much easier to understand now.

 -- John

[Iain Dunning]

I think I share the general Julia preference for minimal-ness so if we can deprecate "require", that'd be great. I'd also be in favour of getting rid of importall - I feel like if it is indeed desired functionality, then we can add it to import/using/a merger of the two somehwo.

--

Iain Dunning

PhD Candidate / MIT Operations Research Center

http://iaindunning.com  /  http://juliaopt.org

[Stefan Karpinski]

Importall should only be used during development.

On Wed, Aug 13, 2014 at 7:04 PM, John Myles White <johnmyl...@gmail.com> wrote:

[gael....@gmail.com]

@TR NS: your comparison with inheritance vs. composition is great and I think this is why I prefer full name resolution if I have the slightest doubt about code clarity.
Raw 'USE' statements in Fortran and 'from foo import *' in Python are not heavily discouraged just because full names are more funny to write.

+1 on include and import only.

I guess someone already proposed it but, why not using the Python approach here?

import Foo:***

to import all exported stuff (current using).

import Foo: bar, baz

to import exported bar and baz. Non-exported stuff throws an error.

import Foo

to import the module Foo only.

import Foo.priv_bq

to import anything from Foo, even non-exported stuff.

import Foo.***

to import everything, exported or not from Foo.

The rules are simple:

- *** means all
- : means access to the exported attributes
- . means unrestricted access to any attribute.

[Trans]

On Wed, Aug 13, 2014 at 7:01 PM, Stefan Karpinski <ste...@karpinski.org> wrote:

This is definitely an area that needs simplification. It has evolved over several years and there are some leftover bits and pieces that aren't really needed anymore. For example, there's no reason to use `require` these days – we should unexport it and deprecate it. The `importall` function is only meant to be used out of laziness when you don't want to mess around with specific imports while developing.

The only functions that are really relevant are include, import and using. I've proposed merging import and using above, which would bring it down to two ways of loading code – include and import. These are quite different and both useful. The include function loads a file as if it were included in the current file. This is a simple mechanism for splitting a file up into multiple files, which one sometimes wants to do. It also allows the same code to sed in different contexts with different meanings.

Include is a terrible way to load code as I tried to explain before. It totally destroys modular encapsulation. I ran into this problem immediately on my first Julia program when I tried to use submodules and hit issues where the submodules could not be found. In the end I had to make a choice between using the submodules or the file layout I wanted. It doesn't work well with documentation either (which is really just a symptom of the bad encapsulation). Consider if you have file a.jl made up of a bunch of functions. Then you include it in Foo and Bar modules in f.jl. How do we document the functions in a.jl? Will they be documents twice as functions of module Foo and Bar, or just once as the file a.jl, but then how clear is it how they are imported? Include is really anti-encapsulation --it isn't modular composition at all, it's just a cheap trick to copy and paste code automatically.

Unless... you get rid of `module` altogether, and just make files = modules. You can do that. Then `include` and `import` become one and the same thing. But then we have to live with the one file = one module limitation. 

The import and using keywords are both used to acquire package resources. They load a package if necessary and make it and its contents available. A package is only ever loaded once by this mechanism; after the first time, all these keywords do is manipulate bindings. The only difference between import and using (and importall) is what they do with bindings.

This is very Rubyesque – I'm assuming that's where your inspiration is coming from. Using path-based loading is ok in Ruby. The convention that paths are lowercase while classes are uppercase can be annoying at times (how exactly is the lowercased version of the camelcase class name you want is spelled – is there an underscore or not?). But mostly it's pretty good. And this is exactly how things used to work in Julia. That's why require exists. But it doesn't work well in Julia.

That is making the assumption that file and module are somehow equivalent. That is a very Javaesque way of thinking. In Ruby the file and module names do not have to have anything to do with each other. They can if you want them to, and often do in the case of class definitions, but there are plenty of great use case where they are not. For example, Ruby Facets (a library I developed) is actually broken down into cherry pickable extensions. So the files are named after the methods not modules. Also `require 'facets/common'` will load the most common selection of these methods. Again the naming of the files is not dependent on any module.

 

First, why does it work well in Ruby? In Ruby classes are namespaces. Defining a Ruby class is enough to start using it. Given the Foo class, you can construct a Foo object by doing `foo = Foo.new()`. Then you can use functionality of the Foo class by writing foo.bq(). The only thing that Ruby's require keyword needs to do is ensure that each path is only loaded a single time.

Why does this not work well in Julia? There aren't any classes. The functions you want to use on types aren't "inside" the objects they apply to – they are separate, external entities. When you require("foo"), you still don't have access to any of the things that are defined in Foo – you need to have bindings to those things also.

But there are modules, and functions are defined *in* modules, i.e. units of encapsulation.

 

Thus, inevitably, the code looks like this:

require("Foo")

using Foo

This redundant code was rampant for a while. And it was really annoying. Why would I require "Foo" if I didn't want to use it? Likewise, if I write `using Foo` and there's nothing called Foo yet, it's not a wild stretch of the imagination that I want to require whatever provides Foo – typically a file named Foo.jl. Why can't this just do both for me at once: load the code if necessary *and* create the bindings I want. This is precisely what `using` now does. If you write `using Foo` and Foo does not exist, Julia looks in the usual places (Pkg.dir() and LOAD_PATH) for Foo.jl and loads it, with the expectation that it will define a module named Foo.

Because Bar might be defined in Foo.jl too. So you have to do this anyway to get Bar, and now things are getting confusing. It may seem annoying, but it is simple. Instead of `using Foo` looking for a file called `Foo.jl`, use `require 'Foo'` and Foo and Bar are both made available. If you really feel it necessary to load and use in one shot, `use Foo from "foo"` is very clear. So the annoyance is gone but you also don't have to worry about the other annoyance of file and module name correspondence.

 

The ability to reopen classes in Ruby is arguably the most heavily abused feature in the language. "Monkey patching" – the colloquial term for opening a class up and altering it – is one of the biggest problems with the Ruby ecosystem. It's such a big problem that Ruby 2.0 attempted to add "refinements" – i.e. dynamically scoped monkey patching (it sounds much less, um, refined when you put it that way, right?). I say attempted, because the feature is so complex that 2.0 didn't end up fully implementing it.

I am an expert in this, and you completely misunderstand it. It is actually one of Ruby greatest strengths! It is what made Rails even possible. "Monkey patching" is a term that came from outside of the Ruby community and has been hawked by people who really didn't know what they are talking about. Refinements are just a way Matz devised that we could use to be more careful to not to step on each others toes. But many in the community actually don't think refinements are necessary b/c it's not something that is often a problem and when it is it is quickly resolved. Refinements haven't gotten much play, to say the least. But open classes are not going anywhere. I should point out too, that open classes are not just about "patching", they also make is simple to *add* functionality, including submodules, and thus conversely, simple to decompose functionality.

 

 

Hey, btw, what do you all think of Go's approach to exporting functions by capitalizing them?

I do not like it. Do all public names have to begin with an uppercase ASCII character? What about people coding in languages where there is no such thing as uppercase – do they still need to start everything with an English letter? What about operators? If you're allowed to start public names with characters in other alphabets, does the parser assume US English locale? Would being in a different locale change the meaning of code? Unicode capitalization is a horrorshow. Dealing with it in strings is bad enough, I really don't want it anywhere near the parser.

 

Actually I agree. I do not like it either. In fact I something wish it wasn't necessary to capitalize module names. I think capitalization is a poor syntax dependency. On the other hand it does allow for things to be a bit more concise. So it's a trade off.

[Jared Kofron]

On the subject of submodules, I find the way that they are declared very confusing.  Something I was wondering while trying to make sense of it the other day - what about a simple syntax for submodule declaration such as

submodule Bar of Foo

...

end

which gets around the include(...) altogether (which I don't think is very pretty).  

[Iain Dunning]

A submodule isn't anything special, and doesn't extra syntax. I'm not sure what that syntax would gain you?

I'm really not convinced these are problems. Maybe I'm being overly defensive, but while there is room for debate over the design I feel like the tone is that something is horribly wrong with Julia's module system right now. I'd suggest reading through the codebases of some of the larger Julia projects to get a feel for how these work in practice, because the only thing I can think of that is really causing any problems in them is conditional module includes, which is an open issue.

[Jared Kofron]

Inlined

On Wednesday, August 13, 2014, Iain Dunning <iaind...@gmail.com> wrote:

A submodule isn't anything special, and doesn't extra syntax. I'm not sure what that syntax would gain you?

Despite the lack of specialness, to me the gain is readability. If I read a separate source file with a separate module declaration, the connection to the parent module is obfuscated. 

 

I'm really not convinced these are problems. Maybe I'm being overly defensive, but while there is room for debate over the design I feel like the tone is that something is horribly wrong with Julia's module system right now. I'd suggest reading through the codebases of some of the larger Julia projects to get a feel for how these work in practice, because the only thing I can think of that is really causing any problems in them is conditional module includes, which is an open issue.

 I wouldn't say horribly wrong. But it can definitely be a little confusing and the syntax doesn't do a great job of communicating what's happening. 

[Jacob Quinn]

Issue filed for deprecating require, and perhaps import/importall: https://github.com/JuliaLang/julia/issues/8000  (<- that's a nice round number there :)

Trans,

I don't understand the following:

 Consider if you have file a.jl made up of a bunch of functions. Then you include it in Foo and Bar modules in f.jl. How do we document the functions in a.jl? Will they be documents twice as functions of module Foo and Bar, or just once as the file a.jl, but then how clear is it how they are imported? Include is really anti-encapsulation --it isn't modular composition at all, it's just a cheap trick to copy and paste code automatically.

`include`, for me, has great advantages working at the REPL (for reloading modules, running a script, etc.), and I find it convenient for simple file management in the packages I've developed. Take ODBC.jl for example, there's a huge API.jl file that defines all the `ccall` wrapper functions, but I don't want them cluttering up other files with user-facing functions. And I don't want to make a submodule, because it's just a bunch of `ccall` wrappers. `include` is perfect here because right at the beginning of my ODBC.jl root module file, I can do `include("api.jl")` and all those low-level wrappers are available. I think this pattern comes up all over the place: having lower level or utility functions that don't necessitate a submodule, but that you want to push into a separate file and just "copy-paste" into a root module. Anyway, just thought I'd share how I've personally used it.

-Jacob 

[Mike Innes]

Personally I really like Julia's module system, largely because files and modules are orthogonal thanks to include. It's not universally correct to have huge modules with hundreds of definitions or small ones with a few exports each, so Julia lets you pick either option without stopping you from organising the code itself in a logical way.

[include] totally destroys modular encapsulation.

In the end I had to make a choice between using the submodules or the file layout I wanted.

Like I said, file layout and modules are independent, so this shouldn't really be an issue. You could have a hundred nested modules in one file or one module across a hundred files or whatever you want.

Consider if you have file a.jl made up of a bunch of functions. Then you include it in Foo and Bar modules in f.jl.

I think I see what you mean about destroying encapsulation now, but this demonstrates some misunderstanding of include's purpose and use – you'd never do this in Julia. What you might do is something like:

#a.jl

module A

...

end

#f.jl

include("a.jl")

module Foo

using A

...

end

module Bar

using A

...

end

Include is a way to split up big files into smaller files and nothing more. Encapsulation comes separately via modules.

As Iain suggests, I think it would be worthwhile to scan over some packages and get a better feel for the Julian way of doing these things. In general if you want to take full advantage of Julia you'll probably have to embrace a slightly different way of doing things (just the same as if you switched from Ruby to Java, for example).

[Trans]

On Wed, Aug 13, 2014 at 9:26 PM, Mike Innes <mike.j...@gmail.com> wrote:

As Iain suggests, I think it would be worthwhile to scan over some packages and get a better feel for the Julian way of doing these things. In general if you want to take full advantage of Julia you'll probably have to embrace a slightly different way of doing things (just the same as if you switched from Ruby to Java, for example).

Actually I did exactly that when I first ran into the issues with include. What I discovered is that every Julia program I could find did pretty much the exact same thing: they dumped all their functions from separate files into one main module.

This is thread to which I am referring: https://groups.google.com/d/msg/julia-users/rtS2j6A4sUE/sIWvzTqewt0J

[Jeff Bezanson]

Mike is right, nobody has ever or would ever claim that `include` is
some kind of approach to modularity. It's not; it just tells the
system to read and execute a file.

Over time a couple people have felt that julia "dumps everything into
one namespace" and causes collisions. This isn't really true. If you
don't use `importall`, then everything you define is a fresh,
module-local binding, not colliding with anything. e.g.

julia> module Sin
sin(x) = 0
end

julia> Sin.sin(1)
0

julia> sin(1)
0.8414709848078965

Yes, within the `Sin` module the Base `sin` definition is shadowed,
but this can't be helped. An un-qualified name can only have one
binding within a given scope.

On Wed, Aug 13, 2014 at 9:26 PM, Mike Innes <mike.j...@gmail.com> wrote:

[Iain Dunning]

For a package using submodules extensively, checkout Gadfly, e.g. here is the Geom submodule: https://github.com/dcjones/Gadfly.jl/blob/a7182a044355b2575146f0630d26cd246c316d9b/src/geometry.jl

which itself `includes` various bits that separated for the sake of developer convenience.

i.e. Gadfly.Geom is a submodule for design/user-visible reasons

but its code is made up with includes for development reasons.

[Jared Kofron]

Definitely a good example.

Just my two cents: I think that this example also illustrates that the submodule system needs a little

further explanation in the documentation.  It doesn't feel when I'm reading the code that there's any real

mechanism behind the submodule - it essentially feels like a workaround to accomplish nested modules.

The Geom submodule doesn't indicate in any way that it's a submodule of Gadfly - in fact without more

context, it could be a submodule of any of the other modules that are in the using declarations.

That's fine, but it *is* confusing for somebody who is expecting the module/submodule system to be

a tool for code organization beyond the include mechanism.  Without any way to declare that the 

code "belongs" to some larger structure, the submodule system when used across multiple files

can really obfuscate organization.

[John Myles White]

FWIW, I think the debate here is that some people want modules to behave like absolute paths (in the sense that a module’s name immediately makes it clear that it “belongs” to a larger structure), whereas the current module system behaves as if only relative paths were meaningful.

The use of "relative paths” does make local reasoning a little harder because you can’t immediately know what the parent of a module is by looking at its code. But it also means that relocating modules is easier because they never embed their absolute position, only their relative position.

 — John

[Trans]

Is code re-usability being considered? Some questions:

1. How does one make a package that has more than one reusable module? For instance I have a Corpus program that can do word analysis, bigram analysis and letter analysis. As things stands importing Corpus makes all them available. How could I just import `Corpus.Bigrams` for example and not the rest?

2. If there is a submodule in another package, and I realize that it is almost exactly what I need for my program, how can I import it so that I can use it but with a modification (e.g. redefining a method or two)? 

3. If I find a useful set of functions in another package (not encapsulated in a module) how to I import these into my program?

I think another way to express my issue with the system is that it like there are two systems. The include system for local code reuse, and then the import system for package reuse (which is limited to one entry point). When there could be just one unified system.

[Iain Dunning]

1. Yes, of course.

module Corpus

  export Bigrams

  module Bigrams

    export foo 

    foo()

  end

  bar()

end

# One way...

using Corpus

Bigrams.foo()

# Another way

import Corpus

Corpus.Bigrams.foo()

# or even

import Corpus: Bigrams

Bigrams.foo()

2. Not entirely sure what you mean, but let me try

module Corpus2Point0

  export Bigrams

  import Corpus: Bigrams

end

Not really sure precisly what you mean re: redefining.

3. This sounds hypothetical. I feel like in any language you'd just copy the code.

[Steven Sagaert]

On Wednesday, August 13, 2014 9:26:08 PM UTC+2, Iain Dunning wrote:

Well-written post. 

I think a lot of the things you mention are preferences based on your past experience, I can't really comment on most of that as its mostly opinion/personal preference - I like most of things you dislike, so, not much point in exploring that further.

Other more actionable things:

In the absence of any code completion in the current development tool this little thing quickly becomes a source of irritation.

The Julia REPL has code completion, and there are plugins for many editors - what development tool do you use? JuliaStudio. I haven't really looked for it. I just assumed it didn't exist (or would be just text search which is usually to coarse) based on experiences with other dynamic languages. sorry! I am just a Julia newbie.

* modules: I find the current way how modules are used in client code confusing & lacking.

 

using: bring all exported names into the namespace

import: bring only what you specifically import

The updated manual (which you might not have seen if you've only looked at the 0.2 release manual) explains this very well.

The export thing is valid, as is the need to "reexport" things from nested modules. Needs work!

There is no version data in the source code for a module

- specify a version number next to the module name analogous to the C# mechanism & make the package manager aware of version number so that in dependencies can be precisely specified 

 

All packages have version numbers? I don't get this one... We have quite good dependency handling, few rough edges but the core is pretty solid (and working well in production environments, apart from more human issues like people stopping support for old Julia version early). Maybe I'm missing something?

on the client side only public functions can be imported & used (encapsulation is enforced)

Well-discussed above, but this goes against a core Julia philosophy - "I like being treated like an adult." Documentation/naming is the solution to this. I've never found this policy to cause any problem in Julia code.

Well I like being treated as an adult too ;) but being in industry and not academia and having worked on large multiperson projects it's nice to be able to ensure that other people are not going to use your internal code because that might change in the future and then the client code breaks... Not everybody is going to behave nicely and some people will use whatever they can get their hands on if they can take shortcuts even if they are not supposed to. It's all about being able to refactor large codebases in the future.

 I find myself writing a lot of type annotation in my Julia code.

I've seen people do this, and I'm not sure why. It doesn't usually help performance at all... its odd behaviour. I also see people putting `local` in their code and things like that.

I only really use type annotation on function signatures to be defensive. It doesn't actually improve performance though. This is an education thing, I suppose.

well I don't do things like: local var::String="test, I write that as: var="test" but in function signatures all params that do not have a default value get a type annotation. Also once you start doing that types creep in in initalization code like : paths=String["test","test2"] 

* packages should not be just a linear list of names (like in R which sucks big time if there a thousands) but be hierarchically organised in broad categories (like in Java,C#) and this organisation is reflected as directories in the filesystem. Julia standard libs should start with julia and external lib with the name of the firm/organisation/other meaningful name and have namesegments with a separator (e.g. a dot). For example julia.core,julia.net, julia.graphics,julia.gui,julia.db,julia.io,... Logically organising the packages like this will pay off in the long run when there are many.

I agree that organization is good - we already have that in the human and technical senses of the word, e.g. 

http://juliaopt.org

http://juliastats.github.io/

http://juliadiff.org

The main benefit of this is interoperability of similar packages, and I think we've demonstrated that this is possible.

I don't see any benefit to reflecting this in the file system thing, or what problem it solves. Java does this, but I've never been sure why. Certainly Python, for example, does not seem to have hit any problems with its approach.

Well I certainly agree that package naming & organisation in Julia is a lot better than in R (which is mostly a big mess. I mean names likes "cars" for regression?). I very much like the fact that you try to organize package around top level broad categories like optimization, stats,... & also that when multiple small third party packages appear that you try to reorganize them in bigger more coherent packages (this  the "framework" philosophy from Java). That's certainly way better than thousands of little packages only containing a few functions and dozens overlap or try to do the same thing like on CRAN. Putting aside the fact if that organization should be reflected on the filesystem, I think it might be useful to reflect some hierarchy in the package naming (Since Julia has submodules could they be used for this? of course I equating modules and packages again... maybe a bad thing). The reason I say this is that it allows for new user just to "browse" the package hierarchy and see what there directly from their IDE/editor rather than have to go hunt on the internet to find suited packages (like I do in R).

I hope you stick with Julia! Some of the discussion in this thread is really great already, fresh users willing to share their opinions really helps kick the tires on things we just get accustomed to.

That was exactly my intention. Once you have a few months under your belt you do not mind those little idosyncracies that much anymore because you've becomes accustomed to them. 

[Steven Sagaert]

for the exception example:

I have the following situation: repl -> file A -> file B

file A is just a script that call functions from file B which is a separate module. Basically A is the "main" program and B is the library where the real functionality is. Now when an error occurs in B then in the shell I get the line number from A where the exception was caught. That's useless. I want the linenumber (and full stacktrace) from B.

On Wednesday, August 13, 2014 5:49:34 PM UTC+2, Jeff Bezanson wrote:

Thanks for taking the time to write all of this, it's helpful.

I agree being able to write "export function foo(x) ... end" would be
good. Having to write things in 2 places is always annoying.

> * When an exception is thrown, a full stacktrace should be printed starting at the place (module name, line number, function name) where it is thrown instead of printing the line number where it is caught which is pretty much useless. This would be a big improvement for debugging!

This one confused me: we do print stack traces from where errors are
thrown, e.g.

julia> function foo(x)
        x += 1
        error("oops")
       end
foo (generic function with 1 method)

julia> foo(1)
ERROR: oops
 in foo at none:3

If this were showing where the error was caught it would be somewhere
in the REPL code. Maybe post an example of the behavior you saw?


On Wed, Aug 13, 2014 at 7:25 AM, Steven Sagaert

[Tim Holy]

I think that problem is limited to inlined functions, which obviously present
a special challenge. Can you please file an issue? (I note 9 open, and 53
closed, issues containing "line number". It's an ongoing challenge, but it's
all fixable.)

--Tim

[Tim Holy]

Trans, see "Pkg" (in base julia) and "Debug.jl".

--Tim

[Tim Holy]

On Thursday, August 14, 2014 02:22:26 AM Trans wrote:
> Is code re-usability being considered? Some questions:
>
> 1. How does one make a package that has more than one reusable module? For
> instance I have a Corpus program that can do word analysis, bigram analysis
> and letter analysis. As things stands importing Corpus makes all them
> available. How could I just import `Corpus.Bigrams` for example and not the
> rest?

Two examples: "using Gtk.ShortNames" or "using SIUnits.ShortUnits"

(meaning: see how it works for these particular packages)

>
> 2. If there is a submodule in another package, and I realize that it is
> almost exactly what I need for my program, how can I import it so that I
> can use it but with a modification (e.g. redefining a method or two)?

This is answered in the table:
http://docs.julialang.org/en/latest/manual/modules/#summary-of-module-usage

>
> 3. If I find a useful set of functions in another package (not encapsulated
> in a module) how to I import these into my program?

module MyWrappingModule

include(Pkg("UsefulPackage", "src", "usefulfile.jl"))

[Tobi]

We don't have it yet but when thinking about metapackages which pull in various packages at once it would really be handy to get the `exportall` functionality that is discussed in https://github.com/JuliaLang/julia/issues/1986

Making this through include tightly couples both packages and renaming the file will break `MyWrappingModule`

[Tim Holy]

I was assuming he was interested in extracting just a specific set of
functionality from a package and not the whole package. For example, with Grid
I organized it so that if you like the Counter iterator but don't need
interpolation, you can just load that one file.

--Tim

[Tobi]

Yes sure, include is useful. But when going through package boundaries it might still be of interest to `using` and `reexporting` only a submodule.
So if there would be a Grid.Counter submodule one might want to import and reexport all the methods.

[Trans]

On Thu, Aug 14, 2014 at 6:08 AM, Tim Holy <tim....@gmail.com> wrote:

I was assuming he was interested in extracting just a specific set of
functionality from a package and not the whole package. For example, with Grid
I organized it so that if you like the Counter iterator but don't need
interpolation, you can just load that one file.

Yes! That is what I mean. In all the examples given, the entire package is "put on the line" (so to speak). But I am talking about using components, aka little black boxes. So for example "using Gtk.ShortNames". I looked at the code and `ShortNames` is defined in `Gtk.jl`, so it's not an isolated component. (Also, that makes me wonder, if put `using Gtk.ShortNames` in my program what will Julia try to load file wise?)

I looked up Grid. Counter looks like a good example (of the module-less case). So is this what one would have to do to resuse it?

    include(Pkg("Grid", "src", "counter.jl"))

[Trans]

On Thu, Aug 14, 2014 at 5:07 AM, Tim Holy <tim....@gmail.com> wrote:

On Thursday, August 14, 2014 02:22:26 AM Trans wrote:
> Is code re-usability being considered? Some questions:
>
> 1. How does one make a package that has more than one reusable module? For
> instance I have a Corpus program that can do word analysis, bigram analysis
> and letter analysis. As things stands importing Corpus makes all them
> available. How could I just import `Corpus.Bigrams` for example and not the
> rest?

Two examples: "using Gtk.ShortNames" or "using SIUnits.ShortUnits"

(meaning: see how it works for these particular packages)

>
> 2. If there is a submodule in another package, and I realize that it is
> almost exactly what I need for my program, how can I import it so that I
> can use it but with a modification (e.g. redefining a method or two)?

This is answered in the table:
http://docs.julialang.org/en/latest/manual/modules/#summary-of-module-usage

Hmm... I think maybe I wasn't clear about the parts being "components". I realize I can just import `Gtk`, for example, and then use any submodules it might have. I was meaning, can we reuse a component independent of the rest. For example, in Ruby ANSI library it has a selection of components that can be used independent of the others. The reason they are in the same package is because they all depend on a common coloring module. But beyond that, one doesn't need to load the ANSI::Table class in order to use the ANSI::Progressbar class, for example.

 

> 3. If I find a useful set of functions in another package (not encapsulated
> in a module) how to I import these into my program?

module MyWrappingModule

include(Pkg("UsefulPackage", "src", "usefulfile.jl"))

That's good to know! Albeit a bit awkward syntax, at least it's doable. Thanks.

[Tim Holy]

On Thursday, August 14, 2014 08:31:09 AM Trans wrote:
> I looked up Grid. Counter looks like a good example (of the module-less
> case). So is this what one would have to do to resuse it?
>
> include(Pkg("Grid", "src", "counter.jl"))

Yep.
--Tim

[TR NS]

I thought of good example this morning of how `include` is a weak means of code decomposition.

 

    # t.jl

    x = 10

    include("r.jl")

    # r.jl

    println(x)

When we run `t.jl` it prints `10`. This is analogous to writing a function that can have have values injected into it without going though its interface.

[TR NS]

On Wednesday, August 13, 2014 8:13:06 PM UTC-4, TR NS wrote:

Unless... you get rid of `module` altogether, and just make files = modules. You can do that. Then `include` and `import` become one and the same thing. But then we have to live with the one file = one module limitation. 

Maybe that isn't such a bad idea actually. If files were modules and modules files, then there would be good encapsulation, it would be very clear how programs are organized, we wouldn't need to put `module Foo` and `end` at the top and bottoms of files (which was always weird b/c the style is not to indent), and all this might be doable with just a single "import/using" function. For encapsulation it would be done by giving an import a name, .e.g. `Foo = import foo`.

A related question, why do we have to `import` a function explicitly in order to extend it? Why isn't `using` sufficient?

[Mike Innes]

Again, though, I think you're mixing up what's possible with include and what it's intended for. I mean, sure, there's nothing stopping me from writing something like

const xs = Array(Int, 2)

function add2(x, y)

  xs[1], xs[2] = x, y

  return include("a.jl")

end

#a.jl

xs[1] + xs[2]

And yes, that would be a terrible, terrible way to write that function. But that's not really an argument against include because no one's saying you should use it this way.

There are just as many (if not more) ways to abuse Ruby's module system, because potential for misuse is implied by flexibility. The alternative is to lock everything down Java-style, but that very much goes against the Julia philosophy – it's much better to have the flexibility and power as long as you're careful with it.

[Tim Holy]

include will never go away. It's the lowest-level form of evaluating a file---
everything else (require, using, import) depends on it. You need to be able to
do this, otherwise you don't have a programming language :).

It's also the correct way to write a "script." If you have a job you want to
run on a cluster, you want one machine to be the "task master" instructing all
machines what code to load, what datasets to load, etc. If you want to put
that information into a file, then "include" is the right way to launch it.
Something like "require" will cause all machines to try to be the task master,
launching N copies of the same computation.

--Tim

[Stefan Karpinski]

On Wed, Aug 13, 2014 at 8:13 PM, Trans <tran...@gmail.com> wrote:

Include is a terrible way to load code as I tried to explain before. It totally destroys modular encapsulation. I ran into this problem immediately on my first Julia program when I tried to use submodules and hit issues where the submodules could not be found. In the end I had to make a choice between using the submodules or the file layout I wanted. It doesn't work well with documentation either (which is really just a symptom of the bad encapsulation). Consider if you have file a.jl made up of a bunch of functions. Then you include it in Foo and Bar modules in f.jl. How do we document the functions in a.jl? Will they be documents twice as functions of module Foo and Bar, or just once as the file a.jl, but then how clear is it how they are imported? Include is really anti-encapsulation --it isn't modular composition at all, it's just a cheap trick to copy and paste code automatically.

As Jeff said, include isn't intended for modularity. It just loads code. Whether that's "cheap" or not, that's what it does.

First, why does it work well in Ruby? In Ruby classes are namespaces. Defining a Ruby class is enough to start using it. Given the Foo class, you can construct a Foo object by doing `foo = Foo.new()`. Then you can use functionality of the Foo class by writing foo.bq(). The only thing that Ruby's require keyword needs to do is ensure that each path is only loaded a single time.

Why does this not work well in Julia? There aren't any classes. The functions you want to use on types aren't "inside" the objects they apply to – they are separate, external entities. When you require("foo"), you still don't have access to any of the things that are defined in Foo – you need to have bindings to those things also.

But there are modules, and functions are defined *in* modules, i.e. units of encapsulation.

Yes, but that doesn't address the key difference: to conveniently use a class, you just need the class object itself. To conveniently use a module, you typically need to not just load the module, but also make bindings to things in it. The appropriate language to compare with here is probably C++, not Ruby, Java or Python. Not coincidentally, C++ also has both import and using and they work very similarly to Julia's.

Thus, inevitably, the code looks like this:

require("Foo")

using Foo

This redundant code was rampant for a while. And it was really annoying. Why would I require "Foo" if I didn't want to use it? Likewise, if I write `using Foo` and there's nothing called Foo yet, it's not a wild stretch of the imagination that I want to require whatever provides Foo – typically a file named Foo.jl. Why can't this just do both for me at once: load the code if necessary *and* create the bindings I want. This is precisely what `using` now does. If you write `using Foo` and Foo does not exist, Julia looks in the usual places (Pkg.dir() and LOAD_PATH) for Foo.jl and loads it, with the expectation that it will define a module named Foo.

Because Bar might be defined in Foo.jl too. So you have to do this anyway to get Bar, and now things are getting confusing. It may seem annoying, but it is simple. Instead of `using Foo` looking for a file called `Foo.jl`, use `require 'Foo'` and Foo and Bar are both made available. If you really feel it necessary to load and use in one shot, `use Foo from "foo"` is very clear. So the annoyance is gone but you also don't have to worry about the other annoyance of file and module name correspondence.

With modules there's no good reason for a package not to live in a single module – the namespace is decoupled from units of functionality (types and functions). If Bar is separate enough to warrant its own module, then it belongs in its own package. So the best practice is that a package provides exactly one top-level modules. That's been working out quite well.

The ability to reopen classes in Ruby is arguably the most heavily abused feature in the language. "Monkey patching" – the colloquial term for opening a class up and altering it – is one of the biggest problems with the Ruby ecosystem. It's such a big problem that Ruby 2.0 attempted to add "refinements" – i.e. dynamically scoped monkey patching (it sounds much less, um, refined when you put it that way, right?). I say attempted, because the feature is so complex that 2.0 didn't end up fully implementing it.

I am an expert in this, and you completely misunderstand it. It is actually one of Ruby greatest strengths! It is what made Rails even possible. "Monkey patching" is a term that came from outside of the Ruby community and has been hawked by people who really didn't know what they are talking about. Refinements are just a way Matz devised that we could use to be more careful to not to step on each others toes. But many in the community actually don't think refinements are necessary b/c it's not something that is often a problem and when it is it is quickly resolved. Refinements haven't gotten much play, to say the least. But open classes are not going anywhere. I should point out too, that open classes are not just about "patching", they also make is simple to *add* functionality, including submodules, and thus conversely, simple to decompose functionality.

Since Ruby was my primary programming language for much of the past decade, I rather doubt that I'm completely misunderstanding this. Call it what you will – "duck punching", "hot fixing", whatever – it seems inconsistent to consider Ruby's rampant use of monkey patching just dandy while castigating `include` for "destroy[ing] modular encapsulation". Wat? The Rails community does seem to have made peace with monkey patching, but that's largely because activesupport, which does the most extensive modifications to the standard library, staked out the territory first and you either fall in line with that, or nobody will use your package. This may explain why the Rails community doesn't seem to care much about refinements while Matz and others outside of Rails feel the need for them – refinements would free them from the de facto dictatorship that Rails has over the Ruby ecosystem.

[Stefan Karpinski]

On Thu, Aug 14, 2014 at 11:22 AM, Stefan Karpinski <ste...@karpinski.org> wrote:

Yes, but that doesn't address the key difference: to conveniently use a class, you just need the class object itself. To conveniently use a module, you typically need to not just load the module, but also make bindings to things in it. The appropriate language to compare with here is probably C++, not Ruby, Java or Python. Not coincidentally, C++ also has both import and using and they work very similarly to Julia's.

Oops – I meant that C++ has both *include* and using, not import.

[TR NS]

On Thursday, August 14, 2014 10:18:20 AM UTC-4, Tim wrote:

include will never go away. It's the lowest-level form of evaluating a file---
everything else (require, using, import) depends on it. You need to be able to
do this, otherwise you don't have a programming language :).

It's also the correct way to write a "script." If you have a job you want to
run on a cluster, you want one machine to be the "task master" instructing all
machines what code to load, what datasets to load, etc. If you want to put
that information into a file, then "include" is the right way to launch it.
Something like "require" will cause all machines to try to be the task master,
launching N copies of the same computation.

Well, I played around with this: https://gist.github.com/trans/08883c340ed53de116f4

To me the original Gtk.jl is a convoluted mess. It has a module declaration, includes, usings, relative usings, imports, relative imports, imports with specific functions, relative imports with specific functions, and even an include to a relative parent path for dependencies which really boggles my mind. I am sure it all makes sense to the very smart people who created it, but for me as a Bear of Very Little Brain, well, it's difficult enough just to write my code. I am not want to wrestle with such a complexity of assembly too. Other systems work well (and even more capably) with simpler systems. Not sure why Julia can't.

Anyway, I won't bring it up any more. Thanks for the discussion.

 

[Stefan Karpinski]

 On Thu, Aug 14, 2014 at 3:44 AM, Steven Sagaert <steven....@gmail.com> wrote:

Well-discussed above, but this goes against a core Julia philosophy - "I like being treated like an adult." Documentation/naming is the solution to this. I've never found this policy to cause any problem in Julia code.

 

Well I like being treated as an adult too ;) but being in industry and not academia and having worked on large multiperson projects it's nice to be able to ensure that other people are not going to use your internal code because that might change in the future and then the client code breaks... Not everybody is going to behave nicely and some people will use whatever they can get their hands on if they can take shortcuts even if they are not supposed to. It's all about being able to refactor large codebases in the future.

Wanting to make refactoring safer is a very good cause, but I'm not sure that enforced encapsulation is really that much better than encapsulation by convention. When encapsulation – or any other compiler restriction – gets in the way, people tend to find ways around those restrictions. My experience is that while enforced encapsulation makes it harder to work around API design issues and other problems with encapsulation, it doesn't make those problems any less common. So the workarounds still, they're uglier and more fragile.

In general, I think it might be best if we stop using the "consenting adults" phrase as a counterargument. It comes across as a bit condescending – "Oh, you want restrictions on what you can do? Are you a child?" Maximal freedom isn't always the best – restrictions are sometimes a worthwhile tradeoff. Julia, like many other dynamic languages is not very keen on restrictions for their own sake – you'd better be getting something really good in exchange for those restrictions. This takes a bit of getting used to if you're coming from static languages. I do think that in the future, once the language is more settled, adding more compile-time checks will be a good idea, but also think it's a bit early for that.

[TR NS]

On Thursday, August 14, 2014 11:23:12 AM UTC-4, Stefan Karpinski wrote:

 

With modules there's no good reason for a package not to live in a single module – the namespace is decoupled from units of functionality (types and functions). If Bar is separate enough to warrant its own module, then it belongs in its own package. So the best practice is that a package provides exactly one top-level modules. That's been working out quite well.

So you are advocating for many tiny packages? Hence in the case of the Ruby ANSI library for instance, I should create a ANSIColor package, an ANSITable package, an ANSIProgressbar package, an ANSIString package, and so on?

 

Since Ruby was my primary programming language for much of the past decade, I rather doubt that I'm completely misunderstanding this. Call it what you will – "duck punching", "hot fixing", whatever – it seems inconsistent to consider Ruby's rampant use of monkey patching just dandy while castigating `include` for "destroy[ing] modular encapsulation". Wat? The Rails community does seem to have made peace with monkey patching, but that's largely because activesupport, which does the most extensive modifications to the standard library, staked out the territory first and you either fall in line with that, or nobody will use your package. This may explain why the Rails community doesn't seem to care much about refinements while Matz and others outside of Rails feel the need for them – refinements would free them from the de facto dictatorship that Rails has over the Ruby ecosystem.

 

They are generally called "core extensions". I've been programming Ruby since 2002 myself. And the fact that you call it rampant and contrast it to my castigating `include` I don't think is fitting on either point. First of all extensions are an amazingly useful and powerful feature. It is one of the main pillars that makes Ruby great. They make it possible to "extend" the functionality of preexisting objects. Extensions have one downside, that there could be name clash. And that is the only reason Ruby was given refinements. Matz could have easily given Ruby closed classes at any time if he thought they were bad, but if he had done so, so many gems would not have been possible, not just Rails. And I am pretty sure Ruby would have never have been popular. By contrast include isn't doing anything of the sort, it's just copy and paste. So I am castigating include because it is not *modular* encapsulation. You say that's not what include is intended for. But in package after package that is basically how it is being utilized among the "consenting adults". I am seeing lots of Julia code that is not modular in design. Instead it's just a big bucket of functions using include to pour functions in from pseudo-modular files. Maybe that's what you all want. I am not sure why Julia should even bother have separate modules at all then (at least then the file would be the module and vice versa). But the way it is seems like a bad design to me, and very much reminds me of PHP.

Anyway at this point I suppose we'll just have to agree to disagree. Thanks for taking the time to consider my concerns.

[TR NS]

On Thursday, August 14, 2014 1:40:12 PM UTC-4, TR NS wrote:

But the way it is seems like a bad design to me, and very much reminds me of PHP.

"But the current way ..."

 

[Steven Sagaert]

Hi,

Wow a lot of reactions. I guess  opened a can of worms ;) So far however not a single reply about my proposal for a catch that would be restricted to a specific type. So let's make it more concrete. How about the following:

try

<bunch of code>

catch e::<ErrorType1>

<code that handles ErrorType1>

catch e2::<errorType2>

<code that handles ErrorType2>

[ other catch clauses  ]

end

Error types not specified in the catch clauses would be thrown upwards.

[Tobi]

Hey Steven,

Jacob gave you the answer in the first reply you got. Please have a look at https://github.com/JuliaLang/julia/issues/7026

where the discussion takes place.

In short: There are some that really like catching specific exception (including me). There are others that think that exception hierarchies are not so good. In the end it might just need someone to implement the feature.

Cheers,

Tobi

[Kevin Squire]

On Thu, Aug 14, 2014 at 8:22 AM, Stefan Karpinski <ste...@karpinski.org> wrote:

With modules there's no good reason for a package not to live in a single module – the namespace is decoupled from units of functionality (types and functions). If Bar is separate enough to warrant its own module, then it belongs in its own package. So the best practice is that a package provides exactly one top-level modules. That's been working out quite well.

I agree with (or at least respect!) everything else you've said,... and I have a possible counter-example to this. 

As soon as some BinDeps issues are resolved, I'll be releasing VideoIO, which is a wrapper around ffmpeg/libav. That project itself consist of a number of base libraries (libavutil, libavcodec, libavformat) and independent libraries which depend on the core (libswscale, libavdevice, libswscale, libavresample, libavfilter).  Functionality-wise, many of these belong in separate modules, as they have nothing to do with one another, and it is frequent that one would only want the functionality of one or two of them.  But putting them in separate packages, while possible, would be really annoying, as the source for everything is distributed together (the only exception being Debian/Ubuntu, which builds separate packages).

Of course, to me, this is the exception that proves the rule: the libav/ffmpeg ecosystem is rather complex, and few projects should need to deal with things in this manner.

Cheers,

   Kevin

[samoconnor]

On Friday, August 15, 2014 4:39:43 AM UTC+10, Steven Sagaert wrote:

So far however not a single reply about my proposal for a catch that would be restricted to a specific type. 

I agree with you that there is a need to be abled to handle multiple exceptions in different ways; and not to catch all exceptions by default. However, catching by type seems limiting. There must be a way to have an exception filtering syntax that can filter on a field of the exception just as easily as the type. e.g. you might want to say "catch e if e.http_code == 404" or "catch e if e isa OutOfMemoryException".

I've written some macros to try this sort of thing out: https://groups.google.com/d/msg/julia-dev/DQFEhzZcjvQ/X0WulUYOWQsJ 

[Stefan Karpinski]

On Thu, Aug 14, 2014 at 1:40 PM, TR NS <tran...@gmail.com> wrote:

On Thursday, August 14, 2014 11:23:12 AM UTC-4, Stefan Karpinski wrote:

 

With modules there's no good reason for a package not to live in a single module – the namespace is decoupled from units of functionality (types and functions). If Bar is separate enough to warrant its own module, then it belongs in its own package. So the best practice is that a package provides exactly one top-level modules. That's been working out quite well.

So you are advocating for many tiny packages? Hence in the case of the Ruby ANSI library for instance, I should create a ANSIColor package, an ANSITable package, an ANSIProgressbar package, an ANSIString package, and so on?

No, you would have an ANSI package, which defines an ANSI module. The ANSI module would contain ANSI.Color, ANSI.Table, ANSI.ProgressBar, ANSI.String, etc. The package provides exactly one top-level module. My whole point is that with modules you can make the namespace unit as big as is appropriate instead of the namespace unit being tied to a function code unit like a class. Of course, then you have to deal with name binding, so it's a tradeoff. However, in Julia, there's no obvious functional unit like the class to tie the namespacing mechanism to – so we have modules.

 

They are generally called "core extensions". I've been programming Ruby since 2002 myself. And the fact that you call it rampant and contrast it to my castigating `include` I don't think is fitting on either point. First of all extensions are an amazingly useful and powerful feature. It is one of the main pillars that makes Ruby great. They make it possible to "extend" the functionality of preexisting objects. Extensions have one downside, that there could be name clash. And that is the only reason Ruby was given refinements. Matz could have easily given Ruby closed classes at any time if he thought they were bad, but if he had done so, so many gems would not have been possible, not just Rails. And I am pretty sure Ruby would have never have been popular.

This digression about monkey patching in Ruby has gotten pretty far off topic for a Julia mailing list, but I'll just add that a nice feature of multiple dispatch is that you can extend existing functions by dispatching on new types without risk of breaking anyone else's code and likewise, you can applying new functions to existing types without any fear of breakage.

 

By contrast include isn't doing anything of the sort, it's just copy and paste. So I am castigating include because it is not *modular* encapsulation. You say that's not what include is intended for. But in package after package that is basically how it is being utilized among the "consenting adults". I am seeing lots of Julia code that is not modular in design. Instead it's just a big bucket of functions using include to pour functions in from pseudo-modular files. Maybe that's what you all want. I am not sure why Julia should even bother have separate modules at all then (at least then the file would be the module and vice versa). But the way it is seems like a bad design to me, and very much reminds me of PHP.

I'm really confused about this whole issue. What's the specific problem with include? I haven't seen any problems with the way people are using it. True, it doesn't provide any modularity, but it's not meant to – that's what modules are for (hence the name). The way include is used in most packages just lets you split one large file up into multiple files. If you could give a specific example of "include behaving badly" it might help clarify what the concern is.

[Stefan Karpinski]

Can't they all be submodules of VideoIO? That seems like a reasonable structure.

[Stefan Karpinski]

I rather like that – way cleaner than catch, check, rethrow.

[Jared Kofron]

Personally, my gripe with include is that it's not meant to provide modularity, but it seems to be used that way.  It's common to see places where there is a module defined in a separate file, and then that file is included into its parent 

module definition.

While that's fine as a way to break large files up, in practice what it's doing is being used as a mechanism to "define"

submodules in the sense that they wind up in their own file.  The file containing the submodule then doesn't express

that hierarchy in any way, and for somebody who is reading the code but not aware of the structure of the entire codebase

this can be a little misleading.  For example, if I see 

module Sort

it doesn't suggest to me that actually that module is Base.Sort.

If there were another mechanism to declare a module belonged to some hierarchy, maybe it would help to make the

intent more clear.

[Jameson Nash]

the mechanism is that the code is contained inside of the Base module. when someone writes:

function f()

  x = do_something()

end

you don't expect them to need (or want) to annotate every line:

function f()

  f.x = f.globals.do_something()

end

because the scope is already clear from the context. similarly, when someone writes:

module X

  module Y

  end

end

It is unnecessary to force them to write:

module X

  module X.Y

  end

end

There's no requirement that the `Sort` module reside in `Base` – just convenience to have everything provided by the base Julia residing in the same place. By moving around some include files, that could easily be changed. (I even use this ability to move submodules around in some of my testing frameworks, mainly for doing some isolation/independence/integration testing.)

[Jared Kofron]

Right, exactly. The issue for me is context - if I read Base, it's clear that Sort is a sub module. If I read Sort, it's not. 

[Jeff Bezanson]

I guess what you are saying is that many uses of include() in practice
ought to be replaced by multiple true submodules. That could be true
in some cases, but I don't see why it's such a huge deal. One package
is generally a small enough unit that nobody worries about name
collisions *within* it. Furthermore, users don't like having to write
everything as X.Y.func(). If a package needs to export hundreds of
functions from a single module, so be it. Technical computing often
involves large vocabularies, so this can be convenient. In many cases
dividing the functions into groups would not be of any use.

> I am not sure why Julia should even bother have separate modules at all then

Isn't that a bit of an overreaction? Would it really make no
difference to have no namespacing mechanism *at all*?

[samoconnor]

Here is an example from my exception macros post:

@safe try

    r = s3_get(url)

catch e
    @trap e if typeof(e) == UVError
        println("ignoring socket error: " * string(e))
    end

    @trap e if e.code in {"NoSuchKey", "AccessDenied"}

        r = ""
    end
end

Note that a for this to work, the expression "if e.code in ..." has to fail silently if "e" has no field named "code",

so the trap macro expands to this:

   if try e.code in {"NoSuchKey", "AccessDenied"} end

        r = ""

        e = nothing
   end

[Jameson Nash]

it seems a little odd to call it safe exceptions, but then ignore all errors in the trap code. i understand that the interpretation is that it couldn't be handled, but it feels very wrong. also, in your example, you are also accessing a private field of an unknown exception type, which is strongly discouraged.

i think we should just strive towards having each error type be meaningful, and use rethrow when types aren't sufficient

adding type annotations with multiple catch blocks would be a good feature to have. i think nobody commented because this isn't controversial, just unimplemented. it would be relatively straightforward to rewrite it during lowering as a dispatch over a generic function

[samoconnor]

On Friday, August 15, 2014 10:42:16 AM UTC+10, Jameson wrote:

it seems a little odd to call it safe exceptions, but then ignore all errors in the trap code.

On the contrary, it is called @safe try... because the catch block automatically re-trhows all errors that are not explicitly trapped. 

 

i understand that the interpretation is that it couldn't be handled, but it feels very wrong.

Ignoring UVError is just an example. See here for more realistic examples: https://groups.google.com/d/msg/julia-dev/DQFEhzZcjvQ/X0WulUYOWQsJ

"NoSuchKey" and "AccessDenied" are not ignored. The result "r" is set to the empty string. This implements a semantic where fetching a non-existent URL returns "" by default.

(AccessDenied means the same thing as NoSuchKey in this context, because the system returns AccessDenied instead of NoSuchKey if the caller does not have permission to list all keys. This prevents information about what keys exist leaking to a non-privleaged caller.)

Again, this is deliberately very simple to keep the example short.

 

also, in your example, you are also accessing a private field of an unknown exception type, which is strongly discouraged.

See my description of the @trap macro expansion below. This is very deliberate. You don't want to have to write a whole bunch of guard logic before the field access, that would be way too verbose.

i think we should just strive towards having each error type be meaningful, and use rethrow when types aren't sufficient

My feeling is that having a sub-type for every error is impractical for real world complex systems. If you are interfacing to a library, or a cloud API, that has hundreds of potential error conditions, you'd have to do something like dynamically create new exception types based on a XML/JSON message from the API. It makes more sense to have a FooAPIException class with codes for http_error_code, api_error_code, api_error_string, api_human_readable_message etc...

Filtering exceptions only by type seems like the kind of think you get stuck with in a statically typed system. A dynamic system should be more flexible.

[Jameson Nash]

On Thu, Aug 14, 2014 at 9:05 PM, samoconnor <samoc...@mac.com> wrote:

On Friday, August 15, 2014 10:42:16 AM UTC+10, Jameson wrote:

it seems a little odd to call it safe exceptions, but then ignore all errors in the trap code.

On the contrary, it is called @safe try... because the catch block automatically re-trhows all errors that are not explicitly trapped. 

 

i understand that the interpretation is that it couldn't be handled, but it feels very wrong.

Ignoring UVError is just an example. See here for more realistic examples: https://groups.google.com/d/msg/julia-dev/DQFEhzZcjvQ/X0WulUYOWQsJ

"NoSuchKey" and "AccessDenied" are not ignored. The result "r" is set to the empty string. This implements a semantic where fetching a non-existent URL returns "" by default.

(AccessDenied means the same thing as NoSuchKey in this context, because the system returns AccessDenied instead of NoSuchKey if the caller does not have permission to list all keys. This prevents information about what keys exist leaking to a non-privleaged caller.)

Again, this is deliberately very simple to keep the example short.

errors typically should occur when either the user forgot to check for a precondition (such as whether the key exists) or an unexpected event (such as an unexpected disconnect). other conditions should typically be handled by status codes in the normal flow of logic

 

 

also, in your example, you are also accessing a private field of an unknown exception type, which is strongly discouraged.

See my description of the @trap macro expansion below. This is very deliberate. You don't want to have to write a whole bunch of guard logic before the field access, that would be way too verbose.

verbose or not, you need to check your preconditions

 

i think we should just strive towards having each error type be meaningful, and use rethrow when types aren't sufficient

My feeling is that having a sub-type for every error is impractical for real world complex systems. If you are interfacing to a library, or a cloud API, that has hundreds of potential error conditions, you'd have to do something like dynamically create new exception types based on a XML/JSON message from the API. It makes more sense to have a FooAPIException class with codes for http_error_code, api_error_code, api_error_string, api_human_readable_message etc...

these should typically be normal status conditions, not errors. in fact, julia's current lack of filter exceptions partially stems from the notion that a program receiving an exception should be more focused on how to continue than what error occurred – and on avoiding errors in general. (that's not particularly well worded, but I couldn't think of a better way to phrase it)

 

Filtering exceptions only by type seems like the kind of think you get stuck with in a statically typed system. A dynamic system should be more flexible.

Microsoft SEH (Visual C++) has support for filtering exceptions by value. Python only provides filters by type. I think this is more closely related to language design choices (such as not providing value-based dispatch in Julia, only type-based dispatch), than a fundamental issue.

[Stefan Karpinski]

On Thu, Aug 14, 2014 at 8:41 PM, Jameson Nash <vtj...@gmail.com> wrote:

adding type annotations with multiple catch blocks would be a good feature to have. i think nobody commented because this isn't controversial, just unimplemented. it would be relatively straightforward to rewrite it during lowering as a dispatch over a generic function

I am against adding this feature because I think it's broken and I want that syntax to mean something subtly but importantly different: #7026.

[samoconnor]

On Friday, August 15, 2014 11:43:45 AM UTC+10, Jameson wrote:

errors typically should occur when either the user forgot to check for a precondition (such as whether the key exists) or an unexpected event (such as an unexpected disconnect). other conditions should typically be handled by status codes in the normal flow of logic

...

these should typically be normal status conditions, not errors. in fact, julia's current lack of filter exceptions partially stems from the notion that a program receiving an exception should be more focused on how to continue than what error occurred – and on avoiding errors in general. (that's not particularly well worded, but I couldn't think of a better way to phrase it)

I have a lot of time for the idea of strict documentation and checking of preconditions. My first "real" software engineering job was designing library APIs in Eiffel for Bertrand Meyer's (grand-daddy of design-by-contract) in Santa Barbara.

In a synchronous, deterministic system your basic idea that exception mechanism shouldn't be used much is pretty much right. "Technical computing" is not my core expertise; but I imagine that more often than not, the code that people have been writing in MATLAB, R etc, can be written in Julia with very limited use of try/catch.

My interest in Julia stems from building biometric data processing systems that deal with large amounts of data, interface with multiple cloud-based infrastructure systems and implement a bunch of workflow and business logic in addition to doing low-level signal processing and analysis of the actual biometric data. We currently build theses systems using: a "fast" language, a "scientist friendly" language and a "glue" language. e.g. (C/C++, R/MATLAB/Python/Tcl and Tcl/bash/Python/Tcl). Julia is interesting to me because it has the potential to be fast and scientist friendly while also being high-level enough for "glue" scripting tasks. It would simplify my projects to standardise on a single implementation language.

I can see that the Julia community has plenty of very experienced analytics / technical computing people and is shaping up to be a great tool for that audience. I think Julia also has the potential to be a great systems programming language and a great business-logic  and scripting language. However, it has rough edges in these areas at this point in time.

In real-world asynchronous, eventually-consistent, distributed systems the idea of callers checking pre-conditions does not really apply. When hundreds of thousands of unreliable simultaneous actors are connected by an unreliable network it is meaningless to "check a precondition" first because the world state of the world will have moved on by the time the call is made. The only way to know if a certain request will succeed is to try it. However, if an convenient and expressive exception handling mechanism is available, the normal flow of logic does not have to care about exceptional conditions and can be written in a clear functional and/or procedural style that makes it easy to read. Exceptions can be dealt with at the appropriate level of the application stack and operations can be re-tried, re-schedueled etc.

It is important in this context to have a very clear and expressive syntax for writing exception filters and handling code. In a code review you want 1) the normal-case code to be clearly readable so you can review it for compliance with the spec without getting tied up in exception handling; 2) to have no doubt that the exception filtering has appropriate granularity and no un-intended catching of exceptions that should really go up the stack; 3) clarity about what recovery code runs per exception filter.

Passing error codes explicitly all the way up the stack fails 1). The current Julia try/catch can lead to failure of 2) and 3) because you end up with a if/elseif/else structure that you have to read quite carefully to be sure that the rethrow() at the bottom is reached in the appropriate cases. I think there is scope to make this much better with very little change and to end up with something that is better than what other languages currently offer.

A final thought: It is the job of the API designer, as far as possible, to limit the possibility of an API call throwing an exception (or at least to limit the number of exception cases). "If you meet my preconditions, I will try very hard to not throw exceptions". If the language provides great tools for effective exception handling, the API designer is more likely to handle exceptions in a sensible way within the API and only throw exceptions up the stack when there is really no alternative. The current Julia try/catch seems to discourage fine-grained exception handling. This seems backwards. I have been very annoyed many times because some Java library or other threw a very non-specific exception 30 levels down a call stack inside some code I wan't absolutely nothing to do with. I believe that this happens to to a large degree because java's clunky class-based catch filtering mechanism makes it awkward to do fine-grained exception handling as a matter of course.

[Trans]

On Thu, Aug 14, 2014 at 8:23 PM, Jeff Bezanson <jeff.b...@gmail.com> wrote:

I guess what you are saying is that many uses of include() in practice
ought to be replaced by multiple true submodules. That could be true
in some cases, but I don't see why it's such a huge deal. One package
is generally a small enough unit that nobody worries about name
collisions *within* it. Furthermore, users don't like having to write
everything as X.Y.func(). If a package needs to export hundreds of
functions from a single module, so be it. Technical computing often
involves large vocabularies, so this can be convenient. In many cases
dividing the functions into groups would not be of any use.

I'll try to explain. The problem with include is that provides a mechanism of code reuse that is not modular. (Before I used the word "encapsulation", but modular is a more precise word for what I meant.) So for instance someone can do something like this:

  module A

    include("foo.jl")

  end

  module B

    include("foo.jl")

  end 

This is bad design. It's not the end-of-the-world bad, but it is creating duplication of code where none need exist.

Another example, is one I mentioned before.

    # foo.jl

    x = 10

    include("bar.jl")

    # bar.jl

    println(x)

This prints 10. So we have code leaking across the SOC (separation of concern). 

Another point is the simple fact that includes get scattered all over ones code, partly because the order of includes becomes significant.

None of these are good, and run contrary to modular programming. Which is a very important thing! Now, Stefan argues we are all "consenting adults" and we should just not do those things. I don't agree and think the system should be designed to promote good modular design. And in fact, if it were designed as such it would actually become much easier to use. Stefan worries that such "limitations" would somehow prevent us from doing some things (whatever they might be he doesn't say).

 

> I am not sure why Julia should even bother have separate modules at all then

Isn't that a bit of an overreaction? Would it really make no
difference to have no namespacing mechanism *at all*?

That's not what I mean. I mean the file itself *becomes* the namespace. Thus making the file the unit of modularization. The downside is that two modules can't be defined in the same file. But if modules are closed anyway (i.e. different portions of a module can't be defined in multiple files), that probably doesn't really matter.

Another point about Julia's current system is that it convolutes the two. If not previously defined, `using Foo` will try to load the file `foo.jl` fully expecting `module Foo` to be defined inside. So it sort of is, but it isn't.

[Tobias Verbeke]

On Wed, Aug 13, 2014 at 4:41 PM, Jacob Quinn <quinn....@gmail.com> wrote:

Thoughts inline:

On Wed, Aug 13, 2014 at 7:25 AM, Steven Sagaert <steven....@gmail.com> wrote:

Hi,

I've been programming a recommender system in Julia for a couple of weeks and overall I like much of Julia and the choices that have been made but I've also found some things that annoy me and that I'd like to see changed. I figure that at 0.2 it's still a very young language with a small user base & ecosystem so I might actually have a chance of some of my requests actually make it into a future release.

* naming convention of the standard lib. For types Julia uses camelcase starting with uppercase (like Java for Classes) but for functions it's all lowercase with sometimes an underscore for separation. Having done a lot of Java programming this feels a bit weird to me. Why mix two different systems (camelcase & underscores?) Why not go for Java's system: camelcase (shorter than underscores) : function would start with lowercase (like Java methods) and types start with uppercase. I could live with all lowercase for functions but then I would like to see a consistent use of underscore for word separation. In the standard lib this is very inconsistent (many functions names have no separation where it should: e.g. haskey) and I find this very annoying since you never really know how a function name is written unless you look it up. In the absence of any code completion in the current development tool this little thing quickly becomes a source of irritation.

The trend has actually been moving swiftly away from any method names with underscores, (many have been removed or renamed). If there are still ones (in 0.3, which is about to be released officially any day) that you find irksome, please file an issue and I imagine there will be a good effort to rename or remove it. Apart from that, I personally really like the convention of camel case for types and all lowercase for methods.

I respect anyone's liking, but cannot imagine that all lowercase for composite method names is easier to parse for the human brain than conventions where there is an explicit separation between components of composite names (using camelCase or underscores). Readability has (for me at least) been the first criterion when writing or assessing coding conventions. Still searching for an experimental psychology paper on the matter..

Best,
Tobias

[Kevin Squire]

Whoops!  That's embarrassing. I think the first time I read your post, I understood the exact opposite of what you said.  

Those modules do all exist as submodules of VideoIO. 

Cheers!

    Kevin 

[Stefan Karpinski]

On Fri, Aug 15, 2014 at 12:31 AM, Trans <tran...@gmail.com> wrote:

None of these are good, and run contrary to modular programming. Which is a very important thing! Now, Stefan argues we are all "consenting adults" and we should just not do those things. I don't agree and think the system should be designed to promote good modular design. And in fact, if it were designed as such it would actually become much easier to use. Stefan worries that such "limitations" would somehow prevent us from doing some things (whatever they might be he doesn't say).

We were so close to having a productive conversation here and then this paragraph has to go and ruin it. Perhaps there is a native language issue here – this paragraph comes off as rather rude. At no point in this conversation or any other have I used "consenting adults" as an argument for anything, only ever as a description of more liberal approaches. In fact, the only time I used this phrase at all in this thread, I wrote:

In general, I think it might be best if we stop using the "consenting adults" phrase as a counterargument. It comes across as a bit condescending – "Oh, you want restrictions on what you can do? Are you a child?" Maximal freedom isn't always the best – restrictions are sometimes a worthwhile tradeoff.

Your rhetorical caricature of my position is inaccurate, a bit offensive, and I suspect that it undermines your position more than it helps it.

Steven, I'm sorry that your thread has gotten derailed. You brought up a lot of good points and much of the ensuing discussion has been very positive. If there are issues you still want to discuss, perhaps it would be best to start separate threads about them.

[Jeff Bezanson]

I think I understand your argument a bit better now.

But I am hearing: modularity is good, so ban anything that is not
modular. Yes modularity is good and important, but I also think junk
food is ok once in a while.

I agree there is something fishy about the chain of events in `using
Foo` - `foo.jl` - `module Foo`. It seems like something needs to be
simplified there. I would like to preserve the ability to split large
files without hassles, though. Maybe that can be done with directories
somehow.

We wanted to de-couple program structure from filesystems, but perhaps
that is not realistic (the ultimate negative example is old matlab,
where you need to create a directory to define a class).

[Trans]

On Fri, Aug 15, 2014 at 1:41 AM, Stefan Karpinski <ste...@karpinski.org> wrote:

On Fri, Aug 15, 2014 at 12:31 AM, Trans <tran...@gmail.com> wrote:

None of these are good, and run contrary to modular programming. Which is a very important thing! Now, Stefan argues we are all "consenting adults" and we should just not do those things. I don't agree and think the system should be designed to promote good modular design. And in fact, if it were designed as such it would actually become much easier to use. Stefan worries that such "limitations" would somehow prevent us from doing some things (whatever they might be he doesn't say).

We were so close to having a productive conversation here and then this paragraph has to go and ruin it. Perhaps there is a native language issue here – this paragraph comes off as rather rude. At no point in this conversation or any other have I used "consenting adults" as an argument for anything, only ever as a description of more liberal approaches. In fact, the only time I used this phrase at all in this thread, I wrote:

In general, I think it might be best if we stop using the "consenting adults" phrase as a counterargument. It comes across as a bit condescending – "Oh, you want restrictions on what you can do? Are you a child?" Maximal freedom isn't always the best – restrictions are sometimes a worthwhile tradeoff.

Your rhetorical caricature of my position is inaccurate, a bit offensive, and I suspect that it undermines your position more than it helps it.

Sorry if that came across as rude. I wasn't intending that at all. I was just stating what I thought your position was. As you point out, you said yourself earlier that the phrase "consenting adults" comes off a condescending. So I guess that's part of why it seems that way. I just meant factually that you think it is up to the programmer to "do the right thing". That's all. 

Steven, I'm sorry that your thread has gotten derailed. You brought up a lot of good points and much of the ensuing discussion has been very positive. If there are issues you still want to discuss, perhaps it would be best to start separate threads about them.

I think it is unfair to say the conversation has been derailed b/c you took offense at something I said. As I said, it was not my intent. I apologize for not trying to be more sensitive in my wording. I still think there are salient points made in my response, and it would be a shame to loose them over an emotive misintention.

[Tobi]

While we are at this point: Saying "To me the original Gtk.jl is a convoluted mess." is a pretty strong statement. I agree that this example reveals that the module system might need some more convenient ways as Jeff indicates in his last post. But seriously: Jameson (the main author of Gtk.jl) is not known to write "convoluted mess" but he is one of the smartest hackers here around.

[Mike Innes]

I like this approach, but as a minor point: I recommend against using implicit catch-all try blocks. If you miss a keystroke in that block or anything it calls – or indeed if something it calls throws an important error – it will silently be ignored, and you're going to lose a few hours figuring out why no exceptions are being caught.

[Steven Sagaert]

Hi Stefan,

No offense taken. I'm actually a bit surprised by the many posts. I didn't expect that many. I find the discussion very positive and constructive. I must say kudos to the Julia community for being open to positive critique and willing to question choices made and maybe change some things in the future. I've had other experiences, e.g. with software companies.

[Tobi]

Steven,

I think your feedback is very valuable. As you have raise several independent points it is a good to move the discussion to the specific issues on Github.
I have opened https://github.com/JuliaLang/julia/issues/8005 by the way.

Regarding the exceptions you might have observed that there is no real consensus how to move forward. Would be great if you would join https://github.com/JuliaLang/julia/issues/7026 and give feedback on the different ideas esspecially if you have some experience with exceptions.

Cheers,

Tobi

[Trans]

On Fri, Aug 15, 2014 at 2:42 AM, Tobi <tobias...@googlemail.com> wrote:

While we are at this point: Saying "To me the original Gtk.jl is a convoluted mess." is a pretty strong statement. I agree that this example reveals that the module system might need some more convenient ways as Jeff indicates in his last post. But seriously: Jameson (the main author of Gtk.jl) is not known to write "convoluted mess" but he is one of the smartest hackers here around.

I was not knocking his coding ability. I was talking about the design of include/module system that basically causes it to be a mess beyond anyone's coding control.

 

I'm sorry, but if people are going to be this sensitive and nick-pick every little phrase for possible offense, than I don't think it is even possible to have a reasonable conversation. 

Now I am sorry that I even tried to help make Julia a better language.

[Milan Bouchet-Valat]

This list is among the most friendly that I know of in the free software world. You can't expect people not to react to your qualifying their own work as "convoluted mess"; at least they have remained very polite and constructive. I know places where the discussion would have been much more heated.

Now I am sorry that I even tried to help make Julia a better language.

You got a very detailed discussion, I'm not sure what more you may expect.

Regards

[Jameson Nash]

No offense taken. I agree that Gtk.jl has a lot of exports and plays a few tricks to make them flexible. I'm not sure how that relates to 'include' being very simplistic-minded: these features are generally intended for user convenience, and are not essential to the library (some even evolved from trying to be partly API compatible with Tk.jl)

As mentioned previously, requests for "please add a feature that prevents from doing X" tend to be poorly received. X may be something that is generally accepted to be a bad idea, but that alone is not a compelling reason to develop and maintain such a system, unless it also clearly benefits the programmer. Often, that criterion is simply what someone is willing to put the time into making a pull request. 

As Jeff pointed out, the relationship between using-require-filename-module might be improved. But I don't know that we have a concrete proposal that improves that, without going to a folder-file-is-a-module approach -- or perhaps that's necessary anyways for some other reason? -- Putting together a gist showing and describing and alternative system tends to be a good way to analyze how an alternative system would work on a toy/real example. 

[Tim Holy]

I think for many of us, the main problems were outlined by Jameson: basically,
come up with a concrete better proposal. And stop arguing against `include`
:-). I've already explained that it _can't_ go away because it's the primitive
operation on which all file evaluation is built. How would C exist without
`#include <stdlib.h>`? Heck, you can't even _build_ julia itself without the
ability to load code from different files. You're not going to get anywhere if
you keep plugging away at what seems like an impossible request.

Perhaps you're arguing that `include` should exist but never be used in
packages---it should be wrapped in something else that does something
different. If so, please spell out a better system. Filing an issue might be
the best approach, that way your proposal will not be lost to memories of a
mailing list flamewar.

--Tim

[Stefan Karpinski]

One thing that I think I've distilled from this conversation is that it may be good to have a mechanism besides include for loading submodules. This was actually something that we discussed when the module system was originally designed but decided it was too speculative until more experience was had. We've had that experience now and it does seem to be a thing. Consider this situation:

# Foo.jl

module Foo

  include("Bar.jl")

  include("Baz.jl")

end

# Bar.jl

module Bar

  # barfy stuff

end

# Baz.jl

module Baz

  # bazish stuff

end

Clearly include works here, but it would be nicer to write something like this:

# Foo.jl

module Foo

  import .Bar

  import .Baz

end

We could automatically load relative imports relative (not a typo) to the current source file the way we automatically load absolute imports from LOAD_PATH. An open question is whether the file structure should be Foo/Bar.jl and Foo/Baz.jl or just Bar.jl and Baz.jl.

[Tobi]

Stefan,

I think this looks like a very good proposal. Thanks! 

Lets please add an Github issue so that this does not get lost.

I would say that the structure is either:

Foo/Foo.jl

Foo/Bar.jl

Foo/Baz.jl

which would work if Bar and Baz are small. Or

Foo/Foo.jl

Foo/Bar/Bar.jl

Foo/Baz/Baz.jl

if both are large and consist of more modules/files.

Cheers,

Tobi

[Stefan Karpinski]

The shorter version is better. You can always do this:

Foo/Bar.jl

Foo/Bar/...

If Bar has supporting files. This is actually exactly the structure of Pkg.

[Jeff Bezanson]

I believe these issues cover it:
#4600
#8000

[Tobi]

Yes thanks Jeff. #4600 is it in particular. Do module issue always have a number that is a multiple of 100? :-)

[Jason Merrill]

On Friday, August 15, 2014 8:52:07 AM UTC-7, Stefan Karpinski wrote:

One thing that I think I've distilled from this conversation is that it may be good to have a mechanism besides include for loading submodules. This was actually something that we discussed when the module system was originally designed but decided it was too speculative until more experience was had. We've had that experience now and it does seem to be a thing. Consider this situation:

# Foo.jl

module Foo

  include("Bar.jl")

  include("Baz.jl")

end

# Bar.jl

module Bar

  # barfy stuff

end

# Baz.jl

module Baz

  # bazish stuff

end

Clearly include works here, but it would be nicer to write something like this:

# Foo.jl

module Foo

  import .Bar

  import .Baz

end

We could automatically load relative imports relative (not a typo) to the current source file the way we automatically load absolute imports from LOAD_PATH. An open question is whether the file structure should be Foo/Bar.jl and Foo/Baz.jl or just Bar.jl and Baz.jl.

I agree that this would be an improvement over "include".

To me, "include" has some of the bad smell of "with" from some other languages, i.e. an operator that takes an object and dumps all of its properties into the local namespace. We don't like "with" because it makes it impossible to know statically how a piece of code will behave at runtime.

Now "include" isn't as bad as "with" in terms of language semantics, but for a person looking at a program and trying to understand what it does, it causes similar problems. Now you have to go look at another file to know what any identifier means. And if there are several includes, you don't even know which one to go look in. And when you're looking in that other file, you have to think about the context it's being included into to understand it.

Suppose I have

    include("file1")
    include("file2")

    a + b

Now I have to go look in file1 and file2 to know what kind of things "a" and "b" are. And when I'm reading file2, I actually have to know what happened in file1 because its identifiers are now available in the scope I'm running in. Even though in file2, there is no local indication that this would matter.    

Bare "using" to import all the symbols from a module isn't actually much better in this regard. I've come to really like the node module system, where you do,

    var AnyThingIWant = require('somemodule')

Here, you know that the effects of 'somemodule' are scoped to an identifier of your choosing. If you haven't tried this way of doing things, it may sound restrictive, but I will say that after a couple years of use, it feels great to me.

And not great in a "I work in a huge corporation and don't trust my colleagues, many of whom I don't know, to not make ridiculously poor decisions" kind of way, but great in a "I enjoy this way of working for my personal projects because it helps me pick them up again later" kind of way.

This more conservative approach to namespacing does make it harder to factor a large file into smaller files, because you can't just cut and paste code into a different file and then include it and be done. You now have to either unpack the new file's identifiers explicitly at the top, or you have to add a "AnythingIWant." to the front of the places that you're using the new file's identifiers.

But with a little practice, this becomes a good thing. The point of splitting a large file into small files isn't just that the large file had too many lines. The point is to decompose things to reduce complexity. If you can't reduce the interaction between two blobs of your source code to a reasonably simple interface, then there isn't much to be gained by splitting those two blobs into separate files anyway.