warnings including module multiple times

[Kuba Roth]

Hi there,
I'd like to load a custom module from a custom location. In order to do this via import/using I need to first put include(moduleFile.jl) statement.
Every time I have include() in more then one file I get: Warning: replacing module.

The custom module is in the same location as my main file and the directory where files are located are not in the LOAD_PATH. 
If I omit "include" this leads me to a "not found" error.

Is there any kind of include guards I need to add to silent this warning?
Or perhaps there is a better way to work load modules from custom locations? Can you please provide some guidelines.

Thanks,
kuba

[Stefan Karpinski]

You shouldn't be using include here – the way to do this is to make sure that moduleFile.jl is in your LOAD_PATH and then do `using moduleFile` or `import moduleFile`.

[Ben Strowbridge]

I wonder if there is also system-specific issues going on here. Under Windows, I need to use the “include()” function to register any updates I make to the code in modules outside of the notebook even though the source file is in a folder within the LOAD_PATH. If I just use the “using myModule” syntax, the module loads but then the notebook is oblivious to any changes I make outside that environment even if I re-execute the “using” command. This issue would go away if Autoreload worked for me under Windows; it works fine under Yosemite but doesn't release the write permission on the source code file under Windows 7 (with Julia 0.3.3).

There are two other unexpected things, at least to me, regarding “include” and “using”. First, to access exported functions you seem to have to execute two “using” statements if your module file name is not the same as the actual module name. I had assumed that executing a “using MyModuleFileName” command would bring in all the exported functions since it already brought in “MyModule” in a way that enabled “MyModule.foo()” to work. Executing an “include” with the file name followed by a “using” with the module name also works. This seems like another case where we have to use the “include” statement (or two “using” statements) even though everything is within the LOAD_PATH (at least under Windows.)

The other issue is that functions that are exported seem to be frozen—I cannot find a way to update them in the notebook after I make changes to the module code in an editor. Executing another “include()” statement doesn't seem to work even if the module name and module file name are the same. I am running into these issues since I am developing new functions within a notebook and then, in another notebook session, testing the function after moving it to a module. In most cases, I need to tweak the function after I have placed it in a module.

[Jason Merrill]

On Saturday, December 27, 2014 5:15:38 PM UTC-5, Stefan Karpinski wrote:

You shouldn't be using include here – the way to do this is to make sure that moduleFile.jl is in your LOAD_PATH and then do `using moduleFile` or `import moduleFile`.

I've complained before that "include" in user code is an anti-pattern, and I guess I'm going to complain about it again.

Are there examples of other languages that people admire where the equivalent of "include" is regularly used in user code?

The main place that I think I've seen people use something like it is PHP, and even there, I think "require" is used more frequently in good code.

Languages I've used where include-like functionality is very rarely used include Python (include->execfile), Java, and node-style Javascript (include->exec(fs.readFileSync("filename"))).

I don't think there's much to learn from Matlab here, and Mathematica's culture of user created libraries seems weaker than in many other environments.

Do people frequently use source() in R?

[Stefan Karpinski]

I'm not sure where you get the idea that anyone "admires" using include. I specifically just recommended not using it.

[Milan Bouchet-Valat]

Definitely not in packages, since all .R files in the R/ directory are
concatenated together when building the package. Probably more
frequently when writing large scripts, if you don't want to include
everything in one file. Some people appear to prefer that solution over
creating packages, but that shouldn't be considered as recommended:
http://stackoverflow.com/questions/1266279/how-to-organize-large-r-programs


Regards

[Jason Merrill]

I didn't mean to say that anyone "admires" using include, but empirically, it is used a lot as a code organization tool in Julia (more below).

I was asking about if there was precedent in a different language that shows that "no, the situation is actually totally fine; even though you can misuse include nobody does."

My beef with include is that it is a deeply unstructured way of organizing code, in the sense that it can dump a lot of identifiers into any scope, and it makes no guarantees about how often the top level code in an included file will be run. I brought it up again here because I think OP's question is a good example of that second property biting someone.

Now the empiricism: 9 out the 10 most starred Julia packages use "include" as a code organization tool:

1. https://github.com/JuliaLang/IJulia.jl/search?utf8=%E2%9C%93&q=include
2. https://github.com/dcjones/Gadfly.jl/search?utf8=%E2%9C%93&q=include
3. https://github.com/JuliaStats/DataFrames.jl/search?utf8=%E2%9C%93&q=include
4. https://github.com/stevengj/PyCall.jl/search?utf8=%E2%9C%93&q=include
5. https://github.com/JuliaOpt/JuMP.jl/search?utf8=%E2%9C%93&q=include
6. https://github.com/pluskid/Mocha.jl/search?utf8=%E2%9C%93&q=include
7. https://github.com/JuliaWeb/Morsel.jl/search?utf8=%E2%9C%93&q=include
8. https://github.com/JuliaOpt/Optim.jl/search?utf8=%E2%9C%93&q=include
9. https://github.com/forio/GeneticAlgorithms.jl/search?utf8=%E2%9C%93&q=include (the only one that doesn't use include)
10. https://github.com/nolta/Winston.jl/search?utf8=%E2%9C%93&q=include

Julia core also uses include, e.g. in the markdown parser:

https://github.com/JuliaLang/julia/search?l=julia&q=include&utf8=%E2%9C%93

Now the popular packages tend use include in a way that's fairly harmless--i.e. to allow a lot of different symbols and functions to be exported from a single module, without writing all the code in a single giant file. But given that the good uses of include are pretty restricted, I think we'd be better off if the default tool for doing this kind of thing was more restrictive.

If you want to use include poorly, you can use it very poorly. This last part is entirely for amusement--it's a total strawman--but it just occurred to me that you can use include to simulate labeled GOTO:

# factorial.jl
n = parseint(ARGS[1])
accum = n
include("helper-a.jl")

# helper-a.jl
n = n - 1
if n <= 0
  print(accum)
else
  include("helper-b.jl")
end

# helper-b.jl
accum *= n
include("helper-a.jl")

Now I can run

$ julia factorial.jl 5
120

[Mauro]

> Now the popular packages tend use include in a way that's fairly
> harmless--i.e. to allow a lot of different symbols and functions to be
> exported from a single module, without writing all the code in a single
> giant file. But given that the good uses of include are pretty restricted,
> I think we'd be better off if the default tool for doing this kind of thing
> was more restrictive.

Any thoughts on how a restricted include should work?

I like your goto-include example, nice abuse!

[Jason Merrill]

On Sunday, December 28, 2014 1:34:50 PM UTC-5, Mauro wrote:

> Now the popular packages tend use include in a way that's fairly
> harmless--i.e. to allow a lot of different symbols and functions to be
> exported from a single module, without writing all the code in a single
> giant file. But given that the good uses of include are pretty restricted,
> I think we'd be better off if the default tool for doing this kind of thing
> was more restrictive.

Any thoughts on how a restricted include should work?

Sure: we already have modules. The good thing about them is that top level code only runs once, and it executes in a separate scope from the code where the module is imported.

Modules are great, and you should use one any time you want to run code that is in one file from another file.

I can only speculate about the reason that people still use include, even though Julia has nice modules. There's a bit about "mixins" in the manual (http://julia.readthedocs.org/en/latest/manual/modules/#modules-and-files), but I haven't ever seen include used that way in the wild. The common case appears to be that people sometimes want symbols in a "child file" to show up in a "parent module". If you make the child file into a module, export some of its symbols, and then say "importall Child" in the parent module, you get the exported symbols of the child as non-exported symbols of the parent. But so far there isn't an easy way to

1) make all of the non-exported symbols of a child module into non-exported symbols of a parent module, nor
2) "reexport" all of the exported symbols of a child module from a parent module.

1) feels kind of suspect to me because I think it's better to be explicit about these things. But I'd rather see this behavior enabled if it would convince people to stop using include, which is much worse.

I'm not sure about names, but 1) could be something like

importunexported Child

which would be like using except that it would load all the symbols in Child, not just the ones that were explicitly exported. Even though this is not perfectly explicit, it maintains the advantages that the top level code in Child is known to run only once, and that it runs in a separate scope from Parent so that symbols already in Parent can't affect the behavior of top level code in Child.

2) could be something like

reexport Child

[Mike Innes]

I can only speculate about the reason that people still use include, even though Julia has nice modules.

Well, I for one think that

1) Modules containing a large number of functions are OK (and this is very common in more functional languages)

2) If you're going to have (1), you want to split that module across multiple files.

Languages which embrace (1) but conflate modules and files tend to end up looking like this. Personally, I prefer the Base's organisation, and would be happy to never see a 7,000 line Julia file.

Now, arguably Clojure could benefit from going with (1) a little less than it does, because of the potential for name clashes – but, Julia largely obviates the name clash problem via multiple dispatch, meaning you can have large namespaces very safely. On top of that, having lots of small submodules destroys a huge convenience factor, since you suddenly have to think about lots of internal sub-APIs. But maybe there's a compelling argument for small modules that I've missed.

Another key point here is interactivity. If I reload a file in Julia, I actually want and fully expect that include statements and the relevant files will be reloaded, whereas I don't want to reload every package dependency. It's great that you can make that distinction, and a replacement that doesn't involve that is a deal breaker for me.

[Avik Sengupta]

 > – but, Julia largely obviates the name clash problem via multiple dispatch, meaning you can have large namespaces very safely

 Yes, absolutely, this  is a very important point when comparing namespacing strategies with other languages. I would go further and claim that with multiple disptach, you WANT to have large namespaces. The power of multiple dispatch comes about when you have many functions with the same name. The fact that (+) has 117 methods is a feature, not a problem.

Modules and multiple dispatch have some tension here. This is the reason we have the slight complications of import/using etc. So when comparing module systems with single dispatch languages, we need to consider the special needs of multiple dispatch in this area.

Slightly more generally, I think we are all figuring out how to architect large systems using multiple dispatch. The definitive book on that subject has not been written yet. It certainly will not look like a Java application.

Regards
-
Avik

[Jason Merrill]

On Sunday, December 28, 2014 9:24:38 PM UTC-5, Mike Innes wrote:

I can only speculate about the reason that people still use include, even though Julia has nice modules.

Well, I for one think that

1) Modules containing a large number of functions are OK (and this is very common in more functional languages)

2) If you're going to have (1), you want to split that module across multiple files.

Languages which embrace (1) but conflate modules and files tend to end up looking like this. Personally, I prefer the Base's organisation, and would be happy to never see a 7,000 line Julia file.

I'm in complete agreement with both of your points here--I'm just saying that `include` is a problematic way of achieving these goals.

To rephrase it again, the main thing that I don't like about `include` is that when you are reading a file that ends up being included somewhere else, in general you have to know what context it is going to be included in to know how it will work. That information is not present in the file itself. This essentially forces you to understand new packages in a top down way (because you must first find all of the `include` statements), instead of having the option of reading them bottom up or top down.

The main useful property of a module over a naked `include` is that when you are reading the module, all of the relevant information about scope is present in the file itself, instead of existing in a different file at the `include` site.

I haven't done a careful analysis, but I hypothesize that if you look at how `include` is used in high quality packages, you will generally find that included files don't actually use the scope of the environment they are included into, so from that standpoint, they could usually be modules instead. But I bet you'll also find a few instances where they do use that scope, and you'll never know when without a close reading.

This is why I'm suggesting that maybe what Julia needs is better mechanisms for composing small modules into larger modules. The goal would be to achieve your points (1) and (2) without having lots of files that run in a mystery scope.

[Tobias Knopp]

I think Jason has a point here. The lack of module merging (see exportall, reexport,...) makes include more necessary than it might be. Further we really need #4600.

To be honest: When going through packages I have found myself using the "includes" to understand the package structure.

Cheers

Tobi

[Jason Merrill]

On Monday, December 29, 2014 2:44:55 PM UTC-5, Tobias Knopp wrote:

The lack of module merging (see exportall, reexport,...) makes include more necessary than it might be.

`exportall` is a much better idea than `importunexported` from my post above. I.e. the child module would call `exportall` and the parent module would call either `importall` or `reexport` depending on whether it wanted the child's symbols to be exported or not.

It looks like there has already been a bunch of activity around this question on the issue tracker.

1. The Reexport package already exists: https://github.com/simonster/Reexport.jl
2. There is an issue discussing adding `reexport/reexported` and `exportall` keywords: https://github.com/JuliaLang/julia/issues/1986
3. There is an open pull request to merge some of this functionality into Base: https://github.com/JuliaLang/julia/pull/5626

[Rob Speer]

I surmise that there's a lot of "include"-using Julia code out there because it's the first way that the manual section on "Modules" tells you about putting code in multiple files.

In fact, from the manual, it's really unclear to me how one should distribute code among different files without using "include". The manual mentions very briefly that Julia "searches for modules", but it doesn't say how. Is it assuming some intuition that comes from Matlab? I know my Python intuition doesn't work at all here.

In practice, almost all the Julia code I've written has been one-off single files, with occasional cases where I made a module by trial and error.

The manual section on modules is too much about what you *can* do, and not enough about what you *should* do. I'm looking for some good documentation about best practices for organizing real Julia code.