Compiling multiple modules in one file

[Robert Virding]

Now you can only have one module in each file and the file name is the same as the module name. I rather like this because it make it easy to see what is defined where. However, my work on flavors and defstructs will change this. When you define flavors one extra module is defined for each flavors at compile time. This means that one file can generate its module plus any number of flavor/struct modules. There are no real problems doing this, except for the interface.

Now you just get the "file module" but I would like to change that so the compiler returns a list of the modules it generates. This will change the interface. It will also introduce some new issues. How do we handle it if one module has an error? Generate those that succeed anyway, or have an all-or-nothing attitude?

A more direct problem is that this will change the compiler interface. My immediate questions are:

- How many call the compiler directly through `lfe_comp` or seriously use the return value of `c/1` in the repl?
- And how will this affect tools which call the LFE compiler?

Robert

PS Flavors also generate an extra module for each flavor at runtime but this is no problem. This will stay the same irrespective of how I implement flavors.

PPS With structs you will get the option of defining an interface module for each struct instead of just macros, or both.

PPPS Flavors are coming along, with a slight delay due to the new handling of maps.

[Duncan McGreggor]

First off, this possibility is very cool news -- we'd talked about multi-module files at the LFE mini-summit in Stockholm (EUC2014), and there have been a couple of times since then that I would have used this feature. More below ...

On Wed, Aug 12, 2015 at 5:31 AM, Robert Virding <rvir...@gmail.com> wrote:

Now you can only have one module in each file and the file name is the same as the module name. I rather like this because it make it easy to see what is defined where. However, my work on flavors and defstructs will change this. When you define flavors one extra module is defined for each flavors at compile time. This means that one file can generate its module plus any number of flavor/struct modules. There are no real problems doing this, except for the interface.

Question: to clarify, do you mean that an .lfe module which defines multiple modules in that single file would generate multiple .beam files?

 

I will assume this to be the case for the rest of my responses ...

Follow-up questions:

1) How will those .beam files be named? Will .beam files only take their names from the (defmodule <name> ...) declaration?

2) The naming convention between .lfe file and module will no longer apply:

    2a) Will .lfe files be required to have any particular naming scheme anymore?

    2b) Or will you use a new "module behaviour" so that an .lfe file can be "tagged" as a module(s) file?

    2c) Or will the compiler just assume that an .lfe file is, by default, a module(s) file?

3) Might this be a good time to define a file extension standard for LFE files? E.g., modules = .lfe, shell = .lfsh, lfescript = .lfsc

Now you just get the "file module" but I would like to change that so the compiler returns a list of the modules it generates.

Yeah, that makes sense.

 

This will change the interface. It will also introduce some new issues. How do we handle it if one module has an error?

I just added a breaking change to a module and attempted a compile. This is the message I got:

$ lfec -o ebin src/cmplx.lfe

src/cmplx.lfe:9999: unbound function: #(complex-polar? 2)

If this .lfe module had muItiple modules defined in it, I would imagine:

1) Any module which successfully compiled would have a .beam file generated for it.

2) All errors in compilation would be reported per-file+module, e.g.:

$ lfec -o ebin src/cmplx.lfe

src/cmplx.lfe:complex-polar:9999: unbound function: #(some-polar-func 2)

src/cmplx.lfe:complex-rect:9999: unbound function: #(some-rect-func 4)

Note that rebar now reports the LFE error and then adds its own message:

Compiling src/cmplx.lfe failed:

ERROR: compile failed while processing /Users/oubiwann/lab/lfe/complex: rebar_abort

This will no longer be strictly true ... we may have to push a patch up to rebar's LFE compile support for versions of LFE >= 0.x.x (whichever version has this new feature) to say something like this:

Compiling module complex-polar in src/cmplx.lfe failed:

ERROR: compile failed while processing a module in /Users/oubiwann/lab/lfe/complex: rebar_abort

This will require that a data structure be available via the compile results that allows a developer to associate file, module, and failure. (This is what you were talking about above when you mentioned the changes for what the compile returns.)

 

Generate those that succeed anyway, or have an all-or-nothing attitude?

I'd vote for generating .beam files for all that succeed. This keeps symmetry with how things currently work (e.g., all .lfe files which succeed generate .beam files).

 

A more direct problem is that this will change the compiler interface. My immediate questions are:

- How many call the compiler directly through `lfe_comp` or seriously use the return value of `c/1` in the repl?

I do both a great deal, and I know of other companies which use the former. That being said, everyone I know which uses the LFE compiler functions directly also uses an explicit version of LFE. 

It might be worth releasing a new version of LFE with some warnings about future changes as soon as you've decided upon the new function sigs. This way users will have some warning before the future release which has the compiler changes in it.

Hrm,  in fact, that release might be worth a 1.0 version number ...

 

- And how will this affect tools which call the LFE compiler?

 * I will update lfetool and the Dockerfiles as soon as you release the new version; that's no problem from my end.

 * Existing projects which have been generated by lfetool in the last year or so have pinned LFE versions, so they should be fine

 * Developers who have overridden this behaviour and are pulling from develop will see breakages until they update their projects to use the latest lfetool and rebar

 * depending upon the changes to LFE, there may be other issues with rebar -- I'm very familiar with the rebar LFE compile support and can push a PR which will handle both LFE compile APIs

 * The LFE rebar3 plugins still haven't been finalized, so no worries with that - they can be adjusted as needed

Robert

PS Flavors also generate an extra module for each flavor at runtime but this is no problem. This will stay the same irrespective of how I implement flavors.

PPS With structs you will get the option of defining an interface module for each struct instead of just macros, or both.

Cool! 

PPPS Flavors are coming along, with a slight delay due to the new handling of maps.

Yay! Can't wait to see more :-)

d

 

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[Robert Virding]

Some comments:

- The beam files have to have the same name as the module. This is built into the basic code handling in the code server so there is not much to do about that.
- Seeing we control the LFE compiler we can decouple the .lfe file name from the module name. So we could have foo.lfe ==> foo.beam + foo-more.beam
- I personally am not that hot on explicitly having many modules in one file.
- One idea I am toying with for the return value of the compiler is a list of entries like they are today, one for each module.

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Duncan McGreggor]

On Wed, Aug 12, 2015 at 6:16 PM, Robert Virding <rvir...@gmail.com> wrote:

Some comments:

- The beam files have to have the same name as the module. This is built into the basic code handling in the code server so there is not much to do about that.
- Seeing we control the LFE compiler we can decouple the .lfe file name from the module name. So we could have foo.lfe ==> foo.beam + foo-more.beam

So would this work like ...

1) edit file foo.lfe

(defmodule foo ...)

(defun func-1 () 'in-foo-module)

(defumodule foo-bar ...)

(defun func-1 () 'in-foo-bar-module)

2) Compile --> generate foo.beam and foo-bar.beam?

3) From LFE call (foo:func-1) and (foo-bar:func-1) and get the respective atoms above

 

- I personally am not that hot on explicitly having many modules in one file.

In which case maybe none of what I typed above? Are you thinking instead for this to be purely an LFE internal thing?

I'd almost certainly never use this for a project -- however, I'd *definitely* use it for demos, tutorials, code snipets, and prototyping.

This also kind of starts to come back to the other discussion we had at the summit: switching between virtual modules in the REPL (e.g., instead of only -no-module, allow users to define modules in the REPL and then splurp to them, switch to a different "module" with none of the contents of the previous slurp, then switch back to the previous module with all your vars, funcs, etc. defined).

 

- One idea I am toying with for the return value of the compiler is a list of entries like they are today, one for each module.

Makes sense.

d

 

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

I have just pushed the new version of the compiler which decouples the module name from the .lfe file name. The resultant .beam file still has to be the same as the module name, there is not much I can about that. So you can have a file foo.lfe containing the module foo-bar which when compiled generates the file foo-bar.beam. To_exp, to_core generate files with the original file name as base.

This is the first step in allowing multiple modules in one file.

One problem with this that it will make it harder for tools like rebar to know when a file needs recompiling. I think.

The latest commit of flavors uses this, but you can only have defflavor in a file now, but you can define local functions for the flavor.

Robert

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Duncan McGreggor]

On Tue, Aug 18, 2015 at 12:25 PM, Robert Virding <rvir...@gmail.com> wrote:

I have just pushed the new version of the compiler which decouples the module name from the .lfe file name. The resultant .beam file still has to be the same as the module name, there is not much I can about that. So you can have a file foo.lfe containing the module foo-bar which when compiled generates the file foo-bar.beam. To_exp, to_core generate files with the original file name as base.

This is the first step in allowing multiple modules in one file.

One problem with this that it will make it harder for tools like rebar to know when a file needs recompiling. I think.

Yeah, I can see that.

Fortunately, we'll be in control of LFE compile support in rebar3, so this won't be an issue long-term.

For those still using rebar (the original), we'll probably just need to provide a caveat or two when discussing multi-module files.

 

The latest commit of flavors uses this, but you can only have defflavor in a file now, but you can define local functions for the flavor.

Too fun! Can't wait to try it out :-)

d

 

Robert

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

I am working with this, albeit slowly.

My plan is that when this is complete and works then I will release LFE 1.0 with what we have.

Robert

Robert

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Robert Virding]

I am working on this now. One problem we have to solve is how do we slurp in a file into the shell which contains multiple modules? The shell lifts everything and removes the "module" so all functions become local shell functions. With multiple modules it will be very easy to get macro/function clashes. Three possible solutions are:

- Make it illegal to slurp in multimodule files.
- Just take the first one and ignore the rest.
- Just load each one as it comes and accept eventual conflicts.

We shall see when we get there.

Robert

[Duncan McGreggor]

I have another proposal to offer, entailing waaaay more work :-)

- support "namespaces" in the REPL

and then support switching between namespaces in the REPL. For instance, you have a mods.lfe file with my-mod-1 and my-mod-2 modules defined in it. slurping might then be like the following:

> (slurp "mods.lfe")

#(ok my-mod-1)

#(ok my-mod-2)

> (ns my-mod-1)

#(ok my-mod-1)

my-mod-1> (MODULE)

my-mod-1

my-mod-1> (clashy-func-name 1 2)

2

my-mod-1> (ns my-mod-2)

#(ok my-mod-2)

my-mod-2> (MODULE)

my-mod-2

my-mod-2> (clashy-func-name 1 2)

3

my-mod-2> (my-mod-1:clashy-func-name 1 2)

2

my-mod-2> (ns default)

ok

> (MODULE)

-no-module

> (clashy-func-name 1 2)

exception error: #(unbound_func #(clashy-func-name 2))

> (my-mod-1:clashy-func-name 1 2)

2

> (my-mod-2:clashy-func-name 1 2)

3

Too crazy?

It would be an *amazing* dev experience for those who live in the REPL (you know, like me ...) It would fantastic to be able to switch between modules in the REPL, prototyping/developing multiple ones at the same time :-)

d

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Eric Bailey]

I second this (if not too crazy) as I, too, live in the REPL. Maybe we could borrow from Clojure and have something like in-mod/1 that takes a module name as an atom.

[Robert Virding]

Yes, Duncan's is way more work. :-) Slurping a module just means loading the functions into the interpreter's environment. It would mean a very big modification to how the LFE interpreter handles fully qualified calls (calls with an explicit module name). Now it just calls that (compiled) module, but being able to do it after a slurp they would have to be in the environment under different modules, which doesn't exist today.

Fixing an (in-mod 'foo) repl command so that in the repl you wouldn't need to prefix with module name would be easy. So

> (foo:bar 'the-answer)
42

> (in-mod 'foo)

foo

> (bar 'the-answer)

This would call into the compiled module so you would only reach the exported functions. If this is what you meant?

Robert

[Robert Virding]

I would actually prefer this way as I am not that much a fan of slurp. :-) This is cleaner and a better fit to the system.

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Eric Bailey]

I was thinking in-mod/1 would act (to some extent) like in-ns too, but I think that would necessitate the entirely new module system, as you mentioned.

The prefix ditching behaviour would certainly be helpful in the short term. Maybe you could even call it automatically (or optionally) for the last module loaded, i.e.

> (slurp "foo.lfe") ; or (slurp "foo.lfe" 'in-mod) or similar

#(ok foo)

> (bar 'baz 'qux) ; == (foo:bar ...)

This would especially be helpful with inferior-lfe. I need to revisit my branch of that soon...

Eric

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

The main issue here is that we can't do anything about modules and how code is handled, this is what the erlang VM gives us. The unit of code handling is the module. For everything, for compiling, loading, purging and deleting. So all functions, all code, are in modules and modules only contain compiled code. That's it, period.

So LFE, and Erlang as well, tries to partially get around by having an interpreter which is used by amongst other things the erlang shell and the LFE repl (I sometimes call that a shell as well). LFE uses it to evaluate macros as well, though I am looking into having compiled macros as well. It duplicates work but is very practical.

The LFE evaluator has its own environment with local variables and functions and macros. When you call a function in a module then you make a call to that compiled module with a "normal" erlang/LFE intermodule call. There is no provision in the evaluator for keeping internal modules and explicitly calling functions in them. When you slurp in a file you load in the functions in that module locally into the environment and call them as "local". This may result in overwriting existing functions/macros. This also allows you to call local functions in that module which is normally not at all possible. When you unslurp you roll back to the state before the last slurp, which you also do when you slurp in a new module. There has to be some order.

This is why I am a little sceptical to doing too much with the repl and slurping as it sort of runs parallel to the "real" LFE code world. In production systems all code is in compiled modules. So doing an (in-mod '<mod name>) is fine as it just shortens calling a specific module but you can only call the exported functions. Having a more complex scheme than this is dangerous as in the worst case you can get modules which work fine slurped but not in "real life".

These are my thoughts about it anyway Sorry if I have just regurgitated stuff you already knew.

Robert

[Robert Virding]

I have a sort of working version of this now, but it needs a lot of cleanup before I release it on an unsuspecting public. Amongst other things the return values need to be made consistent and then we need a discussion of what it should look like in the files.

One thing at a time,
Robert

[Duncan McGreggor]

This is a key point:

Having a more complex scheme than this is dangerous as in the worst case you can get modules which work fine slurped but not in "real life".

Having run into minor issues with differences between REPL and compiled module, I'm all for taking the safe route. It may not have all the shiney I could wish for, but your approach will save me days/weeks of pain I would otherwise certainly run into.

d

[Eric Bailey]

+1 for the safe approach. Thanks for the additional background info, Robert.

Eric

[Robert Virding]

OK, I just pushed a new version of the compiler which can handle multiple modules in one file. You basically just concatenate multiple modules and the compiler splits them in to separate modules and generates separate .beam files for each module. The shell 'slurp' and 'c' commands can use it. It seems to work but the interface still needs some work. There are no example files yet but you would write a file like:

(defmodule foo
  (export (a 1))

(defun a (x)
  (* x 42))

(defmodule bar
  (export (m 2))

(defun m (x y)
  (+ (foo:a x) y)

Note that the modules are completely separate and have not coupling at all, exactly like all other LFE/erlang modules. They just happen to be in the same file. So in module 'bar' you must explicitly use the module prefix 'foo:a' when calling it. Trying to implement some form of module interdependency would really screw things up.

I don't know if I like it but it will be useful for flavors so you can have many flavors in one file.

Anyway please experiment with it but be prepared that details may change.

Robert

[Robert Virding]

I am still not convinced that I want multiple modules in one file. Is anyone using this feature and would miss it if it were to disappear?

Robert

[Duncan McGreggor]

I have only used it for some small, unpublished examples. Nothing that depends upon it.

If you get rid of it, what will you do with Flavors?

d

[Robert Virding]

No, there would be no problems as such with flavors. The reason I added multiple modules was to allow defining multiple flavors in one file and also modules using those flavors in the same file. This is necessary as defining a flavor creates a module. It also decouples the name of the file from the name of the resultant module(s).

I would go back to one module per file, but keep the decoupling of the file and module names.

When I fix compiling and exporting macros it would be easier to connect which macros are exported from which module when only having one module. I think anyway.

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Duncan McGreggor]

Awesome. I'm +1, then :-)

d

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[arp...@cryptolab.net]

On 2015-11-03 21:29, Robert Virding wrote:
> I am still not convinced that I want multiple modules in one file. Is
> anyone using this feature and would miss it if it were to disappear?

No, I tried it for a while in toy examples but never felt the need as
I happen to do in Elixir mainly because structs and exceptions must be
defined in their own module.

[Robert Virding]

Yes, that's the main reason I added them. In my case it was flavors :-) but I am planning a defstruct which would encompass at least the existing records and elixir compatible structs and for them it would be nice with multiple modules. So my plan is to leave them and not remove them for the time being. If I can solve the problems I have with them they will remain but if I can't they will go. So please don't base too much on them for the time being.

Btw does anyone know if it is possible to make rebar3 smarter then just using filenames when handling dependencies. This could also be a reason to go back to 1-for-1.

Robert