New macro handling and compiled macros

[Robert Virding]

As I mentioned in the "massive memory leak in lfe_shell ???" discussion I am redoing how macros are handled. The new method makes them and all the forms in eval-when-compile behave like they were in the shell. This will definitely make it easier to do fun things. :-) It seems to work now but needs more testing by real users.

Another thing is having compiled macros in modules. This would allow you to call macros as you would functions and mean you wouldn't need to put them in include files. I have worked a reasonable way of doing this (I think) which works that when the compiler (macro expander) sees a remote call it will go out and check if it is a macro call and expand it. The version I am now working on automatically goes out and checks every remote call and doesn't require any form of import declaration. I don't know if this is a good way to go as it might slow down the compiler too much.

Both of these changes are in the 'dev-macro' branch and really need testing with real code, the more the better. Initially for bugs and speed. You cannot currently export macros but there is included a slightly hacked cl.lfe with a macro definition function to show how I plan to do it. It exports the macros cl:car, cl:cdr and cl:symbol-name which are "callable" as functions in code without declarations.

So please help me if it all works as it should and is not too slow.

Robert

[Robert Virding]

I have now pushed to branch dev-macro a new version which can compile macros. The macros have to be defined after the defmodule and you have to explicitly declare which macros are to be exported using the attribute:

  (export-macro mac1 mac2 ...)              ;Only names

By default no macros are exported. You can then "call" the macros using the normal syntax for calling functions in other modules (foo:bar ...); if they functions they are called, if they are macros they are expanded. When this is complete then we can probably get rid of many include files and use it for things like records and even elixir style structs if we wish.

Note this is still a test version. Two things which must be decided are if we actually want this and if the increased compilation (yes, it takes a bit longer) is acceptable. Only testing will tell.

Robert

[H Durer]

This sounds really great. 

I guess the compilation time impact is something to be tried out.  What is actually the slow bit?  Module loading or checking if the symbol is an exported macro?  

Module loading is a on-off thing with caching already done in BEAM, right?  So we cannot improve on that.  

I'll need to read the code, but are you caching the lookup of the macro definitions?

OTOH, if you already check the module on all explicit mentions in function call position, we can that way generate errors for calling not-exported functions, right?  Or is that un-Erlangy?

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[Robert Virding]

A little about implementation issues. The problem with compiling macros is getting them to expand in the right place, and in this case getting the macros inside the macros to expand in the right place. There are three places it can happen:

- The macro bodies are expanded in the context of the defining module. This is relatively easy and it makes it easy to access other macros and variables (yes we can set internal variables when compiling) defined in the defining module. However then the macros whole bodies are expanded including macros defined in other modules.

- The macro bodies are completely expanded when called. This is also relatively easy but it makes it basically impossible to access macros defined in the defining module without some pretty heavy environment handling. Accessing exported macro would be easy but then they would have to be called with (mod:mac ...) which seems strange to do with local macros.

- Partially expand the macro bodies in the context of the defining module expanding only its local macros. All other expansion is done in the context of the calling module. This is definitely the most beautiful but requires some heavy trickery which I don't have yet.

Currently it's the first alternative which is used, but I can easily change it to the simple second alternative without access to the original modules macros. When I can do the third alternative properly I will migrate towards it.

The slow bit is when you see a call (mod:func ...) to go out and check if this is actually a macro or a function call. Currently I check if the LFE-EXPAND-USER-MACRO function is exported, if so call it, if not check if the module is loaded, if not then try and load it and then try again. I do some caching not to try and load an unloadable file multiple times. Currently it will echo each module it tries to load but can't so you should only see a module once there if my caching works. It can of course handle modules which don't export LFE-EXPAND-USER-MACRO function.

I don't know if this actually takes time as I don't have any large LFE code base to test it on. (yes, yes I know) So any help with testing would be appreciated to see how much it actually affects compile time.

Checking for unexported functions is VERY un-Erlangy! :-) That is the type of thing we do at run-time. Erlang by intention avoids intermodule dependencies. And yes I know that handling macros like this is all about intermodule dependencies but it is not really worse the include files. Anyway this is lisp :-)

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[H Durer]

OK, so the third option you mention would be very much what cl does right?

I now tried to think what this means and came up with this example.

Let's try to emulate this lisp code:

-------------------- 8< common lisp start >8 --------------------
(defpackage :a
  (:use #:cl)
  (:export :m :f :doit))

(defpackage :b
  (:use #:cl)
  (:export :m :f :test))

;;; -----------------------------------------

;;; Package a

;;; -----------------------------------------
(in-package :a)

(defun f ()
  "a:f")

(defmacro m ()
  "a:m")

(defmacro doit (&body b)
  `(list (f) (m) ,@b))

;;; -----------------------------------------
;;; Package b
;;; -----------------------------------------
(in-package :b)

(defun f ()
  "b:f")

(defmacro m ()
  "b:m")

(format t "~s~%"
        (a:doit
         (f) (m)))
-------------------- 8< common lisp end >8 --------------------

This outputs "("a:f" "a:m" "b:f" "b:m")" as expected.

I guess the lfe equivalent would be:

-------------------- 8< a.lfe start >8 --------------------
(defmodule a
  (export all)
  (export-macro m doit))

(defun f ()
  "a:f")

(defmacro m ()
  "a:m")

(defmacro doit (x y)
  `(list (f) (m) ,x ,y))
-------------------- 8< a.lfe end >8 --------------------

-------------------- 8< b.lfe start >8 --------------------
(defmodule b
  (export all))

(defun f ()
  "b:f")

(defmacro m ()
  "b:m")

(defun test ()
  (a:doit (f) (m)))
-------------------- 8< b.lfe end >8 --------------------

right?

And at the moment this outputs "("b:f" "b:m" "b:f" "b:m")" because the expansion of the locally defined macros and functions doesn't happen yet.

Is that a correct understanding?

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

That's a cunning one. I thought I had gotten my description wrong but it is right. What makes it behave differently is CL packages which LFE doesn't, and can't, have. In both cases the backquote pushes all evaluation of what the doit macro returns into the test function in module b. So the (f) and (m) are evaluated when the macro is expanded, in module b. In CL, because of packages, the f and m are symbols in package a not in the ones in package b, which is why they expand to "a:f" and "a:m". In LFE however, they become the f and m in module b which is why they expand to "b:f" and "b:m".

If you define doit as

    (defmacro doit (x y)
      `(list (a:f) (a:m) ,x ,y))

which by being very explicit you get something closer to how CL behaves. Another way which is different but in this case behaves the same is

    (defmacro doit (x y)
      `(list ,(f) ,(m) ,x ,y))

Which just proves that LFE and CL aren't the same, and can never be the same. The question then boils down to how would we like it to behave in this case? Maybe as it is now as it does give you control if you know how to use it? What would cause the least confusion? Is this a reason not to have compile time macros? Hopefully not.

Robert

[Robert Virding]

OK, I have a version of this which works. All that is left is to work out how compiled macros are to coexist with having multiple modules in one file. So if we have the following file:

    (defmacro foo (...) ...)

    (defmodule a
      (export-macro ...))

    (defmacro bar (...) ...)

    (defmodule b
      ...
      (export-macro ...))

    (defmacro baz (...) ...)

with functions, and maybe other macros, defined in the modules a and b.

So the questions are which macros are to be visible where, and which macros can be exported from which module? Some alternate solutions are:

- All macros are "visible" everywhere after they have been defined and can be exported from the module in which they have been defined and those following. So foo can be used everywhere and be exported from modules a and b, bar can be exported from both a and b and baz can be exported from b.

- All macros are "visible" everywhere after they have been defined but they can only be exported from the module in which they have been defined. So foo can be used everywhere but not exported from any module; bar can used after its definition but only exported from module a; and baz can only be exported from module b.

- Macros defined before the first module are "visible" everywhere while macros defined "inside" a module can only be used inside and exported from that module. So foo can be used everywhere; bar can only be used in and exported from module a; and baz can only be used in and exported from module b.

- Macros can only be defined and used inside modules. This means that foo is illegal.

I personally think the first and last options aren't interesting, the first being too lax while the last is way too restrictive. So the second and third are the two options I think are most interesting. I am leaning towards the third as its seems the most consistent.

I am definitely open to a discussion and other suggestions so come with them now before I decide.

Robert

[Duncan McGreggor]

Yeah, as I read through the options, the third was definitely the most appealing. It is the most explicit and has a "natural, expected" behaviour. I hope you keep leaning that way :-)

d

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.

[H Durer]

+1 to that.  I could also live with the last option as a clean, obvious solution, but it is indeed rather restrictive with no real need.

[Robert Virding]

OK, I have now pushed to dev-macro and experimental first version of the new macro handling. It works according the 3rd option. However as yet you cannot export macros. I will fix exporting macros soon. Then we will have to decide if this is the way we want to go.

I have been toying with the idea of making a whole module one form which includes the functions and macros:

(defmodule foo
  (export (a 0) (b 1))

  (defun a () 42)

  (defun b (x) (+ x 42))
)

I am not convinced though so unless someone comes screaming I will leave things as they are.

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[H Durer]

2016-01-12 13:49 GMT+00:00 Robert Virding <rvir...@gmail.com>:

OK, I have now pushed to dev-macro and experimental first version of the new macro handling. It works according the 3rd option. However as yet you cannot export macros. I will fix exporting macros soon. Then we will have to decide if this is the way we want to go.

Cool.

 

I have been toying with the idea of making a whole module one form which includes the functions and macros:

(defmodule foo
  (export (a 0) (b 1))

  (defun a () 42)

  (defun b (x) (+ x 42))
)

Yes, this looks very appealing and I have often wondered why this isn't done that way without finding an answer.

However, neither Common Lisp nor Clojure use that so I assume there must be very good reasons not to.

E.g. interactive development in the repl comes to mind...

 

I am not convinced though so unless someone comes screaming I will leave things as they are.

Sound good.

[Duncan McGreggor]

You know, when I first started learning LFE, for some reason I had expected modules to be defined in that way. Not sure why -- Clojure sets up their modules with a (ns ...) form that closes at the top of the file.

I guess I'm saying that if you *did* decide to switch, part of me would enjoy that :-)

d

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

I have just pushed a new version of handling multi-module files and exporting macros to the 'dev-macro' branch. This is a test version.

Multi-modules is as before in that you can have multiple modules defined in one file and compiling the file compiles them all. It is as before.

The handling of macros now follows the 3rd option. Macros defined before the first module are visible in the whole file, in all the modules, while macros defined inside a module [*] are only visible inside that module. This also applies to records which are just macros.

In the eval-when-compile sections you can evaluate expressions, bind variables and define functions which can then used inside macros. Records already do this but now it has been properly defined.

Macros can now be exported from a module using the (export-macro foo bar baz) declaration in module definitions. These macros can now be "called" from other modules with the normal intermodule call syntax (mod:macro ...) and these will be expanded at compile-time. The defining must have been compiled first for this to work otherwise it will end up as a normal function call. Include files will work as before. This should lessen the need of include files but increase the dependency on compilation order so we have to ensure that our build tools can properly handle compilation order. Circular dependencies will be more difficult to manage, at least with macros, but this is probably a good thing.

What we need to do now is to decide whether this is what we want this or not. I will continue working on it and improving it. But I need feedback.

Don't look at the code though as it is a bit hacky and definitely needs cleaning up before it is properly released. :-)

Here is a small example file mt1.lfe showing some of the features. After the file has been compiled and loaded you can use the macros mt1-a:inside, mt1-a:at, mt1-b:inside and mt1-b:at.

(defmacro before (a b)
  `#(before ,a ,b))

(defmodule mt1-a
  (export (f 0))
  (export-macro inside at)
  )

(defmacro inside (a b)
  `#(mt1-a ,a ,b))

(eval-when-compile
  (set now (erlang:now))
  (set (tuple date time) (tuple (date) (time))))

(defmacro at ()
  `'(at ,date ,time ,now))

(defun f ()
  (list
   (before a 100)
   (inside a 200)
   (at)
   ))

(defmodule mt1-b
  (export (f 0))
  (export-macro inside at)
  )

(defmacro inside (a b)
  `#(mt1-b ,a ,b))

(eval-when-compile
  (set now (erlang:now))
  (set (tuple date time) (tuple (date) (time))))

(defmacro at ()
  `'(at ,date ,time ,now))

(defun f ()
  (list
   (before a 100)
   (inside a 200)
   (at)
   ))

So please test/comment/discuss,

Robert

[*] Inside a module means after the module declaration and before the end of the module irrespective of whether this is because of a following module or the end of file.

[Robert Virding]

I have just pushed a new update of handling multi-module files and exporting macros to the 'dev-macro' branch. This is still a test version but it is closer to production.

You won't really notice any differences but it is cleaner and more modular internally.

I really like that the handling of multi-module files has now been fixed. I intend to keep this handling even if no one really uses it as it is not much slower in compilation than before. It will soon be moved to 'develop' branch.

Robert

[Duncan McGreggor]

Nice!

It's at the top my LFE list to try out ... !

d

[Robert Virding]

So this new handling has been cleaned up and moved over to the develop branch. It will be included in the coming v1.0 release which is on its way real soon now. It will behave basically as has been described here and I also wrote some blogs describing the features. I think this will be a good feature which will give LFE a more lispy feel. As an added bonus the run-time expansion also snuck in.

Robert

[Duncan McGreggor]

Quick check -- you have merged all the macro branch work into develop, yes? (I had a hard time looking at the git log in both branches and seeing that all commits were present from one in the other ...)

If so, I've been having a problem, maybe you can help me see what I'm missing?

When I define this module:

(defmodule mnt-util
  (export (make-fields-macro-name 1))
  (export-macro get-fields))

(defun make-fields-macro-name (record-name)
  (lutil-type:atom-cat 'fields- record-name))

(defmacro get-fields (record-name)
  `(,(make-fields-macro-name `,(cadr record-name))))

I can access the function via mod:fun call, but I can't access get-fields via mod:macro. A macro expansion shows it as being (call 'mnt-util ...) You had warned this is the sort of thing that would happen if the compile time ordering wasn't correct, but the module is getting compiled on the command line, and then the LFE REPL is started up to test it ... so I'm not sure how that could be happening ...

Here's the output of module_info:

> (mnt-util:module_info)
(#(exports
   (#(get-version 0)
    #(get-versions 0)
    #(make-fields-macro-name 1)
    #(module_info 0)
    #(module_info 1)))
 #(imports ())
 #(attributes
   (#(vsn (211590926325113024816083205280682513447))
    #(export-macro (get-fields))))
 #(compile
   (#(options
      (from_core no_bopt
       #(outdir
         "/home/oubiwann/lab/lfe/moneta/_build/default/lib/moneta/ebin")
       debug_info debug_info
       #(src_dirs ("test"))
       #(i "include")))
    #(version "5.0.2")
    #(time #(2016 3 11 4 25 31))
    #(source "/home/oubiwann/lab/lfe/moneta/src/mnt-util.lfe"))))

What am I missing?

Update 1:

I ended up getting the same result, whether I used the develop branch or the dev-macro branch.

Update 2:

Huh. when I compiled from the REPL, it worked:

> (c "src/mnt-util.lfe")
(#(module mnt-util))
> (mnt-util:get-fields 'employee)
(id name salary gender phone room-number)

Any ideas on why regular compiling isn't working?

d

On Mon, Mar 7, 2016 at 8:41 PM, Robert Virding <rvir...@gmail.com> wrote:

So this new handling has been cleaned up and moved over to the develop branch. It will be included in the coming v1.0 release which is on its way real soon now. It will behave basically as has been described here and I also wrote some blogs describing the features. I think this will be a good feature which will give LFE a more lispy feel. As an added bonus the run-time expansion also snuck in.

Robert

[Duncan McGreggor]

Ugh.

Update 3:

Everything works as expected. It was a problem with me overlooking rebar3 (rebar.lock was pinned to an earlier version).

d

[Duncan McGreggor]

More ugh ...

Update 4:

I take it back, everything does not work as expected ... the .beam file I had already compiled manually was still sitting in the dir. I removed it and the problem persists, even with the latest LFE and an updated rebar.lock (which now I think wasn't even part of the problem...).

I guess that means I'm too tired to debug this.

Off to bed ...

d

[Robert Virding]

OK, there are a number of points here:

- Yes, everything is now in the 'develop' branch. 'dev-macro' is no longer supported, I have just forgotten to delete it. You will now also see the LFE mug banner when you start it.

- You don't seem to be using the latest version with exported macros. An easy way to test if you are using the latest version is that there should be exported functions LFE-EXPAND-USER-MACRO/3 and $handle_undefined_function/2

- I have not really tested with rebar, only calling from the REPL.

- I get it to work and I get:

> (c 'mnt-util)
(#(module mnt-util))
> (mnt-util:module_info)
(#(exports
   (#(make-fields-macro-name 1)
    #($handle_undefined_function 2)
    #(LFE-EXPAND-USER-MACRO 3)

    #(module_info 0)
    #(module_info 1)))
 #(imports ())

 #(attributes
   (#(vsn (258936588455996020644114316153695039184))
    #(export-macro (get-fields))))
 #(compile
   (#(options (from_core no_bopt))
    #(version "5.0.1")
    #(time #(2016 3 11 16 24 7))
    #(source "/Users/rv/lfe/lfe-develop/mnt-util.lfe"))))
> (mnt-util:make-fields-macro-name 'sune)
fields-sune
> (macroexpand '(mnt-util:get-fields (one two three)))
(fields-two)
> (macroexpand '(mnt-util:get-fields '(one two three)))
exception error: #(expand_macro
                   (: mnt-util get-fields '(one two three))
                   badarg)
  in (: erlang list_to_atom (102 105 101 108 100 115 45 one two three))
  in mnt-util:make-fields-macro-name/1 (./mnt-util.lfe, line 5)
  in mnt-util:LFE-EXPAND-USER-MACRO/3 (./mnt-util.lfe, line 1)
  in lfe_macro:exp_call_macro/5 (src/lfe_macro.erl, line 944)
  in lfe_macro:exp_predef/3 (src/lfe_macro.erl, line 919)
  in lfe_macro:exp_predef_macro/3 (src/lfe_macro.erl, line 655)
  in lfe_macro:expand_expr_loop/3 (src/lfe_macro.erl, line 99)
  in lfe_macro:expand_expr/2 (src/lfe_macro.erl, line 92)

In the last case the badarg generated an error.

The easiest way to check if you have the right version is the LFE banner. As I have said I have not tested it in rebar. Which rebar?

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Duncan McGreggor]

Yeah, you are undoubtedly correct. I'm not on that machine right now, but once I'm done with work today (and before I get too sleepy!) I'll try it again. Thinking about it now, I'm guessing that the command I was using to invoke the LFE REPL was for the correct version of LFE, but I think the src builds were being done with a different (older) version of LFE.

Robert, I have to say, the new macro code is *incredibly* convenient. After I'd compiled the modules in the REPL, I was able to create macros for Mnesia that were much, much cleaner than the ones I had created a few years ago.

Thanks!

d

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

Yes, it is really useful. I am actually surprised I didn't do it earlier. :-)

Robert

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Duncan McGreggor]

We'll I'm glad for the ska bara -- since now we have it :-)

I have re-checked under the full light of day and wakefulness, and that was the source of the problem -- two different LFE executables getting called. I'm seeing all the right stuff in the REPL now from the compile modules.

Thanks again,

d

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Duncan McGreggor]

So much for wakefulness -- I was wrong. The one thing I hadn't re-checked was the presence of .beam files in ./ -- after removing those, I saw the same behaviour as I was seeing last night.

However, I have thought the problem through carefully  ... and I'm pretty sure it's now obvious what's going on. Again, it's thanks to Robert's clear thinking that the answer became apparent (asking about what's actually doing the compiling). The thing which is actually compiling LFE, all the plugins, and all of the project dependencies is rebar3 with the lfe-rebar3/compile plugin. In order for that plugin to work, to actually compile LFE, it depends upon LFE itself, and will download it. As you can see here, it defines the version of LFE to use:

* https://github.com/lfe-rebar3/compile/blob/master/rebar.config

So, even though I have updated my rebar.lock file and have pointed the moneta project to LFE's develop branch, the tool which is actually doing the compiling is pinned to a released version of LFE.

(I hope you forgive the verbosity here -- I figure this will be useful as other developers run into unexpected lack of features when trying to develop against LFE devel when their tools are synced to actual releases).

So here's what I had to do in order to test my project against a non-released version of LFE:

* rm -rf _build/default/plugins/lfe

* Change _build/default/plugins/lfe-compile/rebar.config to point to LFE's develop branch
* rm -rf _build/default/lib/lfe

* rm ebin/*

* rebuild

Now I see the following in the exports:

    #($handle_undefined_function 2)
    #(LFE-EXPAND-USER-MACRO 3)

and everything is working correctly *without* having to compile from inside the REPL (or forgetting that I had done that!).

Hope this helps other rebar3 users ...

d

[Mason Staugler]

Doesn't this mean that LFE should definitely not use rebar3 to compile itself? If it just used erlc, then the rebar3 plugin handles the compiling of .lfe files and no one needs to include LFE as a project dependency.

(And then just drop support for rebar2.)

[Duncan McGreggor]

Quick note: For those who are just tuning in, I had previously brought up the idea of migrating LFE proper from using rebar(2) to using rebar3.

Yeah, I think you're right Mason. Using rebar3 to compile itself is going to expose us to some potentially knotty bootstrapping issues ... and some much more subtle dependency issues.

I'm +1 on dropping rebar from LFE and not introducing any other build tools into LFE's chain.

d

[Duncan McGreggor]

Sorry, meant "... using rebar3 to compile *LFE* itself ..."

[H Durer]

My grasp of rebar3 is tenuous at best, but if lfe is a dependency of some plug-in, won't we need rebar compilation support to be able to use the plug-in?

[Duncan McGreggor]

The LFE rebar3 compiler plugin is written in Erlang; it has a dependency upon LFE not for compilation of itself, but for the compilation of LFE projects and other plugins. All other LFE plugins (so far) are written in LFE and have the LFE rebar3 compiler plugin as a dependency.

[H Durer]

How will rebar know how to compile lfe for those other plug-ins if lfe doesn't have a rebar.config?

[Duncan McGreggor]

It wouldn't. To be clear, I'm not advocating for removal of the rebar.config -- quite the reverse, actually (I use rebar3 excessively and need rebar.config files in LFE projects). With a simple enough rebar.config file (like it has), LFE can be compiled both by rebar(2) and rebar3.

I'm advocating for the removal of the rebar commands from the Makefile in LFE, so LFE doesn't have to worry about tooling at all -- can can just use Erlang and lfec to do its job, nothing else.

[H Durer]

Ah.  But wouldn't that mean problems with the rebar.config file are less likely to be detected?

I guess at the moment the reverse might be true and problems with the Makefile's fallback logic might go undetected...

[Robert Virding]

I think the only difficulty with rebar.config is to make it first compile the .erl files and then use bin/lfec to compile the .lfe files. I think.

Robert

[Duncan McGreggor]

I'm pretty sure with rebar3 and the rebar3 LFE plugin, the .erl files will always get compiled first.

With rebar2, there are many problems. It really should be abandoned by the community as soon as possible. The rebar3 team was saying that at least a year ago (maybe more...) and they were quite right. The new generation of LFE tooling coming out of the rebar3 work is finally getting to the point where this is now possible, if not completely pain free. The more that make the swtich, though, the more fixes we put in place and the less pain overall :-)

In my LFE work, the single biggest pain I had with rebar(2) was the infamous bug #170 (will I ever forget that number?) ... it caused me a lot of pain before I finally stumbled upon the issue. The second biggest pain that rebar(2) has caused for me has been the issues that LFE users have encountered as a result of using it, when running make Just Works. I take a great deal of responsibility there, though, having encouraged its use so much in 2012 and 2013. I have been rewarded soundly for that.

Given that rebar(2) is deprecated and that we consistently are fielding bugs and a poor user experience due to its use, I am very strongly in favor of dropping it from the LFE Makefile as soon as humanly and humanely possible. I have yet to think of even one good reason for keeping it as a build option -- die-hard fans of rebar(2) will still be able to build LFE with it when(!) we take it out of the Makefile ... *is* there any reason to keep it?

d

On Sun, Mar 13, 2016 at 12:55 PM, Robert Virding <rvir...@gmail.com> wrote:

I think the only difficulty with rebar.config is to make it first compile the .erl files and then use bin/lfec to compile the .lfe files. I think.

Robert

[Eric Bailey]

_____________________________
From: Duncan McGreggor <dun...@mcgreggor.org>
Sent: Sunday, March 13, 2016 4:28 PM
Subject: Re: [lisp-flavoured-erlang] Re: New macro handling and compiled macros
To: Lisp Flavoured Erlang <lisp-flavo...@googlegroups.com>

I'm pretty sure with rebar3 and the rebar3 LFE plugin, the .erl files will always get compiled first.

The general practice here is to use lfe-compile as a post-hook to rebar3's compile. So I think your "always" depends on that. I think we've documented that well though.

With rebar2, there are many problems. It really should be abandoned by the community as soon as possible.

Agreed. The rebar3 team pushed 3.0.0 last week, so I say we go all in.

d