Import .hrl files into LFE

[rvirding]

A question which cropped up in Twitter this week was if it was a way to import .hrl files into LFE and get hold of record definitions in them. Currently there isn't. There exists a tool to convert them into LFE files, https://github.com/cadar/hrl-to-lfe.

It is not too difficult and we can even manage macro definitions with the following limitations:

- All type information will be ignored
- In record definitions the default values cannot contain macro calls
- Macros must expand to complete syntactic elements and not contain macro calls

With those limittations it should not be too difficult. The only issue remaining is how to call it. We already have (macros)

(include-file "filename")
(include-lib "libpath/filename")

One alternative is to add a new (include-hrl "filename"). A better one, I think, would be to modify the existing ones so that if the file name ends in .hrl then read and process it as a .hrl file, otherwise as a lfe file.

What do you think?

Robert

[dun...@cogitat.io]

Fwiw, I'm +1 on the second option (treating *.hrl files intelligently using the existing functions).

d
 

[rvirding]

I have now committed a very first test version of including .hrl files and extracting record definitions. It is on the 'develop' branch at https://github.com/rvirding/lfe.

It assumes that if the name of the include file ends in ".hrl" then it is to be parsed as an erlang file; this applies to both include-file and include-lib.It extracts the record definitions and, where possible, converts the to LFE record definitions. It ignores everything else in the file.

As an extra bonus macro definitions are also parsed, applied in the .hrl file, and where possible extracted and converted to LFE macro definitions. This is not always trivial and the are some caveats:

- macros where the bodies expand to multiple expressions are ignored as this makes the backquoting difficult
- it can handle multiple definitions with different number of arguments for a macro but if both the -define(foo, ...) and -define(foo(), ...) occur then the no-argument function-like version shadows the "base" one. So here the foo() definition will be used
- these are straight LFE macros so they are called as function calls
- it tends to silently ignore those macros it can't translate

This is definitely an alpha version so please try it and comment.

Robert

[Duncan McGreggor]

I've finally gotten to this part of the user guide :-)
  http://lfe.github.com/user-guide/data/5.html

But as you can see, this section is still blank. The reason is that I'm not really sure how includes work in Erlang and all of my experiments in LFE have failed :-/

I'm not sure if you can run (include-file from the REPL, but I tried multiple variations and nothing seemed to work. I also tried compiling and slurping (from the REPL) an .lfe file that did (include-file "some.hrl"), and that didn't work for me either.

Any pointers?

Thanks!

d

Robert

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-flavoured-erlang?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[rvirding]

The REPL is rather limited in what you can create in it. You can only define functions and macros by slurping in a file and that will replace all previous definitions with the new ones. When a file is slurped the functions and macros are not compiled but interpreted. Include-file is a macro so it can't be called directly from the shell.

Include-file does not have a search path, it just tries to open the file. The .hrl file handling is only in the 'develop' branch yet as it has not been tested. Could you please send an example and I will try and see what went wrong.

Robert

[dun...@cogitat.io]

Before I get help on .hrl files, I think I need help on regular includes ;-)

When I try to compile a file with an include-file call in it, I get errors:

 $ cd lfe/test/visual
 $ ../../bin/lfe -pa ../../ebin
 Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

LFE Shell V5.10.1 (abort with ^G)
> (slurp '"test_inc.lfe")
exception error: #(badmatch #(error (#(4 lfe_macro #(expand_macro ...))) ()))

> (c '"test_inc")       
./test_inc.lfe:4: error expanding (include-file (116 101 115 116 ...))
error
>

Similarly, when I try to compile from the command line:

$ ../../bin/lfec test_inc.lfe
./test_inc.lfe:4: error expanding (include-file (116 101 115 116 ...))

Or this:

$ erl -pa ../../ebin -s lfe_comp file test_inc -s erlang halt
Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V5.10.1  (abort with ^G)
1> ./test_inc.lfe:4: error expanding (include-file (116 101 115 116 ...))

Any ideas?

Thanks!

d

[Robert Virding]

I think the error is that it can't find the include file. The compiler
has no search path for include files but they are only relative to the
current working directory. Unfortunately. Try compiling with:

> (c '"mst" '(verbose return))
./mst.lfe:5: error expanding (include-file (108 102 101 47 ...))
#(error
(#("./mst.lfe"
(#(5
lfe_macro
#(expand_macro
(include-file "lfe/include/match-sspec.lfe")
#(#(none file enoent) "lfe/include/match-sspec.lfe"))))))
())

and you should get a similar output which means it can't find the file
(the enoent). I have just pushed a patch to the develop branch which
outputs a little better macro error messages. The handling of
include-file and include-lib wrt search paths needs to be improved.

Robert

[Duncan McGreggor]

On Tue, Mar 26, 2013 at 2:31 AM, Robert Virding <rvir...@gmail.com> wrote:

I think the error is that it can't find the include file. T

The odd thing about that, though, is that I was doing this against the test files in the lfe repo... it wasn't something I had created. I did have the same problems with my own code, but I wanted to eliminate the chance of a typo, etc., until I understood the system.

Do the includes in lfe test suites still work for you? If so, what's your pwd when you compile the code? And what's the exact command you use?

More below...
 

he compiler
has no search path for include files but they are only relative to the
current working directory. Unfortunately. Try compiling with:

> (c '"mst" '(verbose return))
./mst.lfe:5: error expanding (include-file (108 102 101 47 ...))
#(error
  (#("./mst.lfe"
     (#(5
        lfe_macro
        #(expand_macro
          (include-file "lfe/include/match-sspec.lfe")
          #(#(none file enoent) "lfe/include/match-sspec.lfe"))))))
  ())

and you should get a similar output which means it can't find the file
(the enoent). I have just pushed a patch to the develop branch which
outputs a little better macro error messages. The handling of
include-file and include-lib wrt search paths needs to be improved.

I did pull down the latest and attempted to compile from the lfe repl, but no luck. I tried every permutation of relative and absolute paths I could think of, and I still got the new, cleaned-up error message. Here's the error from the last thing I tried:

> (c '"gp-util" '(verbose return))
./gp-util.lfe:1: error expanding (include-file (47 85 115 101 ...))
#(error
  (#("./gp-util.lfe"
     (#(1
        lfe_macro
        #(expand_macro
          (include-file
           "/Users/oubiwann/lab/erlang/lfe-gp/include/gp-const.lfe")
          undef)))))
  ())
>

Here's a relative path error, after I moved the file into an include dir:

> (c '"gp-util" '(verbose return))
./gp-util.lfe:1: error expanding (include-file (46 47 105 110 ...))
#(error
  (#("./gp-util.lfe"
     (#(1
        lfe_macro
        #(expand_macro (include-file "include/gp-const.lfe") undef)))))
  ())
>

I also tried it with (include-file "./include/gp-const.lfe"), but no change.

I've triple-checked the spelling and even copied the name after an "ls" and pasted that into the (include-file ...) to be extra-sure.

I'm 100% sure I'm doing something dumb :-/ Thanks for your help!

d

[Robert Virding]

This is strange. Could you please send me the files and I will check
if there is some other error hiding in there and being shown as an
include-file error. Give me something to do on the flight, at least
until the battery runs out. :-)

I notice that the actual error is 'undef' which means a call to an
undefined module or function. I broke out the handling of the include
macro into a separate file lfe_macro_include.erl. You have compiled
this and it is in the right directory?

Anyway send me the files and I will check some more.

Robert

[Duncan McGreggor]

On Tue, Mar 26, 2013 at 8:36 AM, Robert Virding <rvir...@gmail.com> wrote:

This is strange. Could you please send me the files and I will check
if there is some other error hiding in there and being shown as an
include-file error. Give me something to do on the flight, at least
until the battery runs out. :-)

I notice that the actual error is 'undef' which means a call to an
undefined module or function. I broke out the handling of the include
macro into a separate file lfe_macro_include.erl. You have compiled
this and it is in the right directory?

See, I *knew* I was doing something dumb :-)

I've done the obvious now and recompiled lfe, but I'm getting a new error that seems to indicate that there is problem with the macros or the include file:

> (c '"gp-util.lfe" '(verbose return))        
./gp-util.lfe:8: error expanding (include-file "include/gp-const.lfe")
#(error
  (#("./gp-util.lfe"
     (#(8

        lfe_macro
        #(expand_macro (include-file "include/gp-const.lfe") undef)))))
  ())
>

I can slurp the include file and use the defined macros without issue, but compile fails...

I've pushed the nascent code here:
  https://github.com/oubiwann/lfe-gp

Thanks again :-)

d

[Robert Virding]

I don't know what is going wrong but for me it works. I pulled the
lfe-gp master branch into directory lfe-gp-master which was under my
lfe directory. My lfe files are in the lfe/ebin directory so I
reference them explicitly. Then I did:

rv ~/erlang/lfe$ cd lfe-gp-master/
rv ~/erlang/lfe/lfe-gp-master$ ../bin/lfe -pa ../ebin
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe]
[kernel-poll:false]

LFE Shell V5.9.1 (abort with ^G)
> (c '"gp-util")
#(module gp-util)
> (: gp-util random-nth (: lists seq 1 100))
45

I then uncommented your functions and recompiled:

> (c '"gp-util")
#(module gp-util)
> (: gp-util random-element '(a b c d e))
> (c '"gp-util")
#(module gp-util)
> (: gp-util random-nth (: lists seq 1 100))
51
> (: gp-util random-nth (: lists seq 1 100))
32
> (: gp-util random-element '(a b c d e))
=input=
> (: gp-util random-element '(a b c d e))
=input=
> (: gp-util random-element '(a b c d e))
(c
(b
=input=
(c
(c
(a =input= (c (a =input= =input=) (a (a =input= =input=) =input=)))
(b =input= (a (a =input= =input=) =input=)))
(d =input= =input=)))
=input=)

Slurping went well and I could use the macros. Why it would work when
you slurp and not compile I don't know as they "should" be doing the
same thing. Include-file is a bit hacky and there is no handling of a
search path or such so the path is relative to where *you are* and not
where the including file is. This is almost a bug and I will fix real
soon and push it immediately. What I will probably do is to not make
it a macro but a special form handled by the compiler/slurper in which
case it will be much easier to handle a search path. Same thing for
include-lib.

I followed the link and this looks interesting. When you finish it I
would definitely want to include it in my examples, as well as the
web-page if you wish to describe it there.

Robert

[Duncan McGreggor]

Hrm, I've tried lots and lots of different things, eliminating one possibility after another for the error. I now suspect that there is some change in R16B that breaks this :-/

After dinner I will download  an R15B from somewhere (if I can get it to not conflict with the Homebrew install of R16B) and give that a try...

d

[Duncan McGreggor]

I just downloaded and installed R15B03 from the Erlang Solutions web site, and it works like a charm. I re-read the release notes (http://www.erlang.org/download/otp_src_R16B.readme), but nothing jumped out at me as a likely cause...

Given the difficulties with R16B, I will test the .hrl support with R15B03 instead :-)

d

[Robert Virding]

That definitely sounds like something for me. I will ger onto it when I get home.

Robert

From my Nexus

[Robert Virding]

This has now been fixed for R16B and I have now pushed the fix onto the 'develop' branch. The cause of the problem was me being naughty and using an undocumented function to test file names, and the little ***s removing it in R16. Anyway it is now fixed. Let me know of more bugs.

Robert

[dun...@cogitat.io]

Thanks, Robert!

d

[Duncan McGreggor]

Possibly not related to your feature: I've tried compiling (with lfec) a very simple eunit .lfe file and I get this error:

test/ledis-api-tests.lfe:4: error expanding (include-lib "eunit/include/eunit")
test/ledis-client-tests.lfe:4: error expanding (include-lib "eunit/include/eunit")

Is eunit not supported?

(Perhaps the macros are too complex for what LFE does with .hrl files?)

Thanks!

d

Thanks, Robert!

d

[Robert Virding]

Try doing:

(include-lib "eunit/include/eunit.hrl")

It does not assume anything about the file name, so you have to give the .hrl as well (and the .lfe).

I did a quick check and it seems to work but it only defines the macros IF, EUNIT_HRL, _cmd_, EUNIT and TEST which is not very useful. I will try and find out what is going on and see what I can do about it.

Robert

[Duncan McGreggor]

On Tue, May 14, 2013 at 5:56 PM, Robert Virding <rvir...@gmail.com> wrote:

Try doing:

(include-lib "eunit/include/eunit.hrl")

It does not assume anything about the file name, so you have to give the .hrl as well (and the .lfe).

Ah, good to know. I had tried all sorts of combinations, and none of them seemed to work. For instance:

$ ERL_LIBS=./deps/lfe:./deps/eredis:./ ./deps/lfe/bin/lfec -o ./.eunit test/*.lfe
test/ledis-api-tests.lfe: no such file or directory
test/ledis-client-tests.lfe: no such file or directory

However:

$ ls -l test/*.lfe
-rw-r--r--  1 oubiwann  staff  114 May 14 18:27 test/ledis-api-tests.lfe
-rw-r--r--  1 oubiwann  staff  117 May 14 18:30 test/ledis-client-tests.lfe

 

Not sure what I'm doing wrong there -- it works just fine for ./src/*.lfe

I did do this, though, and no error was raised:

ERL_LIBS=./deps/lfe:./deps/eredis:./ ./deps/lfe/bin/lfe
Erlang R15B03 (erts-5.9.3) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

LFE Shell V5.9.3 (abort with ^G)
> (c '"test/ledis-api-tests.lfe")
#(module ledis-api-tests)
>

I did a quick check and it seems to work but it only defines the macros IF, EUNIT_HRL, _cmd_, EUNIT and TEST

Ah, that sounds like a new trick I don't know in Erlang: how do you check to see what has been brought in to the current namespace by a compile, slurp, etc.? I half-remember something, but nothing here jogs my memory: http://www.erlang.org/doc/man/c.html ...

 

which is not very useful. I will try and find out what is going on and see what I can do about it.

Cool, thanks!

d

[Duncan McGreggor]

On Tue, May 14, 2013 at 6:41 PM, Duncan McGreggor <dun...@cogitat.io> wrote:

On Tue, May 14, 2013 at 5:56 PM, Robert Virding <rvir...@gmail.com> wrote:

Try doing:

(include-lib "eunit/include/eunit.hrl")

It does not assume anything about the file name, so you have to give the .hrl as well (and the .lfe).

Ah, good to know. I had tried all sorts of combinations, and none of them seemed to work. For instance:

$ ERL_LIBS=./deps/lfe:./deps/eredis:./ ./deps/lfe/bin/lfec -o ./.eunit test/*.lfe
test/ledis-api-tests.lfe: no such file or directory
test/ledis-client-tests.lfe: no such file or directory

Well, nevermind that nonsense. User error (missing mkdir in my make target).

 

[snip]

I did a quick check and it seems to work but it only defines the macros IF, EUNIT_HRL, _cmd_, EUNIT and TEST

Ah, that sounds like a new trick I don't know in Erlang: how do you check to see what has been brought in to the current namespace by a compile, slurp, etc.? I half-remember something, but nothing here jogs my memory: http://www.erlang.org/doc/man/c.html ...

I'm still curious about this one, though!

d

[Robert Virding]

On 15 May 2013 07:32, Duncan McGreggor <dun...@cogitat.io> wrote:

I did a quick check and it seems to work but it only defines the macros IF, EUNIT_HRL, _cmd_, EUNIT and TEST

Ah, that sounds like a new trick I don't know in Erlang: how do you check to see what has been brought in to the current namespace by a compile, slurp, etc.? I half-remember something, but nothing here jogs my memory: http://www.erlang.org/doc/man/c.html ...

I'm still curious about this one, though!

I cheated a bit :-)

> (macroexpand '(include-lib "eunit/include/eunit.hrl"))
(progn
  (defmacro IF ((B T F) `(case ,B ('true ,T) ('false ,F))))
  (defmacro EUNIT_HRL (_ `'true))
  (defmacro EUNIT (_ `'true))
  (defmacro TEST (_ `'true)))

You can also give an extra argument to macroexpand which is the environment to use for macro definitions. With no explicit environment it just uses the default LFE macros.

There is a built-in lfe shell variable $ENV which is set to the current shell environment. So if you have slurped in a file which contains macros then you can test them using:

> (macroexpand '(my-macro a b c) $ENV)

You can of course just evaluate $ENV to see what the current environment looks like. It contains variables, functions and macros. So:

> $ENV
(#(variable *** ())
 #(variable
   **
   (progn
     (defmacro IF ((B T F) `(case ,B ('true ,T) ('false ,F))))
     (defmacro EUNIT_HRL (_ `'true))
     (defmacro EUNIT (_ `'true))
     (defmacro TEST (_ `'true))))
 #(variable
   *
   (progn
     (defmacro IF ((B T F) `(case ,B ('true ,T) ('false ,F))))
     (defmacro EUNIT_HRL (_ `'true))
     (defmacro EUNIT (_ `'true))
     (defmacro TEST (_ `'true))))
 #(variable - (macroexpand '(include-lib "eunit/include/eunit.hrl") $ENV))
 #(variable +++ ())
 #(variable ++ (macroexpand '(include-lib "eunit/include/eunit.hrl")))
 #(variable + (macroexpand '(include-lib "eunit/include/eunit.hrl") $ENV)))

[dun...@cogitat.io]

Thanks! I didn't know about $ENV... I'll highlight that somewhere in the docs ...

d

[Duncan McGreggor]

Okay, I've run through a couple of examples and am able to import .hrl files without issue. Looks good to me :-)

Thanks!

d

[Duncan McGreggor]

I've added $ENV to this section of the LFE docs:
  http://lfe.github.io/user-guide/intro/2.html

Towards the end of that page, you'll see some example usage. In particular, I've found the following little function to be useful when loading modules and in general checking on the contents of $ENV:

> (set filter-env
    (lambda (env)
      (: lists foreach
        (match-lambda
          (((tuple 'function func-name arity _))
           (: io format '"function: ~p/~p~n" (list func-name arity)))
          (((tuple 'macro macro-name _))
           (: io format '"macro: ~p~n" (list macro-name)))
          (((tuple 'variable var-name value))
           (: io format '"variable: ~p~n" (list var-name)))
          ((_)))
        env)))

> (funcall filter-env $ENV)
variable: 'my-var-4'
variable: 'my-var-3'
...

[dun...@cogitat.io]

Update for folks who haven't been watching the repo: Robert has done a ton of additional work around includes and the subtleties of getting odd bits of eunit.hrl to work in LFE.

Thanks again, Robert -- this is really great :-) I've been testing it out tonight for the first time, now that I'm back in town, and assert, assertNot, assertEqual, assertMatch, etc., all seem to be working as expected -- I'm so delighted! Also, I've been amazed that the line numbers are working too; that's a big bonus.

All of this gets me to 90% of where I need to be, and will keep me busy (updating libraries, etc.) for a bit.

However, as you might expect, I do have some questions :-)

1) The include-lib doesn't work for me when I compile the unit tests. However, if I hop into the REPL, do a macroexpand on the .hrl file, and then re-run the unit tests, everything works. I'm guessing there's a step I need to take in the compile process, but I don't know what's going on behind the scenes that would explain this behaviour, so I wanted to ask you about it before I started hacking for a "fix" in my make targets :-)

2) I think this is related, and I'm pretty sure you've answered this before (sorry): when I include-lib the eunit.hrl in the REPL, I get an error. I can, however, macroexpand in the REPL. Is there any way to use the functions/macros defined by eunit.hrl in the REPL for testing? Or is that not possible right now?

3) Do I remember correctly that the (my_test_ () ... (_test ...)) and (my_test_ () ... (_assert* ...)) style tests (with the trailing underscore) aren't going to work in LFE? (When I tried, I got some errors... process-killed errors; but before I dug too deeply, I wanted to check with you.)

4) If my recollection for #3 is correct, then fixtures won't be supported either; I'm guessing I could add support via some custom LFE macros... any pointers that would get me started in the right direction?

Thanks again!!!

d

[Duncan McGreggor]

Self-follow-up, in case future readers are wondering...

On Tue, Dec 31, 2013 at 12:11 AM, <dun...@cogitat.io> wrote:

Update for folks who haven't been watching the repo: Robert has done a ton of additional work around includes and the subtleties of getting odd bits of eunit.hrl to work in LFE.

Thanks again, Robert -- this is really great :-) I've been testing it out tonight for the first time, now that I'm back in town, and assert, assertNot, assertEqual, assertMatch, etc., all seem to be working as expected -- I'm so delighted! Also, I've been amazed that the line numbers are working too; that's a big bonus.

All of this gets me to 90% of where I need to be, and will keep me busy (updating libraries, etc.) for a bit.

However, as you might expect, I do have some questions :-)

1) The include-lib doesn't work for me when I compile the unit tests. However, if I hop into the REPL, do a macroexpand on the .hrl file, and then re-run the unit tests, everything works. I'm guessing there's a step I need to take in the compile process, but I don't know what's going on behind the scenes that would explain this behaviour, so I wanted to ask you about it before I started hacking for a "fix" in my make targets :-)

I am not seeing this error any more; I'm using include-lib in lfeunit now with success.

 

2) I think this is related, and I'm pretty sure you've answered this before (sorry): when I include-lib the eunit.hrl in the REPL, I get an error. I can, however, macroexpand in the REPL. Is there any way to use the functions/macros defined by eunit.hrl in the REPL for testing? Or is that not possible right now?

I did end up finding the email where Robert explained this, and no -- it's no possible in the REPL right now :-)

As a work-around for this, in a regular .lfe file I simply (include-lib ...) any macros I want to test and then (slurp ...) that .lfe file in the REPL.

 

3) Do I remember correctly that the (my_test_ () ... (_test ...)) and (my_test_ () ... (_assert* ...)) style tests (with the trailing underscore) aren't going to work in LFE? (When I tried, I got some errors... process-killed errors; but before I dug too deeply, I wanted to check with you.)

I haven't fully explored this yet. I do have a need for this in a project that's in my queue, and should be able to report back when I do get around to it.
 

4) If my recollection for #3 is correct, then fixtures won't be supported either; I'm guessing I could add support via some custom LFE macros... any pointers that would get me started in the right direction?

Again, more later. Initial tests with fixtures seem to be okay, but I haven't fully pushed the limits of what is going to be possible in LFE.

d

 

Thanks again!!!

d

On Tuesday, May 14, 2013 7:56:18 PM UTC-5, rvirding wrote:

Try doing:

(include-lib "eunit/include/eunit.hrl")

It does not assume anything about the file name, so you have to give the .hrl as well (and the .lfe).

I did a quick check and it seems to work but it only defines the macros IF, EUNIT_HRL, _cmd_, EUNIT and TEST which is not very useful. I will try and find out what is going on and see what I can do about it.

Robert

On 15 May 2013 02:34, Duncan McGreggor <dun...@cogitat.io> wrote:

Possibly not related to your feature: I've tried compiling (with lfec) a very simple eunit .lfe file and I get this error:

test/ledis-api-tests.lfe:4: error expanding (include-lib "eunit/include/eunit")
test/ledis-client-tests.lfe:4: error expanding (include-lib "eunit/include/eunit")

Is eunit not supported?

(Perhaps the macros are too complex for what LFE does with .hrl files?)

Thanks!

d

On Sat, Apr 6, 2013 at 9:48 PM, <dun...@cogitat.io> wrote:

Thanks, Robert!

d

On Wednesday, April 3, 2013 4:51:06 PM UTC-7, rvirding wrote:

This has now been fixed for R16B and I have now pushed the fix onto the 'develop' branch. The cause of the problem was me being naughty and using an undocumented function to test file names, and the little ***s removing it in R16. Anyway it is now fixed. Let me know of more bugs.

Robert

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.