LFE built-in functions

[Robert Virding]

Today we can reach all the erlang BIFs from LFE. They are all in the module erlang but many we can reach without prefixing with the module, the same ones as we can from erlang. I plan to create a set of LFE BIFs which will be in the module lfe and which can also be reached without prefixing the module. In one sense these will be part of the LFE language/system in the same way as we have a collection of standard macros. So far I have plucked out a few from the CL book:

acons, pairlis, assoc, assoc-if, assoc-if-not, rassoc, rassoc-if, rassoc-if-not, subst, subst-if, subst-if-not, sublis, eval

I have no intention of adding interface functions to the standard erlang BIFs. This is not a replacement of the libraries in anyway it just defining which functions are part of the LFE "system".

I am open to suggestions for functions that would be reasonable to include here.

Robert

[Duncan McGreggor]

I think it'd be really helpful to Lispers if we included MAPC, MAPCAR, MAPCAN, MAPL, MAPLIST, and MAPCON. Most of these have different behaviours than related (lists:*) functions, so it wouldn't be adding an interface to include them ...

Also ADJOIN is a good addition to the others you mentioned ... I'd also use RPLACA and RPLACD.

MACLISP had XCONS; I'd like to use that on occasion.

Ones that I  define for myself on a semi-regular basis are what I call rcdr, rcar, and rcons, where:

lst -> '(1 2 3 4)

(rcar lst) -> 4

(rcdr lst) -> '(1 2 3)

(rcons '(1 2 3) 4) -> '(1 2 3 4)

I'd like to be able to use rcons in pattern matching where I could match "all-but-last" and "last" (the reverse of matching with cons where one gets head and tail).

That's all I can think of...

d

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.
Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[Adam Krupicka]

How about the functions from lutil's include/core.lfe?

next, range, take, drop, split-at, split-by, interleave, get-in, reduce, repeat (if it gets merged).

Plus include/compose.lfe: function composition, partial application, and the threading macros --- all very useful stuff.

Or is that too many or too high-level to be included in the built-ins?

[Duncan McGreggor]

One thing to consider is that those functions are very much inspired by Clojure, while LFE has historically embraced a CL/MACLISP lineage. So, just from a consistency and style perspective, including the Clojure-inspired stuff might be a little schizophrenic...

That being said, I constantly use the Clojure-inspired functions :-) I personally don't mind including lutil/include/core.lfe when I need them. (And yes, your repeat function will get merged!)

d

P.S. I've begun a new library that will do two things: 1) take the most generally useful includes from lutil, and 2) explore a unifying interface across various similar data types in Erlang (e.g., list-like, dict-like, set-like). Ideally, the ultimate solution will be a common set of functions that will work across not only applicable data types from the Erlang standard library, but also any community-created libraries that declare a list-like, set-like, dict-like, etc., behaviour. Not quite sure how I'm going to do this yet, but it might required custom data types to be gen_servers with a new function that then stores the type (and reference to the module implementing the functions) so that the core library could call back into it. That's going to require a lot of experimentation to get right.

--

[Duncan McGreggor]

On Fri, Jun 5, 2015 at 10:11 AM, Duncan McGreggor <dun...@mcgreggor.org> wrote:

One thing to consider is that those functions are very much inspired by Clojure, while LFE has historically embraced a CL/MACLISP lineage. So, just from a consistency and style perspective, including the Clojure-inspired stuff might be a little schizophrenic...

That being said, I constantly use the Clojure-inspired functions :-) I personally don't mind including lutil/include/core.lfe when I need them. (And yes, your repeat function will get merged!)

d

P.S. I've begun a new library that will do two things: 1) take the most generally useful includes from lutil, and 2) explore a unifying interface across various similar data types in Erlang (e.g., list-like, dict-like, set-like). Ideally, the ultimate solution will be a common set of functions that will work across not only applicable data types from the Erlang standard library, but also any community-created libraries that declare a list-like, set-like, dict-like, etc., behaviour. Not quite sure how I'm going to do this yet, but it might required custom data types to be gen_servers with a new function that then stores the type (and reference to the module implementing the functions) so that the core library could call back into it. That's going to require a lot of experimentation to get right.

I've got a spreadsheet I created a few weeks ago to track all the functions for the libraries; I've shared it here:

 * https://docs.google.com/spreadsheets/d/1TBx8WIsm9r5eWmZXDqpRGEumSdL1UDRI69Wn7-u9ij8/edit?usp=sharing

I haven't finished integrating functions from ETS/DETS, but that's next. Then grouping by list/dict/set.

d 

[Adam Krupicka]

On Friday, June 5, 2015 at 5:11:59 PM UTC+2, Duncan McGreggor wrote:

One thing to consider is that those functions are very much inspired by Clojure, while LFE has historically embraced a CL/MACLISP lineage. So, just from a consistency and style perspective, including the Clojure-inspired stuff might be a little schizophrenic...

You are right; for a moment, I forgot the 'L' in 'LFE' actually stands for 'Lisp', not 'Clojure'.

Short background: a while ago, I used Clojure for a short project. I had no prior knowledge of any Lisp-like language, yet I was able to get the basic hang of Clojure in a few days. I do not know what the Lisp way is, but the Clojure way is pretty damn awesome.

So anyway, I was thinking we could make LFE better (in the most subjective sense) by including some Clojure-like functionality (Clojure on the BEAM? Sounds awesome!), but I can see now how that might be unwelcome and perhaps better kept in a separate library.

I guess what I'm trying to say is, sorry for trying mess up your thing :)

[Duncan McGreggor]

On Fri, Jun 5, 2015 at 11:46 AM, Adam Krupicka <krupic...@gmail.com> wrote:

On Friday, June 5, 2015 at 5:11:59 PM UTC+2, Duncan McGreggor wrote:

One thing to consider is that those functions are very much inspired by Clojure, while LFE has historically embraced a CL/MACLISP lineage. So, just from a consistency and style perspective, including the Clojure-inspired stuff might be a little schizophrenic...

You are right; for a moment, I forgot the 'L' in 'LFE' actually stands for 'Lisp', not 'Clojure'.

Short background: a while ago, I used Clojure for a short project. I had no prior knowledge of any Lisp-like language, yet I was able to get the basic hang of Clojure in a few days. I do not know what the Lisp way is, but the Clojure way is pretty damn awesome.

I've heard the same thing from many folks -- and I think you're all quite correct :-)

I did come to Clojure after using Lisp for a while, but it really was something special. Rich Hickey is quite brilliant at language design... 

 

So anyway, I was thinking we could make LFE better (in the most subjective sense) by including some Clojure-like functionality (Clojure on the BEAM? Sounds awesome!), but I can see now how that might be unwelcome and perhaps better kept in a separate library.

I guess what I'm trying to say is, sorry for trying mess up your thing :)

No, man! You didn't mess a thing! :-)

(Keep it comin'!)

d

[Robert Virding]

I just want to confirm what Duncan said: Adam, you are not messing anything up so please keep the comments coming.

Clojure has some basic properties which are different to LFE so some things from one won't work on the other.

I just want to say that we have to be careful what we put it in the LFE BIFs. I don't see this as a library but more as a part of the actual LFE language. Like erlang BIFs. So this is not really a library nor a replacement for libraries.

Just a thought,

Robert

[Mário Guimarães]

Hi,

I don't think that reusing CL names will do any good regarding adoption of LFE.

Instead, LFE should bring to Erlang VM the high flexibility of Lisp (i.e., user and reader macros, hot code swapping, logging into a running system, etc), but with a modern nomenclature much like that in other functional languages like Clojure and Haskell.

A modern nomenclature would not only help programmers using other modern/mainstream languages come to Lisp and Erlang VM, but will be easier for beginners and students to get hands on Lisp.

CL has made its contribuition, and it was enourmous, but we need to improve on that, and regarding this, drop, take,map,reduce,foldl,foldr, first, second, rest, and the like, seems easier to read and more obvious than cdr, car, mapc, mapcar, mapX, mapY, map...

Remember: make the right choices for LFE, while it is still being created, as there will not second opportunity to make it win.

Just another topic: one thing that I would appreciate very much in LFE, in which CL is good at, and Erlang is a real pain, is to have a good FFI to C/C++. This would be a great win!

I am just trying to help shape LFE into a Lisp I would definitely use on a great VM, while it is possible ;-)

Mário

[Duncan McGreggor]

These are very good points, well worth considering.

Just a note, though: we can have both :-) I absolutely adore LFE for providing some of the classic Lisp forms that you can't get as a built-in in "modern" Lisps. I will always treasure a Lisp that lets me use the old forms, even if they are tucked away in a dusty corner of the language and no one else ever uses them :-)

(I will, of course, also use the new stuff ... except for first/last ... I think I'll always use car/cdr!)

d

[H Durer]

+1 to the idea of looking forward rather than backward when designing a Lisp.  I believe it is now around 50 years that the names "car" and "cdr" were recognised as a mistake and to this day people are sticking with them as if that was a badge of honor.

Use names that make sense and design functions that suit lfe, don't just parrot what CL is saying.

[Robert Virding]

While the discussion has been interesting it has wandered away from my original intent. This was to define a set of core LFE functions that aren't just the existing Erlang ones. So of my original list probably only eval should remain. So what we have now at this base level is:

- the LFE core forms
- the erlang module of Erlang base functions
- a set of LFE core macros

So my question really was do we need a set of LFE base functions as well? The list I showed was some functions I picked up out the CL manual and probably only 'eval' should remain. But are there others at this level we need? So this is not about library functions but below that.

We should probably make a module 'cl' containing a set of CL standard functions which would include most mine, plus a bunch more. Then we provide them but are are explicit. This could very well be a part of the standard LFE release.

Robert

[Duncan McGreggor]

On Sun, Jun 7, 2015 at 7:00 PM, Robert Virding <rvir...@gmail.com> wrote:

While the discussion has been interesting it has wandered away from my original intent. This was to define a set of core LFE functions that aren't just the existing Erlang ones. So of my original list probably only eval should remain. So what we have now at this base level is:

- the LFE core forms
- the erlang module of Erlang base functions
- a set of LFE core macros

So my question really was do we need a set of LFE base functions as well? The list I showed was some functions I picked up out the CL manual and probably only 'eval' should remain. But are there others at this level we need? So this is not about library functions but below that.

We should probably make a module 'cl' containing a set of CL standard functions which would include most mine, plus a bunch more. Then we provide them but are are explicit. This could very well be a part of the standard LFE release.

That's a *great* idea :-)

d

 

Robert

On Friday, June 5, 2015 at 5:44:14 AM UTC+2, Robert Virding wrote:

Today we can reach all the erlang BIFs from LFE. They are all in the module erlang but many we can reach without prefixing with the module, the same ones as we can from erlang. I plan to create a set of LFE BIFs which will be in the module lfe and which can also be reached without prefixing the module. In one sense these will be part of the LFE language/system in the same way as we have a collection of standard macros. So far I have plucked out a few from the CL book:

acons, pairlis, assoc, assoc-if, assoc-if-not, rassoc, rassoc-if, rassoc-if-not, subst, subst-if, subst-if-not, sublis, eval

I have no intention of adding interface functions to the standard erlang BIFs. This is not a replacement of the libraries in anyway it just defining which functions are part of the LFE "system".

I am open to suggestions for functions that would be reasonable to include here.

Robert

[Duncan McGreggor]

On Sun, Jun 7, 2015 at 7:02 PM, Duncan McGreggor <duncan.m...@gmail.com> wrote:

On Sun, Jun 7, 2015 at 7:00 PM, Robert Virding <rvir...@gmail.com> wrote:

While the discussion has been interesting it has wandered away from my original intent. This was to define a set of core LFE functions that aren't just the existing Erlang ones. So of my original list probably only eval should remain. So what we have now at this base level is:

- the LFE core forms
- the erlang module of Erlang base functions
- a set of LFE core macros

So my question really was do we need a set of LFE base functions as well? The list I showed was some functions I picked up out the CL manual and probably only 'eval' should remain. But are there others at this level we need? So this is not about library functions but below that.

We should probably make a module 'cl' containing a set of CL standard functions which would include most mine, plus a bunch more. Then we provide them but are are explicit. This could very well be a part of the standard LFE release.

That's a *great* idea :-)

Hrm, one more thought about this: it might be nice to 

1) define the functions in an include like lfe/include/cl.lfe, and then

2) include this in a module like lfe/src/cl.lfe

This would allow users to take either approach: treat it like a library, making calls such as (cl:mapcar ...) or treat it like a "built-in" by including it and just calling (mapcar ...).

[Adam Krupicka]

On Monday, June 8, 2015 at 2:00:40 AM UTC+2, Robert Virding wrote:

So my question really was do we need a set of LFE base functions as well? The list I showed was some functions I picked up out the CL manual and probably only 'eval' should remain. But are there others at this level we need? So this is not about library functions but below that.

The issue at hand is that it is necessary to decide where the limit for these built-ins is, exactly.

I don't know if you are familiar with Clojure, but if you look at the Clojure cheat sheet [http://clojure.org/cheatsheet], you can see that Clojure provides many functions out of the box. To me, most interesting are the sections "Seq in, Seq out", "Using a Seq" and "Functions".

So you can see that Clojure provides quite a few built-in functions, and I would argue that it is one of the reasons it's been gaining on popularity. The large amount of standard functions makes it easier to get started with the language; you can jump right in without any need for additional libraries.

Obviously, the use-case of LFE is different, so perhaps it doesn't make that much sense to include functions like those. I don't know. I can only say that developing any kind of AI is greatly aided by the kind of standard functions you find in Clojure - but I understand that this is not the primary concern here.

I wanted to point out that designing a set of standard functions is not a straightforward task. It may look like we're straying off-topic with these debates, but I think the opposite is true.

[Robert Virding]

On Monday, June 8, 2015 at 2:31:08 AM UTC+2, Duncan McGreggor wrote:

On Sun, Jun 7, 2015 at 7:02 PM, Duncan McGreggor <duncan.m...@gmail.com> wrote:

On Sun, Jun 7, 2015 at 7:00 PM, Robert Virding <rvir...@gmail.com> wrote:

While the discussion has been interesting it has wandered away from my original intent. This was to define a set of core LFE functions that aren't just the existing Erlang ones. So of my original list probably only eval should remain. So what we have now at this base level is:

- the LFE core forms
- the erlang module of Erlang base functions
- a set of LFE core macros

So my question really was do we need a set of LFE base functions as well? The list I showed was some functions I picked up out the CL manual and probably only 'eval' should remain. But are there others at this level we need? So this is not about library functions but below that.

We should probably make a module 'cl' containing a set of CL standard functions which would include most mine, plus a bunch more. Then we provide them but are are explicit. This could very well be a part of the standard LFE release.

That's a *great* idea :-)

Hrm, one more thought about this: it might be nice to 

1) define the functions in an include like lfe/include/cl.lfe, and then

2) include this in a module like lfe/src/cl.lfe

This would allow users to take either approach: treat it like a library, making calls such as (cl:mapcar ...) or treat it like a "built-in" by including it and just calling (mapcar ...).

A 'cl' module seems like a good idea, could even include it in the base LFE release. But I am not fond of defining functions in an include file as you then will define many functions in your module which you don't use or even know about. I would prefer to define them in 'cl'. You could, however, have a 'cl' include for defining the standard 'cl' macros.

I am looking at if we could have macros defined in a file and call them like (cl:macro ...) so you don't need the include. This is just speculation for now, it would be feasible but whether it is practical is another question.

Robert

[Robert Virding]

Many of these clojure functions are what in the Erlang world come in modules. Except for the basic forms for handling the built-in data types they are in libraries. Baking them into LFE in the same manner would make it feel different. So even if they come with the standard LFE release I would still keep them in modules.

Robert

[Duncan McGreggor]

And Adam, it would be pretty trivial to put together a third-party library that offered the option of using an include to pull these in for a more Clojure-like experience. Less work than using import, and we could probably do some introspection with beam_lib:chunks so that we wouldn't have to manually maintain the list of module functions.

In other words, everyone can win with macros ;-)

d

[Adam Krupicka]

On Tuesday, June 9, 2015 at 1:53:53 AM UTC+2, Robert Virding wrote:

A 'cl' module seems like a good idea, could even include it in the base LFE release. But I am not fond of defining functions in an include file as you then will define many functions in your module which you don't use or even know about. I would prefer to define them in 'cl'. You could, however, have a 'cl' include for defining the standard 'cl' macros.

I am looking at if we could have macros defined in a file and call them like (cl:macro ...) so you don't need the include. This is just speculation for now, it would be feasible but whether it is practical is another question.

Perhaps it would be best to allow the in-module macro calls and do away with include altogether? Pulling in all the definitions into your module seems like a bad idea. If you include the same include file in multiple modules, do you get multiple definitions of the same functions?

In addition, the import statement could be extended to allow importing the whole module instead, something like (import lists).

As for the Clojure-like functions, I'm all for keeping them in a separate module, especially if you could import them all with a single statement. I suppose a third-party library could work as well.

[Mário Guimarães]

So we will have:

(cl:xpto ...)

(clojure:xpto ...)

(scheme:xpto ...)

(racket:xpto ...)

more ? If all these come with the base install I think it will distract (can't come up with a better word) programmers leading them to think: "Is this Lisp Flavoured Erlang or CL Flavoured Erlang or Clojure Flavoured Erlang?!?!"

I believe that's too much complication!

LFE must stand on itself and pick modern names for the main/core language functions

Whomever wants other names should do it in another public project.

$5c

Mário

--

[Adam Krupicka]

Mário: I agree that LFE should stand on itself, however, all of Erlang stdlib is available out of the box in LFE. So it kind of stands on the shoulders of Erlang already :)

Then again, there is always room for improvement. The Erlang stdlib is certainly not perfect, but it's kind of useless to run rampant defining aliases for core Erlang functions, because we don't like their current names. As an example of what I think is a nice improvement, repeat in lutil/include/core.lfe works as an alias for lists:duplicate - unless you pass in a function, in which case the duplicate elements are constructed by repeated calls to the function (Clojure's repeat and repeatedly in one).

That being said, I think LFE should at least provide function composition and partial application (from lutil's include/compose.lfe), and perhaps even the threading operators. Not sure about other stuff atm.

Another thing, shouldn't LFE include lfex/ltest? Erlang certainly comes with unit testing macros; I see no reason for not including such macros in LFE directly.

Oh, and has anyone looked at rebar3 LFE integration? I want to write a rebar3 LFE plugin some time in the coming weeks, if no one has attempted it already.

You received this message because you are subscribed to a topic in the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lisp-flavoured-erlang/uDt3IX3jnm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lisp-flavoured-e...@googlegroups.com.

[Duncan McGreggor]

On Thu, Jun 11, 2015 at 8:27 AM, Adam Krupicka <krupic...@gmail.com> wrote:

Mário: I agree that LFE should stand on itself, however, all of Erlang stdlib is available out of the box in LFE. So it kind of stands on the shoulders of Erlang already :)

Then again, there is always room for improvement. The Erlang stdlib is certainly not perfect, but it's kind of useless to run rampant defining aliases for core Erlang functions, because we don't like their current names. As an example of what I think is a nice improvement, repeat in lutil/include/core.lfe works as an alias for lists:duplicate - unless you pass in a function, in which case the duplicate elements are constructed by repeated calls to the function (Clojure's repeat and repeatedly in one).

That being said, I think LFE should at least provide function composition and partial application (from lutil's include/compose.lfe), and perhaps even the threading operators. Not sure about other stuff atm.

Another thing, shouldn't LFE include lfex/ltest? Erlang certainly comes with unit testing macros; I see no reason for not including such macros in LFE directly.

Yeah, Robert and I have discussed that in the past. ltest is still not 100%, though (needs more work on some of the fixtures macros and the test runner needs to support more edge cases which it can silently swallow right now).

 

Oh, and has anyone looked at rebar3 LFE integration?

Yes indeed. rebar3 is very nicely done. I really like their plugin architecture. I ported the rebar LFE compilation feature to rebar3 here:

 * https://github.com/oubiwann/rebar3/pull/1/files

See the PR message for more info (and the linked ticket there for the full context of that work and the discussions had with the rebar3 guys).

d

[Mário Guimarães]

Hi Adam,

I don't know if I quite understood your point but here it goes mine:

It would be better for LFE to come already with definitions for the common functions of map, fold, etc, even if those are just aliases for Erlang counterparts, so that we could write (map ...) instead of (lists:map ...) for two reasons:

1st) the code would be more succint and clear

2nd) everyone would use the same aliases instead of their own, thus making the code more consistent across different programmers

Having more high-level functions out-of-the-box, e.g., function composition, would be a plus as well.

And having a good FFI to C/C++ would be greater.

Cheers,

Mário

[Duncan McGreggor]

On Thu, Jun 11, 2015 at 9:09 AM, Mário Guimarães <mario.luis...@gmail.com> wrote:

Hi Adam,

I don't know if I quite understood your point but here it goes mine:

It would be better for LFE to come already with definitions for the common functions of map, fold, etc, even if those are just aliases for Erlang counterparts, so that we could write (map ...) instead of (lists:map ...) for two reasons:

1st) the code would be more succint and clear

2nd) everyone would use the same aliases instead of their own, thus making the code more consistent across different programmers

Having more high-level functions out-of-the-box, e.g., function composition, would be a plus as well.

And having a good FFI to C/C++ would be greater.

You might want to check out these links:

 * http://www.erlang.org/eeps/eep-0007.html

 * http://muvara.org/crs4/erlang/ffi/

 * http://erlang.org/pipermail/erlang-questions/2007-September/029123.html (start of the thread)

Robert has more context on why this wasn't selected and that the NIF approach was chosen instead.

But if you want a general solution for this, you're probably looking to support distributed nodes, so a C node might be what you want to read up on:

 * http://www.erlang.org/doc/tutorial/overview.html#id61317

 * http://www.erlang.org/doc/tutorial/erl_interface.html

I know it looks crazy compared to most FFIs, but it's really not fair to compare them. In fact, it's usually not fair to compare BEAM languages against other languages that aren't both 1) systems programming languages, and 2) designed from the ground-up to support fault-tolerant distributed code.

Those two points made for a very different developer experience in a language, since the goals are so completely different when writing for the distributed world vs. everything else. This is probably the biggest source of cognitive impedance mis-match for devs coming to Erlang from anywhere else.

To belabour the point a bit more: while it would be possible to ignore the requirements of distributed systems in an FFI library, and a good FFI API would really allow developers to get up to speed quickly and get some nice language interop going, all hell would break loose as soon as the first person decided that they'd like to integrate their FFI-based library/solution/etc. within the context of a distributed BEAM application.

That being said, I think there's an interesting middle ground: an LFE library that defines a comprehensive set of ports patterns in a collection of intuitive functions and macros with lots of examples and good documentation (hey, we're dreaming, so let's dream big!). This approach would make full use of ports, taking distribution into account from the start, and would fine-tune the API through extensive use in all the identified usage patterns.

d

[Mário Guimarães]

hhuumm , don't know if I understood your rationale. Let me put this way:

What is the difference between coding a NIF (a real pain) or having a general FFI library that would do much the same that a NIF does? For example, I do not see any difference between a NIF or using an FFI with respect to distribution and fault tolerance ...

MG

[Robert Virding]

On Thursday, June 11, 2015 at 6:57:07 PM UTC+2, Mário Guimarães wrote:

hhuumm , don't know if I understood your rationale. Let me put this way:

What is the difference between coding a NIF (a real pain) or having a general FFI library that would do much the same that a NIF does? For example, I do not see any difference between a NIF or using an FFI with respect to distribution and fault tolerance ...

MG

I think you have to start looking at what you need in a serious FFI:

- The most basic is a layer so you can call C functions from Erlang and send data backwards and forwards.
- You need to be able to be able to test arguments and extract data from them, basically matching on the arguments.
- You need to be able to safely build erlang data structures so you can return data.
- You need support to be able to run in a multi-threaded environment, includes being able to manage threads inside the C functions so they coexist with the BEAM threads.

You need these things a proper FFI, and that is what NIFs provide. The FFI EEP Duncan mentioned just allows sending binaries between Erlang and C. This would work but would be very limiting and lead to using the such an FFI to be more complex and would basically entail moving the complexity. And you would still need some data functions anyway.

Robert

P.S. In one way this is similar to the issue of whether to write simple server processes by hand or use behaviours. Your simple hand-written server quickly become quite complex when you add the functionality you need and you send up implementing most of the behaviour anyway.

[Mário Guimarães]

Robert,

all those requirements are satisfied by NIFs, but a much nicer FFI would exempt programmers from dealing with details like converting data structures between Erlang/LFE and C, as needed when writing NIFs.

It seems that such FFI has never been a priority for Erlang, probably:
1) because most projects where Erlang has been used seldom have that need,
2) or because the way to interface Erlang to other languages should be via distributed message passing across language nodes, not via FFI, which makes perfect sense in the context of a "Let it crash" / fault-tolerant designs (and NIFs are only there just to say "we can call C directly too").

Can you tell me if the case for the lack of a nicer FFI is 1) or 2), or something else?
My guess is 2), and that's fine: it's Erlang's/LFE's way of making its systems resilient to failures.

Cheers

MG

[Robert Virding]

On Sunday, June 14, 2015 at 8:24:10 PM UTC+2, Mário Guimarães wrote:

Robert,

all those requirements are satisfied by NIFs, but a much nicer FFI would exempt programmers from dealing with details like converting data structures between Erlang/LFE and C, as needed when writing NIFs.

How are you then going to pass data between Erlang and C? It has to be done in some way. The internal representations of data inside Erlang are often things you directly want people to access, and they have to be built correctly or you will crash the machine.

It seems that such FFI has never been a priority for Erlang, probably:
1) because most projects where Erlang has been used seldom have that need,
2) or because the way to interface Erlang to other languages should be via distributed message passing across language nodes, not via FFI, which makes perfect sense in the context of a "Let it crash" / fault-tolerant designs (and NIFs are only there just to say "we can call C directly too").

Can you tell me if the case for the lack of a nicer FFI is 1) or 2), or something else?
My guess is 2), and that's fine: it's Erlang's/LFE's way of making its systems resilient to failures.

Sort of 2).  There has always been ports from the very beginning, and we have had C-nodes for a while. When they (finally) got around to doing a proper FFI it was for many purposes: implementing library functions in C, both for OTP and for others[*]; and to provide another way of accessing the outside world. Sometimes a function interface is just better for what you want to do than a message based interface.

 Robert

[*] Originally the only way to make callable c functions was to make BIFs of them, which is why the 'erlang' module is filled with such a variety of stuff (junk). Then you could do special hacks for adding c-functions but it was definitely deep in the BEAM, and not for users. Which NIFs are.

Cheers

MG

2015-06-14 3:43 GMT+01:00 Robert Virding <rvir...@gmail.com>:

On Thursday, June 11, 2015 at 6:57:07 PM UTC+2, Mário Guimarães wrote:

hhuumm , don't know if I understood your rationale. Let me put this way:

What is the difference between coding a NIF (a real pain) or having a general FFI library that would do much the same that a NIF does? For example, I do not see any difference between a NIF or using an FFI with respect to distribution and fault tolerance ...

MG

I think you have to start looking at what you need in a serious FFI:

- The most basic is a layer so you can call C functions from Erlang and send data backwards and forwards.
- You need to be able to be able to test arguments and extract data from them, basically matching on the arguments.
- You need to be able to safely build erlang data structures so you can return data.
- You need support to be able to run in a multi-threaded environment, includes being able to manage threads inside the C functions so they coexist with the BEAM threads.

You need these things a proper FFI, and that is what NIFs provide. The FFI EEP Duncan mentioned just allows sending binaries between Erlang and C. This would work but would be very limiting and lead to using the such an FFI to be more complex and would basically entail moving the complexity. And you would still need some data functions anyway.

Robert

P.S. In one way this is similar to the issue of whether to write simple server processes by hand or use behaviours. Your simple hand-written server quickly become quite complex when you add the functionality you need and you send up implementing most of the behaviour anyway.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Mário Guimarães]

No dia domingo, 14 de junho de 2015, Robert Virding <rvir...@gmail.com> escreveu:

On Sunday, June 14, 2015 at 8:24:10 PM UTC+2, Mário Guimarães wrote:

Robert,

all those requirements are satisfied by NIFs, but a much nicer FFI would exempt programmers from dealing with details like converting data structures between Erlang/LFE and C, as needed when writing NIFs.

How are you then going to pass data between Erlang and C? It has to be done in some way. The internal representations of data inside Erlang are often things you directly want people to access, and they have to be built correctly or you will crash the machine.

Data is passed as done for NIFs, but a nicer FFI would generate the needed conversions from the signature of the C function. There are some efforts in that direction (see https://github.com/parapluu/nifty) but I was expecting this advanced support to come out-of-the-box with Erlang/LFE.

Mário

 

Cheers

MG

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-e...@googlegroups.com.
To post to this group, send email to lisp-flavo...@googlegroups.com.

[Robert Virding]

On Monday, June 15, 2015 at 2:16:44 AM UTC+2, Mário Guimarães wrote:

No dia domingo, 14 de junho de 2015, Robert Virding <rvir...@gmail.com> escreveu:

On Sunday, June 14, 2015 at 8:24:10 PM UTC+2, Mário Guimarães wrote:

Robert,

all those requirements are satisfied by NIFs, but a much nicer FFI would exempt programmers from dealing with details like converting data structures between Erlang/LFE and C, as needed when writing NIFs.

How are you then going to pass data between Erlang and C? It has to be done in some way. The internal representations of data inside Erlang are often things you directly want people to access, and they have to be built correctly or you will crash the machine.

Data is passed as done for NIFs, but a nicer FFI would generate the needed conversions from the signature of the C function. There are some efforts in that direction (see https://github.com/parapluu/nifty) but I was expecting this advanced support to come out-of-the-box with Erlang/LFE.

That would be possible, but you would still have to handle type errors in a sensible way. Just because the C-function has a type signature doesn't mean the caller will abide by it, and to make it feel erlangy it would have be able to handle polymorphic functions. And, most important, I am old-school C programmer so I never use a function signature. :-)

Robert
 

[Robert Virding]

I suppose I am very used to the erlang way of organising function into separate modules instead of having them as "predefined" [*]. So in this respect I would not provide many predefined functions but leave users to use the libraries. It would then be up to us to point out that we have these libraries and that they should use them and not write their own.

I think the naming in many of the libraries is reasonably modern and more "functional" than classic lisp naming of functions. The reason I suggested for adding a module like 'cl' with a set of CL functions was to make it easier for people from that type of background to get into LFE, maybe this was wrong. The reason for explicitly keeping them in separate directories was that this wasn't part of LFE core.

My reason for the 'lfe' directory is that this is where we would have those functions which are part of the LFE core language, like eval. There wouldn't be many of these as many/most of these are already in 'erlang'. For those functions we do put in 'lfe' you wouldn't need to explicitly write the 'lfe' module in the same way as with the 'erlang' module. We would then get a set of core functions and core macros. Which we already have actually.

Robert

[*] The erlang module contains a lot of extra functions because of history. Originally this was the only way to add functions written in C. Now we use NIFs for this.

On Thursday, June 11, 2015 at 4:09:03 PM UTC+2, Mário Guimarães wrote:

Hi Adam,

I don't know if I quite understood your point but here it goes mine:

It would be better for LFE to come already with definitions for the common functions of map, fold, etc, even if those are just aliases for Erlang counterparts, so that we could write (map ...) instead of (lists:map ...) for two reasons:

1st) the code would be more succint and clear

2nd) everyone would use the same aliases instead of their own, thus making the code more consistent across different programmers

Having more high-level functions out-of-the-box, e.g., function composition, would be a plus as well.

And having a good FFI to C/C++ would be greater.

Cheers,

Mário

2015-06-11 14:27 GMT+01:00 Adam Krupicka <krupic...@gmail.com>:

Mário: I agree that LFE should stand on itself, however, all of Erlang stdlib is available out of the box in LFE. So it kind of stands on the shoulders of Erlang already :)

Then again, there is always room for improvement. The Erlang stdlib is certainly not perfect, but it's kind of useless to run rampant defining aliases for core Erlang functions, because we don't like their current names. As an example of what I think is a nice improvement, repeat in lutil/include/core.lfe works as an alias for lists:duplicate - unless you pass in a function, in which case the duplicate elements are constructed by repeated calls to the function (Clojure's repeat and repeatedly in one).

That being said, I think LFE should at least provide function composition and partial application (from lutil's include/compose.lfe), and perhaps even the threading operators. Not sure about other stuff atm.

Another thing, shouldn't LFE include lfex/ltest? Erlang certainly comes with unit testing macros; I see no reason for not including such macros in LFE directly.

Oh, and has anyone looked at rebar3 LFE integration? I want to write a rebar3 LFE plugin some time in the coming weeks, if no one has attempted it already.

On Tue, Jun 9, 2015 at 12:39 PM Mário Guimarães <mario.luis.guimaraes@gmail.com> wrote:

So we will have:

(cl:xpto ...)

(clojure:xpto ...)

(scheme:xpto ...)

(racket:xpto ...)

more ? If all these come with the base install I think it will distract (can't come up with a better word) programmers leading them to think: "Is this Lisp Flavoured Erlang or CL Flavoured Erlang or Clojure Flavoured Erlang?!?!"

I believe that's too much complication!

LFE must stand on itself and pick modern names for the main/core language functions

Whomever wants other names should do it in another public project.

$5c

Mário

2015-06-09 10:11 GMT+01:00 Adam Krupicka <krupic...@gmail.com>:

On Tuesday, June 9, 2015 at 1:53:53 AM UTC+2, Robert Virding wrote:

A 'cl' module seems like a good idea, could even include it in the base LFE release. But I am not fond of defining functions in an include file as you then will define many functions in your module which you don't use or even know about. I would prefer to define them in 'cl'. You could, however, have a 'cl' include for defining the standard 'cl' macros.

I am looking at if we could have macros defined in a file and call them like (cl:macro ...) so you don't need the include. This is just speculation for now, it would be feasible but whether it is practical is another question.

Perhaps it would be best to allow the in-module macro calls and do away with include altogether? Pulling in all the definitions into your module seems like a bad idea. If you include the same include file in multiple modules, do you get multiple definitions of the same functions?

In addition, the import statement could be extended to allow importing the whole module instead, something like (import lists).

As for the Clojure-like functions, I'm all for keeping them in a separate module, especially if you could import them all with a single statement. I suppose a third-party library could work as well.

--

You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lisp-flavoured-erlang/uDt3IX3jnm8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

Visit this group at http://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.