LFE is not just a programming languge

[Duncan McGreggor]

That thread on Twitter was really sad to see. Setting aside the rather disturbing emotional content and just focusing on a key issue, I think it's important to clarify for *everyone* that Erlang, and thus LFE, are not just "regular" programming languages.

Erlang can't really be compared to other programming languages. People get fed up with it when they try to use it, not only because of the syntax, but because there seems to be all sorts of boilerplate and "weird" tasks that one has to perform, often perceived as needless overhead. There is a lot of temptation to hide this behind convenience functions, macros, etc.

The thing is, Erlang is a language of distributed, communicating Erlang processes running as part of Erlang systems. The traditional concepts in general programming very often can't simply be applied. As Robert has tried to explain many times, this is why there is no gensym in LFE. How does one define gensym in the context of distributed processes? Define it in such a way that it is useful in all the contexts of the language?

So that is the overriding view which every decision about the language is made. Questions like "Does it hide or obfuscate core Erlang concepts or capabilities? Will if cause developers to misinterpret the language or distributed systems? Will it lead them down incorrect conceptual paths? Will it lead to incorrect, fatally flawed software?

I think this approach to a distributed language design is not only reasonable and desirable, but necessary. If one needs gensym and one is not writing distributed systems, or will only use gensym locally (yikes! now you have to be EXTREMELY careful about how you deploy your code and which code publishes to other nodes, etc. HERE THERE BE DRAGONS), then you can write one. I think I've seen two different implementations for gensym in LFE. Hell, make a library that offers this (but there better be a whole slew of DANGER warnings in the README ...)

The desire to make LFE more like Clojure, while I do understand the love that many have for Clojure's syntax (you're talking to the guy that originally wrote the clj library), Clojure doesn't actually have an answer for syntax + distributed systems. 

Ages ago Rich Hickey talked about the possibility of supporting distributed nodes in Clojure at the language level (even pointing to the possibility of using JInterface from Erlang) ... but you'll note that he still hasn't done this. It's a tricky problem, and to make it actually part of the language, he'd have to change many, many things. Essentially creating a new language. 

Akka and Quasar/Pulse are good examples of what wiring for distributed notes in Clojure is like ... each provides a very different approach, but use of each in Clojure diverges pretty far from standard Clojure idioms (and even sytnax). Pulsar gets really close to LFE, actually ... seeing how much they were inspired by (and directly copied) Erlang (up to and including behaviours), this makes sense. The fact that, even in Clojure, Pulsar is verbose should give some hints to the complexity of the underlying technical hurdles that have to be overcome and can't be hidden with sugar.

Anyway, the point with all that is that even while trying to support distributed nodes and all the complexity, configuration, options, contingencies, failure conditions, ability to respond to these, etc., in Clojure, you end up with a whole mess of new stuff ... that is really not pretty (e.g., not simple and elegant like standard Clojure). And that's what you get when you decide to take a bite out of the programming for communicating, distributed nodes. You get a whole new world. And a whole new type of programming. Syntax that makes it pretty is necessarily going to be hiding many things that will eventually need to be uncovered and used directly by the programmer. You don't want to set up a world view for new programmers that they are going to have to completely tear down in order to build things that the language was really designed for.

This is one of the reasons that I argue in favour of banishing all "Hello World" BEAM examples that don't involve genserver and supervision trees. That way, from the very start, all new BEAM devs would be faced with the reality of the situation: BEAM is for distributed systems. All that "boilerplate" actually means something, and will change depending upon the type of nodes/systems you are building.

I'm sorry that some of you hate that LFE is not Clojure. Unlike Robert, I really do like Clojure :-) But I don't want LFE to be Clojure -- I really love it the way it is (modulo the occasional miscellaneous feature addition/minor tweak). I find it an absolute joy to write in.

Fun story: I was asked to join a company a few years back, but they still wanted to do the full interview/hire process with me. They did a technical live coding interview for part of it, where the candidate can choose any language. I chose LFE and they were both Clojure programmers. It was the single most AMAZING and fun interview I've ever had (in fact, one of the all-round best programming experiences of my life, period). For an hour, I live coded LFE, solving all sorts of problems for them (and very quickly -- we got more done than they usually saw). At the end, they had TONS of questions for me. They were *fascinated* by LFE. They'd never seen anybody write in it, much less write so quickly and solve problems in such a concise manner. I did not try to emulate Clojure -- I simply used LFE as it is, with its own strengths and features. Obviously, a complete joy for me to write in -- but even more importantly, when used to best effect, something that seasoned Clojure programmers could not only appreciate, but be amazed by.

d

[mi...@forsterfamily.ca]

On Thursday, December 15, 2016 at 8:55:06 AM UTC-6, Duncan McGreggor wrote:

That thread on Twitter was really sad to see. Setting aside the rather disturbing emotional content and just focusing on a key issue, I think it's important to clarify for *everyone* that Erlang, and thus LFE, are not just "regular" programming languages.

[...]

Very well said.

Mike

[Mário Guimarães]

Hi,

can someone give a clear explanation of the problem of gensym in the context of distributed programming / LFE ?

I tried to find a comprehensive discussion but failed.

Thanks

Mário

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.
Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[Robert Virding]

There are 2 main problems with gensym:

- There is no way to make an atom that is guaranteed to be unique and not equal to any other atom. The underlying Erlang/BEAM just don't support this.

- All atoms are global and interned in an atom table and once created they are never removed from the atom table. This means that a program which dynamically creates atoms will eventually fill the atom table which crashes the BEAM. Unavoidably.

One solution to solve the first problem would be to use UUIDs as atom names which would make them unique but eventually crash the system when the atom table fills up. Another suggested solution is for every file reuse the same sequence of atoms every time which would solve the second problem but may create duplicates.

That is the problem with gensym.

Note that dynamically creating atoms and reusing the same names will not fill the atom table, it is the total number of different atoms which is the key issue.

Hope this helps,

Robert

[Mário Guimarães]

Robert,

very clear about the atoms stuff.

But aren't gensyms mostly (only?) needed to generate unique variable names in a module, to avoid unintended variable capture, before final compilation takes place? At least this is what I understood after reading "Ansi Common Lisp" and "OnLisp".

Perhaps I am not getting something ...

Mário

2016-12-22 12:53 GMT+00:00 Robert Virding <rvir...@gmail.com>:

There are 2 main problems with gensym:

- There is no way to make an atom that is guaranteed to be unique and not equal to any other atom. The underlying Erlang/BEAM just don't support this.

- All atoms are global and interned in an atom table and once created they are never removed from the atom table. This means that a program which dynamically creates atoms will eventually fill the atom table which crashes the BEAM. Unavoidably.

One solution to solve the first problem would be to use UUIDs as atom names which would make them unique but eventually crash the system when the atom table fills up. Another suggested solution is for every file reuse the same sequence of atoms every time which would solve the second problem but may create duplicates.

That is the problem with gensym.

Note that dynamically creating atoms and reusing the same names will not fill the atom table, it is the total number of different atoms which is the key issue.

Hope this helps,

Robert

On Thursday, 22 December 2016 13:14:18 UTC+1, Mário Guimarães wrote:

Hi,

can someone give a clear explanation of the problem of gensym in the context of distributed programming / LFE ?

I tried to find a comprehensive discussion but failed.

Thanks

Mário

2016-12-20 23:16 GMT+00:00 <mi...@forsterfamily.ca>:

On Thursday, December 15, 2016 at 8:55:06 AM UTC-6, Duncan McGreggor wrote:

That thread on Twitter was really sad to see. Setting aside the rather disturbing emotional content and just focusing on a key issue, I think it's important to clarify for *everyone* that Erlang, and thus LFE, are not just "regular" programming languages.

[...]

Very well said.

Mike

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsubscri...@googlegroups.com.

To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.
Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[James Laver]

Gensyms are generally useful whenever something needs to be named anonymously. The model I was working with before I gave up trying to get gensyms into LFE was that you'd have a counter per top level form in a module that would be incremented every gensym in it. To reuse the gensym counters, you just go onto the next expression. The downside of this is that you can't use gensym to declare function names in your modules, but why would you want that anyway?

/j

[Robert Virding]

The trouble with that method is that it is not safe and breaks down as soon as you need to pass over each form more than once. Which the compiler actually does now. The first pass to split the file into its separate modules containing the right module, macro and function definitions, and the second pass to expand each module separately.  You could do it in one pass but it becomes much more complex.

[Robert Virding]

If you are running a system for any length of time where you are compiling new modules or just using the REPL and calling macros then you would be calling gensym all the time. This would eventually overflow the atom table and crash the system.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.
To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.
Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

[Fred Hebert]

On 12/23, Robert Virding wrote:
>The trouble with that method is that it is not safe and breaks down as soon
>as you need to pass over each form more than once. Which the compiler
>actually does now. The first pass to split the file into its separate
>modules containing the right module, macro and function definitions, and
>the second pass to expand each module separately. You could do it in one
>pass but it becomes much more complex.
>

If you know the number of modules ahead of time, you can still use
counters such as '$SymVarN' where `N' is allocated based on an initial
counter and a step that is equal to the number of modules in order to
get a bunch of free reuse in names.

For 1 module, start=1, step=1:

N = 1,2,3,4,...

For 3 modules, start=1,2,3, step=3 (each module can carry its initial
params for this)

A = 1,4,7,10,...
B = 2,5,8,11,...
C = 3,6,9,12,...

If you need to do a first general pass that you want to make sure has no
conflict, consider the first pass as either a 4th module or as a thing
that already increments the step? I'm not familiar with the internals so
I don't know if that actually works, but you could therefore have
either:

First = 1,2,3,4
A = 5,8,11,14,...
B = 6,9,12,15,...
C = 7,10,13,16,...

Or =

First/Shared = 1,5,9,13,...
A = 2,6,10,14,...
B = 3,7,11,15,...
C = 3,8,12,16,...

As long as you do not have the ability to create modules dynamically as
part of the second pass, you can use that counter trick keep reusing the
same scheme for all modules no matter what and get more variable reuse
that way. You do get edge cases for files containing thousands of
modules each using thousands of gensyms but aside form that this should
be pretty safe?

Then again, I have no idea how that stuff is implemented!

[Mário Guimarães]

I am sorry but this module discussion seems unrelated to my question ...

Robert,

What I am really not understanding is why do you need atoms for gensyms ... 🤔

Why just don't you generate a "special string" X-SOME-UUID, instead of an atom, when calling (gensym)? Why do you need atoms for this? This "special string" type would be something for LFE consumption only, that is, it would be used to put the name X-SOME-UUID somewhere during macro expansion at a variable's place.

Or there are some early implementation decisions that require the use of atoms and make this unfeasible?

If gensym is not working/implemented/implementable, how can you avoid variable capture today in LFE? Example? (no example in tutorial).

Thanks

Mário

[Duncan McGreggor]

On Fri, Dec 23, 2016 at 11:47 AM, Mário Guimarães <mario.luis...@gmail.com> wrote:

I am sorry but this module discussion seems unrelated to my question ...

Well, it's a fairly ingenious work-around for the multie-pass compilation and creating gensyms ... it would be useful for you if you wanted to write a safe(r) gensym function ...

 

Robert,

What I am really not understanding is why do you need atoms for gensyms ... 🤔

So, you know that modules and functions are named with atoms, right? E.g. (call 'mod 'function ...). So, if you have a function generated in a macro and you're using a gensym implementation to create a "guaranteed unique" (in quotes, because once you cross the node boundary and are executing code remotely, this becomes something less easily determinate) function, it will eventually have to be converted to an atom ...
 

Why just don't you generate a "special string" X-SOME-UUID, instead of an atom, when calling (gensym)? Why do you need atoms for this? This "special string" type would be something for LFE consumption only, that is, it would be used to put the name X-SOME-UUID somewhere during macro expansion at a variable's place.

Or there are some early implementation decisions that require the use of atoms and make this unfeasible?

If gensym is not working/implemented/implementable, how can you avoid variable capture today in LFE? Example? (no example in tutorial).

This is a great question -- and should be added (with answer) to the LFE tutorial.

d

 

Thanks

Mário

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

[Mário Guimarães]

ok, functions are named as atoms, and atoms are limited at runtime, that's the way of Erlang, and for now (or for ever) there is no escape to that, and LFE is just a Lisp skin for the Erlang VM.

I think many of the questions in this mailing list would be avoided if we had a full and illustrated reference for LFE. My feeling when I look to the existing documentation is that it is probably outdated, and as important it is to keep coding on LFE, I think it is also important to commit updated documentation along the code (this macro creating functions stuff and the variable capture stuff are just two use cases worth mention).

It would also be nice to have documentation clearly showing what is basic LFE, i.e., what is the Lisp basic skin to Erlang,
and what is there in LFE that goes beyond it (maybe a two-part LFE reference / tutorial).

Also, I am not clear how much of today's Erlang is skinned by LFE ...

Again, we need more "gas" from Erlang Solutions on LFE. Robert, any good news this Christmas?

Duncan, thanks again for your clarifications.

Cheers,

Mário

PS: It would be nice to have some good reference to the Erlang VM internals, a spec. ref. like Java has, where these and other stuff would be specified in detail, but it seems there is none yet, besides the promise of a book [1] which I have been waiting for a long long time.

[1] https://www.amazon.com/Erlang-Run-Time-System-Erik-Stenman/dp/1449362125/ref=sr_1_17?s=books&ie=UTF8&qid=1482537585&sr=1-17&keywords=erlang

2016-12-23 21:14 GMT+00:00 Duncan McGreggor <dun...@mcgreggor.org>:

[Sasha Fonseca]

@Mário Guimarães, AFAIK that book has been canceled :( http://support.oreilly.com/oreilly/topics/is-the-book-the-erlang-run-time-system-by-stenmans-cancelled

I concur that the Erlang VM's specs are scarse or often scattered through a lot of disperse documents.

 

Thanks

Mário

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

To post to this group, send email to lisp-flavo...@googlegroups.com.

Visit this group at https://groups.google.com/group/lisp-flavoured-erlang.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

To post to this group, send email to lisp-flavo...@googlegroups.com.

[Claudia Doppioslash]

It would seem that Pragprog picked it up, and it should come out in 2017: https://pragprog.com/book/eserlang/the-erlang-runtime-system

On 26 December 2016 at 14:35, Sasha Fonseca <sash...@gmail.com> wrote:

@Mário Guimarães, AFAIK that book has been canceled :( http://support.oreilly.com/oreilly/topics/is-the-book-the-erlang-run-time-system-by-stenmans-cancelled

I concur that the Erlang VM's specs are scarse or often scattered through a lot of disperse documents.

--

Claudia Doppioslash

@doppioslash

http://www.doppioslash.com

http://www.lambdacat.com   | Functional Programming

http://www.shadercat.com    | Shader & Graphics Programming

[Robert Virding]

There have been a number of presentations by Lukas Larsson and Erik Stenman at Erlang factories and user conferences describing some of the internals of the BEAM. Lukas has also written some blogs about the internals as well. These are very informative, but still not a book.

Robert

[Mário Guimarães]

Hi Robert,

nice to ear. Can you send us the links to that cool stuff?

Thanks

Mário

--

You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.

To post to this group, send email to lisp-flavoured-erlang@googlegroups.com.

[Mário Guimarães]

(in case you have them at your fingers, otherwise we'll search ...)

2016-12-29 16:26 GMT+00:00 Mário Guimarães <mario.luis...@gmail.com>:

Hi Robert,

nice to ear. Can you send us the links to that cool stuff?

Thanks

Mário

2016-12-29 16:22 GMT+00:00 Robert Virding <rvir...@gmail.com>:

There have been a number of presentations by Lukas Larsson and Erik Stenman at Erlang factories and user conferences describing some of the internals of the BEAM. Lukas has also written some blogs about the internals as well. These are very informative, but still not a book.

Robert

On Monday, 26 December 2016 15:47:30 UTC+1, Claudia Doppioslash wrote:

It would seem that Pragprog picked it up, and it should come out in 2017: https://pragprog.com/book/eserlang/the-erlang-runtime-system

On 26 December 2016 at 14:35, Sasha Fonseca <sash...@gmail.com> wrote:

@Mário Guimarães, AFAIK that book has been canceled :( http://support.oreilly.com/oreilly/topics/is-the-book-the-erlang-run-time-system-by-stenmans-cancelled

I concur that the Erlang VM's specs are scarse or often scattered through a lot of disperse documents.

--

Claudia Doppioslash

@doppioslash

http://www.doppioslash.com

http://www.lambdacat.com   | Functional Programming

http://www.shadercat.com    | Shader & Graphics Programming

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsubscri...@googlegroups.com.

[Robert Virding]

Here are some off the top of my keyboard:

http://www.erlang-factory.com/conference/SFBay2012/speakers/LukasLarsson
http://www.erlang-factory.com/euc2014/lukas-larsson
https://www.erlang-solutions.com/blog/erlang-19-0-garbage-collector.html
https://www.youtube.com/watch?v=tBAM_N9qPno
https://www.youtube.com/watch?v=QbzH0L_0pxI
https://www.youtube.com/watch?v=RcyN2yS5PRU
https://vimeo.com/44231138
https://vimeo.com/185351875

(no specific order)

Robert

On Thursday, 29 December 2016 17:27:54 UTC+1, Mário Guimarães wrote:

(in case you have them at your fingers, otherwise we'll search ...)

2016-12-29 16:26 GMT+00:00 Mário Guimarães <mario.luis.guimaraes@gmail.com>:

Hi Robert,

nice to ear. Can you send us the links to that cool stuff?

Thanks

Mário

2016-12-29 16:22 GMT+00:00 Robert Virding <rvir...@gmail.com>:

There have been a number of presentations by Lukas Larsson and Erik Stenman at Erlang factories and user conferences describing some of the internals of the BEAM. Lukas has also written some blogs about the internals as well. These are very informative, but still not a book.

Robert

On Monday, 26 December 2016 15:47:30 UTC+1, Claudia Doppioslash wrote:

It would seem that Pragprog picked it up, and it should come out in 2017: https://pragprog.com/book/eserlang/the-erlang-runtime-system

On 26 December 2016 at 14:35, Sasha Fonseca <sash...@gmail.com> wrote:

@Mário Guimarães, AFAIK that book has been canceled :( http://support.oreilly.com/oreilly/topics/is-the-book-the-erlang-run-time-system-by-stenmans-cancelled

I concur that the Erlang VM's specs are scarse or often scattered through a lot of disperse documents.

--

Claudia Doppioslash

@doppioslash

http://www.doppioslash.com

http://www.lambdacat.com   | Functional Programming

http://www.shadercat.com    | Shader & Graphics Programming

--
You received this message because you are subscribed to the Google Groups "Lisp Flavoured Erlang" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lisp-flavoured-erlang+unsub...@googlegroups.com.