Clojure on an Erlang-vm-os on Xen?

[Frank Siebenlist]

Just became aware of this effort: "http://erlangonxen.org/"

which shows off some impressive properties:

* Startup time of a new instance is 100ms
* Instances are provisioned after the request arrival - all requests get handled
* No instances are running waiting for requests - the cloud footprint is zero
* the size of infrastructure is proportional to the maximum load - 8 servers may be enough
* …

All that begs the Q: would Clojure on an Elang-VM be feasible and make sense?

-FrankS.

[Frank Siebenlist]

Just got this link "http://joxa.org" about a new lisp-like language thru prismatic:

<quote>

Joxa is a small semantically clean, functional lisp. It is a general-purpose language encouraging interactive development and a functional programming style. Joxa runs on the Erlang Virtual Machine. Like other Lisps, Joxa treats code as data and has a full (unhygienic) macro system.

Joxa (pronounced 'jocksah') isn't Erlang, though its very compatible. Its not Clojure though there is plenty of shared syntax. It's not Common Lisp though that is the source of the macro system. While Joxa shares elements of many languages, it is its own specific language. of all these languages, and knowing these languages will help you get up to speed with Joxa, but it is its own unique language.

</quote>

Just skimming the example shows a syntax that resembles clojure a little bit…

Wonder if that implementation as an example would make it easier to have a clojure-on-erlang-vm implementation.

Enjoy, Frank.

[juan.facorro]

Hi Frank,

I'm wondering why no one ever posted a question/comment/reponse to this thread. Personally I think Clojure is a great language and the Erlang VM is a feat of engineering. I would love to think what other think about this thoughts and about the possibility of having Clojure on the Erlang VM in general.

There are some language features that don't quite map from Clojure to Erlang though, which are a challenge to implement. For example it is my understanding that all of Clojure abstractions rely either on Java interfaces (on the JVM) or on protocols (ClojureScript). So I guess a good implementation for the protocol mechanism would be needed for Clojure's implementation on the Erlang VM. 

In my head this means:

1. Values would need to have a type tag, in order to be able to figure out if they do or do not implement a certain protocol.
2. A possible implementation would be a tuple whose first element is an atom which represents the type (i.e. {type_whatever, ...})
3. All functions that support this scheme and also the ones that participate in the protocol mechanism need to "wrap/unwrap" values for "type checking". 
4. Previous points make interop with Erlang more complicated, since Erlang code would need to account for this wrapping & unwrapping. Although there could be a macro that figures some of these things automatically, but it would still not be a seamless Erlang/Clojure interop.

There is also the matter of namespaces and vars, Clojure's bread and butter :P. 

In ClojureScript namespaces are not a thing you can manipulate [1] (you can't use *ns* for example), which could be a possible workaround to avoid maintaining runtime information about namespaces.

Vars on the other hand are not reified in ClojureScript [2] (they map to regular JavaScript variables). Vars either have a root value, are unbound or (when they are dynamic) might have a thread bound value. The first two are global state related to a var while the last one is local to each process (when talking about the Erlang VM). There are ways of implementing this in the Erlang VM, just to name one (which is probably not be the best one) using an ETS for global vars state and the process dictionary for the processes bindings (maybe implementing something analogous to Clojure's frames approach).    

Anyways...

These are just a couple of challenges in a possible implementation on the Erlang VM, which I think are key.

Cheers!

[1] At least not the last time I checked, although ClojureScript is on its way to being self-hosted so, who knows?

[2] The var special symbol provides compile time information though in ClojureScript.

[Fergal Byrne]

Hi Juan/Frank,

Elixir has taken the key features of Clojure (especially macros, protocols) and used them to build a wonderful language on the BEAM. I'd take a look at Elixir if I were you - it's just too much for too little to try shoehorning Clojure notions into a BEAM-hosted system.

That being said, Francesco, CEO of Erlang Solutions was here in Dublin last week, and after our chats he'll hopefully soon be in touch with people at Cognitect about interfacing between Erlang/Elixir and Clojure/Datomic. I see a big opportunity for a ClojureScript/Om front end, a Phoenix/Elixir/Erlang server, and a Datomic persistence layer.

Regards,

Fergal

--

Fergal Byrne, Brenter IT

Author, Real Machine Intelligence with Clortex and NuPIC
https://leanpub.com/realsmartmachines

Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/
and at LambdaJam Chicago, July 2014: http://www.lambdajam.com

http://inbits.com - Better Living through Thoughtful Technology
http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne

e:fergalby...@gmail.com t:+353 83 4214179
Join the quest for Machine Intelligence at http://numenta.org
Formerly of Adnet edi...@adnet.ie http://www.adnet.ie

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Juan Martín]

Hi Fergal,

This is great news! Even more so since I'm currently working at Inaka (a company acquired by Erlang Solutions last year). Also, I've been working in a project that uses ClojureScript/Om in the FrontEnd and Elixir/Plug/Erlang in the backend and it has been great.

Regards,

Juan

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/A6LPAYjmjbY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Juan Facorro

[adrians]

Frank, I've also just found out about Lisp Flavored Erlang, which, based on what I've read, does take some inspiration from Clojure, as well as Common Lisp. It seems to be more active compared to Joxa.

[Sebastian Bensusan]

Hi Frank,

I've been thinking about this for some months now. The actor model is a great fit for a number of applications and the BEAM is made for it.Though it seems feasible, I'm afraid the result might not feel so close to Clojure, at least with any reasonable performance. I hope I'm wrong. I got this *impression* from attending several Erlang meetups[1] where the language developers explain their design decisions and how they are implemented in the BEAM. The conclusion seems to be: it's very hard to deviate from the baked in semantics. The biggest example is that the BEAM comes with a set of Data Types / Data Structures and they can't be extended. On the other hand, porting the actor model to Clojure and the JVM is within reach and Pulsar[2] is already there.

LFE shows some of the trade-offs involved when implementing a Lisp on the BEAM[3]. Robert Virdring has been implementing BEAM languages for years; I recommend anybody interested to watch his talks. He repeats how hard it is to add new semantics to the BEAM. On the other hand, Elixir has done it to some extent. In any case, it would be a *limited subset* of Clojure (much more limited than ClojureScript is).

To get a better answer, I propose to list Clojure's "features" and then iterate through them to see if they can be reasonably implemented or if they even make sense in the platform:

protocols

multi-methods

lazy collections

refs (might make sense on top of mnesia)

atoms (syn

vars

etc.

Hope this helps

Sebastian

[1] https://www.youtube.com/watch?v=fctrWbgbJg0

[2] https://github.com/puniverse/pulsar

[3] https://www.youtube.com/watch?v=BvCBTpnlqs8

[juan.facorro]

Hi Sebastian,

Thank you for your thoughts.

The biggest example is that the BEAM comes with a set of Data Types / Data Structures and they can't be extended.

While this is true, it doesn't mean it is not possible to implement new data structures in Erlang. There are a number of modules that implement additional data structures like array, gb_sets and others, which are mostly implemented using of tree like structures. Admittedly, this is not the same as being able to implement persistent data structure using mutable values, but might be enough for some (or maybe most) practical purposes.

In any case, it would be a *limited subset* of Clojure (much more limited than ClojureScript is).

 

What makes you say this? Do you have a specific Clojure feature in mind that you think might be impossible to implement in the Erlang VM?

To get a better answer, I propose to list Clojure's "features" and then iterate through them to see if they can be reasonably implemented or if they even make sense in the platform:

I think the key phrase is reasonably implemented. I think your suggestion of going over each Clojure feature to consider how it could be implemented is spot-on. I have done this exercise and thought about possible implementations. For me a reasonable implementation is one that has good interoperability capabilities and doesn't add too much overhead because of Clojure language constructs, which is quite a challenge.

What do you think?

Cheers,

Juan

On Friday, September 18, 2015 at 11:06:53 AM UTC-3, Sebastian Bensusan wrote:

Hi Frank,

I've been thinking about this for some months now. The actor model is a great fit for a number of applications and the BEAM is made for it.Though it seems feasible, I'm afraid the result might not feel so close to Clojure, at least with any reasonable performance. I hope I'm wrong. I got this *impression* from attending several Erlang meetups[1] where the language developers explain their design decisions and how they are implemented in the BEAM. The conclusion seems to be: it's very hard to deviate from the baked in semantics. The biggest example is that the BEAM comes with a set of Data Types / Data Structures and they can't be extended. On the other hand, porting the actor model to Clojure and the JVM is within reach and Pulsar[2] is already there.

[Sebastian Bensusan]

Hola Juan,

First, I want to stress that my Erlang skills are very weak and most of what I say is speculation -> add AFAIK to every statement.

1. Data structures, deftype, defrecord

New data structures(deftype, defrecord) can follow the Erlang convention of using records and then naming them with `-type`. Clojure's data structures '(), [], {}, #{}  can naively use erlang:lists, erlang:array, erlang:map/erlang:dict, and erlang:set. The first task here is to check if the performance guarantees and basic operations are maintained. For example, erlang:array s can't be directly compared for equality.

2. Namespaces, symbols, and keywords

It would be desirable to keep a namespace -> module one to one mapping. In Erlang atoms are used both as symbols and keywords and are not namespaced. Namespacing can probably be solved either by a munging mechanism or by using . as described here[1]. For interop, Clojure keywords should probably be represented as plain atoms and symbols as a new type. I'm not sure how this would work.

3. Protocols and multimethods

Erlang doesn't have type based dispatch and the concept of extending an implementation is somewhat supported by `-behavior` and `-callback`. I don't know enough about Erlang's module system to know if that is enough to implement Protocols and multimethods. It is very important that they are extensible which might be hard/impossible in Erlang. I expect somebody with more knowledge to try. Since they would be using the same dispatch mechanism, there would be no performance difference between them.

4. Vars, atoms, refs, and agents

In an async context only agents make sense, in the same way that only atoms make sense in ClojureScript. The problem is that while Clojure allows applying arbitrary functions/closures over the reference, in a message passing context only values can be serialized and used. Therefore the set of actions over the agent would be closed as opposed to Clojure's agents[2]. Closed-agents could be implemented on top mnesia or similar libraries.

5. Strings

Strings can be problematic in Erlang. I would suggest to just copy Elixir's model wholesale[3] and then figure out how well it maps to clojure.string and if the list of characters vs unicode string mismatch leaks out.

With those five groups there is already a lot to figure out. Then there is the problem of integrating necessary Erlang semantics into Clojure:

1. Pattern matching

I don't know if there are any gaps between Erlang's pattern matching and core.match. In any case, it's a good model to follow.

2. PIDs, messaging! and the actor model

Try out syntax to represent ! and receive that works in Lisp and Clojure idioms. I would look at LFE and Pulsar for this.

Once all those points are covered through prototypes/mocks figure out if the effort is worth it. It might as well be that LFE/Pulsar are enough to have a Clojure-like-thing with the desired semantics or Clojure with almost-the-desired-semantics. I don't have a burning need for Clojure on the BEAM and no time to actively pursue it but it does interest me. It would be cool to ask a Google Summer of Code student interested in language design to get a prototype working.

Best

Sebastian

[1] http://stackoverflow.com/questions/10896638/erlang-atoms-and

[2] http://clojure.org/agents

[3] http://elixir-lang.org/docs/v1.0/elixir/String.html

[Wade Moschetti]

How has this progressed. Really interested in using Phoenix/Elixir, Datomic and Om together.

On Friday, September 18, 2015 at 2:06:04 AM UTC+10, Fergal Byrne wrote:

Hi Juan/Frank,

Elixir has taken the key features of Clojure (especially macros, protocols) and used them to build a wonderful language on the BEAM. I'd take a look at Elixir if I were you - it's just too much for too little to try shoehorning Clojure notions into a BEAM-hosted system.

That being said, Francesco, CEO of Erlang Solutions was here in Dublin last week, and after our chats he'll hopefully soon be in touch with people at Cognitect about interfacing between Erlang/Elixir and Clojure/Datomic. I see a big opportunity for a ClojureScript/Om front end, a Phoenix/Elixir/Erlang server, and a Datomic persistence layer.

Regards,

Fergal

--

Fergal Byrne, Brenter IT

Author, Real Machine Intelligence with Clortex and NuPIC
https://leanpub.com/realsmartmachines

Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/
and at LambdaJam Chicago, July 2014: http://www.lambdajam.com

http://inbits.com - Better Living through Thoughtful Technology
http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne

e:fergalb...@gmail.com t:+353 83 4214179

[Juan Martín]

Hi Wade,

There are no roadblocks that I know of for using ClojureScript and Om along with Pheonix/Elixir. 

I haven't tried using Datomic from an Elixir (or Erlang) project though. There is a library called datomex (which you probably already know), although I don't have any expierience using it. It uses Datomic's HTTP REST API, I'm not sure if that presents any limitations.

Cheers,

Juan

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/A6LPAYjmjbY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Juan Facorro

[Wade Moschetti]

Hey Juan,

I have spoken with David about it and he said Om Next is designed to be backend agnostic (I assume Om 'Now' to a degree is like that as well) the only thing is that Transit apparently makes life a LOT easier. I emailed the creator of the datomex library and he said it is unmaintained and there is no plans to maintain it however he will accept pull requests and/or someone taking over the project.

I'm personally at a bit of a crossroads as a new programmer. I actually prefer the Lisp syntax and find Clojure to be a very easy language to reason about. However the community as a whole is still currently aimed more at the intermediate to advanced programmers with nothing outside of Luminus to provide guidance on developing a web application.

I know someone who is using the Phoenix, Om, Datomic stack who has offered to provide me assistance when I need it.. which is great. I was just hoping there was more work in making it a seamless experience being done. 

Thanks for getting back to me
- wade