Large Joxa Refactoring

[Eric Merritt]

Guys,

 After much consideration I think its time to do a big Joxa refactoring. The code we have comes mostly from the Joxa prototype stage. Its convoluted and hard to follow for a number of reasons. I propose we work on cleaning that up. The goals I would like to accomplish are as follows:

### Change the Joxa Target Language

Right now Joxa is targeted at Core Erlang. This works pretty well for a small simple language. However, we loose support for the debugger and dialyzer completely. That really bothers me and makes it harder for folks to get work done. We might also loose some higher level optimizations (I am less sure about that).

I propose we target the Erlang AST. Its a more complex AST and *does* not have an API, but it does not change the compiler much and gives us the above that we are missing.

### Better layout

We have at least moved the most of the code to seperate files, but they are still not factored well. I would like us to take a hard look at whats in Joxa and split it out into more logic modules based on whats there. 

### More functional compiler

Right now we spin off a gen_server on compiler start to handle the context. Then we send messages to that context. This actually makes things more complex rather then simpler. I propose that we change the context to a simple object that gets passed to and returned from each function. It adds an argument, but in the end I think it makes things simpler.

In the first pass the goal is to transition the backend without changing any syntax or semantics. Once we have redone the back end we can start thinking about moving the language forward again. Hopefully, on a much more iteratable platform. 

[Klaus Trainer]

Hi Eric,

your plans sound pretty exciting. The fact that I haven't responded earlier doesn't mean that I'm not highly interested ;)

Right now Joxa is targeted at Core Erlang. This works pretty well for a small simple language. However, we loose support for the debugger and dialyzer completely. That really bothers me and makes it harder for folks to get work done. We might also loose some higher level optimizations (I am less sure about that).

Dialyzer support is the only bigger feature that I'm missing right now. Debugging is a good point, too. Mainly because we can call erl_syntax:revert_forms/1, pass the result to a pretty printer function, and see a program in its plain old Erlang syntax representation with macros being expanded and so on.
 

I propose we target the Erlang AST. Its a more complex AST and *does* not have an API, but it does not change the compiler much and gives us the above that we are missing.

Isn't the erl_syntax module *somehow* an API for the Erlang AST? Most of the interesting tools seem to target Core Erlang, though (e.g. see the modules in the 'lib/hipe/cerl' directory of Erlang/OTP). Anyway, when working with Erlang ASTs we can still get the respective Core Erlang representation at any time by using the `to_core` compiler flag.
 

### Better layout

We have at least moved the most of the code to seperate files, but they are still not factored well. I would like us to take a hard look at whats in Joxa and split it out into more logic modules based on whats there. 

Your previous compiler refactoring was already a big win. Having further improvements hereupon would be awesome.
 

### More functional compiler

Right now we spin off a gen_server on compiler start to handle the context. Then we send messages to that context. This actually makes things more complex rather then simpler. I propose that we change the context to a simple object that gets passed to and returned from each function. It adds an argument, but in the end I think it makes things simpler.

Yeah, exactly. I was wondering about that when trying to understand the compiler. Having a simple context object will probably make it easier to understand.
 

In the first pass the goal is to transition the backend without changing any syntax or semantics. Once we have redone the back end we can start thinking about moving the language forward again. Hopefully, on a much more iteratable platform.

Sounds like a plan. I'd like to offer my help, for instance when it comes to reviewing.

What I'm currently only missing and really need is a function that exposes already existing functionality in the compiler, namely a function that allows me to compile a Joxa AST. However, this can be handled independently. For that matter, I'm going to create a new issue on github and probably send a pull request later.

[Eric Merritt]

On Feb 28, 2013, at 2:30 AM, Klaus Trainer <klaus....@gmail.com> wrote:

Hi Eric,

your plans sound pretty exciting. The fact that I haven't responded earlier doesn't mean that I'm not highly interested ;)

Right now Joxa is targeted at Core Erlang. This works pretty well for a small simple language. However, we loose support for the debugger and dialyzer completely. That really bothers me and makes it harder for folks to get work done. We might also loose some higher level optimizations (I am less sure about that).

Dialyzer support is the only bigger feature that I'm missing right now. Debugging is a good point, too. Mainly because we can call erl_syntax:revert_forms/1, pass the result to a pretty printer function, and see a program in its plain old Erlang syntax representation with macros being expanded and so on.

The other alternative to this approach is that we could fix the compiler infrastructure in erlang to solve these problems once and for all. That is, get the abstract code to be included in some sane way (for core erlang) stop dialyzer from using the abstract code and get specs to be first class parts of the module. Get the debugger to stop using the abstract code and use the source file (if available).

This would be actually a lot more work with the payout being a lot less sure. So I suspect we wont pursue it. However, I wanted to put it out as an option.

 

I propose we target the Erlang AST. Its a more complex AST and *does* not have an API, but it does not change the compiler much and gives us the above that we are missing.

Isn't the erl_syntax module *somehow* an API for the Erlang AST?

Yes, but it is a truly crappy one. 

Most of the interesting tools seem to target Core Erlang, though (e.g. see the modules in the 'lib/hipe/cerl' directory of Erlang/OTP). Anyway, when working with Erlang ASTs we can still get the respective Core Erlang representation at any time by using the `to_core` compiler flag.

yup. exactly. It tends to give us quite a bit more flexibility at the cost of more change/churn underneath us.

 

### Better layout

We have at least moved the most of the code to seperate files, but they are still not factored well. I would like us to take a hard look at whats in Joxa and split it out into more logic modules based on whats there. 

Your previous compiler refactoring was already a big win. Having further improvements hereupon would be awesome.

Yea, this is more a continuation and completion of that work then anything else.

 

### More functional compiler

Right now we spin off a gen_server on compiler start to handle the context. Then we send messages to that context. This actually makes things more complex rather then simpler. I propose that we change the context to a simple object that gets passed to and returned from each function. It adds an argument, but in the end I think it makes things simpler.

Yeah, exactly. I was wondering about that when trying to understand the compiler. Having a simple context object will probably make it easier to understand.

It was originally like this actually. It was a pain making sure that the context returned/passed around effectively. I thought this would reduce some of that, but alas it does not. It actually makes things more annoying, but it different ways. So I want to go back there.

 

In the first pass the goal is to transition the backend without changing any syntax or semantics. Once we have redone the back end we can start thinking about moving the language forward again. Hopefully, on a much more iteratable platform.

Sounds like a plan. I'd like to offer my help, for instance when it comes to reviewing.

What I'm currently only missing and really need is a function that exposes already existing functionality in the compiler, namely a function that allows me to compile a Joxa AST. However, this can be handled independently. For that matter, I'm going to create a new issue on github and probably send a pull request later.

Agreed. I would like all of that to be surfaced out. But its something that can be done along side of these changes. That is, these approaches would not block each other.

Eric

--
You received this message because you are subscribed to the Google Groups "joxa" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joxa+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Klaus Trainer]

On Thursday, February 28, 2013 11:12:54 PM UTC+1, Eric Merritt wrote:

The other alternative to this approach is that we could fix the compiler infrastructure in erlang to solve these problems once and for all. That is, get the abstract code to be included in some sane way (for core erlang) stop dialyzer from using the abstract code and get specs to be first class parts of the module. Get the debugger to stop using the abstract code and use the source file (if available).

This would be actually a lot more work with the payout being a lot less sure. So I suspect we wont pursue it. However, I wanted to put it out as an option.

Thanks, I haven't thought about that! Although I wouldn't like to go that direction right now, it's a good point worth mentioning. The nice thing with it is that it would improve the Erlang/OTP platform as a whole.