Sharing of thoughts

[Dexter]

Hi everyone, 

I'm making slow progress on the compiler. Because it is not all that clear for me where to do what. 

Therefore I want to outline the structure of the new compiler design and go into some specifics as well.

The basic usage of the compiler should be (in api form). 

Instantiate a reader object, which receives python sourcecode. 

The reader will transform the sourcecode into ist. And store that ist into a collection (such that multiple files can be handled)

The collection will be given to a set of Transformer objects, which will transform the ist in several steps.

Finally the collection will be given to a Reader object, which will be able to write out the several ist trees.

In the reader step, I still see some issues. 

 - I was thinking to enable the Reader to recognize import statements, and allow them to fetch those files, and read those in turn. But this requires the Reader to receive some sort of python_path list.

 - How to name the files, in such a way they can be imported. The modules need a valid name to leak trough to the javascript code.

The transformer step is the most difficult. And should be done as correct as possible. 

For now, it consists of 3 transformers (the reader does do some simple transformation as well. Mainly for generating valid IST( != AST))

the __ifyTransformer (is there a better name for this transformer?)

 > basically transforming every operator (or most) into __ methods (the easiest step) (thus removing most operators)

the TypeifyTransformer

 > adding all typed functions to literals. (removing all literal operators)

the SemantifyTransformer

 > Transforming classes into function like constructs > (removing classes)

 > Transforming functions in such a way that *args, *kwargs, defaults, keywordarguments are removed.

 > Error handling things.

 > namespacing (getting variables from the right namespace, storing them in the right namespace)

 > modules (will the module node be removed here? will it just transform it here, or will modules stay the same)

 > probably lots more to come here.

Maybe the SemantifyTransformer will become to big. Namespaceing is quite a big deal and might require its own transformer (it should also handle the boxing/unboxing probably) , as do classes.. 

After all these steps, I think we are only left with a handful of ist nodes. which is awesome.

This has become quite a lengthy email. But I would really like some feedback, do I miss things. We do have written down some broad picture. But there are still a lot of things into the air.. 

Greetings, Dexter. 

PS, I've got some basics down on the javascript writer, and some tests do work (its all in https://github.com/neoel/pyjaco/tree/devel/pyjaco.new)