Frege REPL

[Marimuthu Madasamy]

Hi,

A very basic Frege REPL is ready :) Here is a sample REPL session:

frege> "Hello World"

res0 = "Hello World"

frege> res0 ++ ", Frege"

res1 = "Hello World, Frege"

frege> :def add x y = x + y

frege> add 3 4

res3 = 7

frege> :t add

res4 :: Num α => α -> α -> α

frege> :t add 5

res5 :: Int -> Int

frege> res5 8

res6 = 13

frege> :def pure native parseInt java.lang.Integer.parseInt :: String -> Int

frege> parseInt "34322"

res8 = 34322

frege> :def import Data.List

frege> :def fib = 1 : 1 : zipWith (+) fib (tail fib)

frege> take 10 fib

res11 = [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Few things to note:

1) The expressions are compiled and then evaluated. The Java files and the generated class files are in memory.

2) Due to lack of parsers, the expressions and other definitions need to be distinguished. The ":def " serves for this purpose as we can see above. The expressions can be typed in directly, all the other declarations (non-show elements) are prefixed by ":def". When we roll out a parser to detect the expressions, we can remove this redundant ":def ".

3) ":t " will display the type of an expression

It is still at very basic level. I am currently working on ":paste" mode (as in Scala REPL) as well as ":load" to load scripts.

Your suggestions and comments are welcome.

Regards,

Marimuthu

[Daniel]

That is awesome! Two questions: Do you support multi-line definitions?
And can you define data types and type classes in the REPL as well?

Thank you!
Daniel

> distinguished. The *":def "* serves for this purpose as we can see above.

> The expressions can be typed in directly, all the other declarations
> (non-show elements) are prefixed by ":def". When we roll out a parser to
> detect the expressions, we can remove this redundant ":def ".
>

> 3) *":t "* will display the type of an expression

[Ingo W.]

Sounds great!

Looking forward to play around with this.

One question: Couldn't we use "let" in place of "def:"

[Marimuthu Madasamy]

Thanks Daniel.

 

Do you support multi-line definitions? 

Yes, I am working on this. It will be something like (:paste mode in Scala),

frege>:paste

--you can enter multiple lines and when you press CTRL-D, all the definitions will be evaluated at once

can you define data types and type classes in the REPL as well? 

Yes, we can define data types and type classes in the REPL.

[Marimuthu Madasamy]

Thanks.

 

One question: Couldn't we use "let" in place of "def:"

Sure, "let" is more natural. I will make this change. 

[James Earl Douglas]

This is so cool!

[Marimuthu Madasamy]

Hi,

I have attached a basic version of REPL. You can extract it into some folder and run it from the folder (with fregec.jar in the same folder and JRE 7 in path) using:

java -cp ./FregeInterpreter.jar:./jline-1.0.jar:./FregeScriptCompiler.jar:./fregec.jar frege.interpreter.FregeScriptExecutor

Sample REPL session showing the functionalities and the usage:

$java -cp ./FregeInterpreter.jar:./jline-1.0.jar:./FregeScriptCompiler.jar:./frege3.19.112a.jar frege.interpreter.FregeScriptExecutor

Welcome to Frege 3.19.112 (Oracle Corporation Java HotSpot(TM) 64-Bit Server VM, 1.7.0_02)

frege> "Hello World"

res0 = Hello World

frege> :t res0

res1 :: String

frege> let pure native parseInt Integer.parseInt :: String -> Int

frege> show $ parseInt "25"

res3 = 25

frege> :t parseInt "234"

res4 :: Int

frege> :paste

Entering paste mode...

Type :q to exit paste mode

data YesNo = Yes | No

:q

Interpreting...

frege> :paste

Entering paste mode...

Type :q to exit paste mode

instance Show YesNo where

  show Yes = "Yes"

  show No = "No"

sayHello who = "Hello, " ++ who

:q

Interpreting...

frege> let x = Yes

frege> show x

res6 = Yes

frege> :doio println $ sayHello "Foo"

Hello, Foo

frege> :t sayHello

res8 :: String -> String

frege> :q

Things to improve:

1) Loading scripts

2) ability to import modules (Now imports are useful only in :paste mode)

3) better error handling

4) Any other functionalities or improvements?

[James Earl Douglas]

Awesome!

2012/4/23 Marimuthu Madasamy <mmhell...@gmail.com>

[Ingo W.]

Congratulations, Marimuthu!!!

My first session:

frege> let pythagorean = [ (x,y,m*m+n*n) | m <- 2..100, n <- 1 .. m-1, x = m*m-n*n, y=2*m*n ]

frege> show $ take 10 pythagorean

res6 = [(3, 4, 5), (8, 6, 10), (5, 12, 13), (15, 8, 17), (12, 16, 20), (7, 24, 25), (24, 10, 26), (21, 20, 29),

(16, 30, 34), (9, 40, 41)]

frege> all (\(a,b,c) -> c*c == a*a+b*b) $ take 10 pythagorean

1 Error(s):

E :12: type error in

    expression res7

    type is   Bool

    used as   String

frege> show $ all (\(a,b,c) -> c*c == a*a+b*b) $ take 10 pythagorean

res8 = true

I am sure there will come up lots of questions later, but for today lets just enjoy and let's celebrate Marimuthu!

I am going down and open a bottle of Chianti I reserved for special occasions like this!

(This is definitly a shortcoming of the internet ... you can share photos and whatnot, but no wine.)

[Marimuthu Madasamy]

I am sure there will come up lots of questions later

I would like to hear them all and also about the good-to-have features. The REPL is not very robust at the moment. I am trying to eliminate the explicit call to "show" and "doio" to perform IO and I am thinking of writing a parser (using Scala's parser combinator) to parse modules, imports, top-level definitions and expressions individually.

This is definitly a shortcoming of the internet 

maybe online transfer ;)

[Ingo W.]

Am Mittwoch, 25. April 2012 03:41:32 UTC+2 schrieb Marimuthu Madasamy:

I am sure there will come up lots of questions later

I would like to hear them all and also about the good-to-have features.

Yep. First one, when will it be ready for distribution? :)
Seriously, have you thought about integration with the rest of the system? Ideally, we only have 1 jar with standard libraries and tools.
In addition, I'd like to make use of mostly stable code for integration into fregIDE.
 

The REPL is not very robust at the moment.

I checked what happens in the following case: show $ iterate (1+) 1
I see the java process gobble up memory in the ressource monitor, and then, surprise, it came up with the prompt again.
Next try was:
let ones = 1 : ones
show $ length ones
I couldn't stop the process with ^C from the command line (in Windows 7).
So it's actually more robust than expected, in some sense.
 

I am trying to eliminate the explicit call to "show" and "doio" to perform IO

Yes, this is a must.
Should be not too hard to find out if the result type is an instance of Show or if it is and IO action.

 

and I am thinking of writing a parser (using Scala's parser combinator) to parse modules, imports, top-level definitions and expressions individually.

This is where I am a bit sceptical. I oppose having different parsers for compiler and interpreter on the ground that either we end up with 2 different languages or we get into maintenance hell trying to keep 2 parsers identical.
Hence I see the following options:
- adapt the current parser to satisfy your needs
- switch to a new parser in both compiler and interpreter.
But the second option must not result in longer parsing times. Speed is crucial here, i.e. for GUI feedback, etc.

[Marimuthu Madasamy]

Seriously, have you thought about integration with the rest of the system? Ideally, we only have 1 jar with standard libraries and tools.
In addition, I'd like to make use of mostly stable code for integration into fregIDE.

I think it is not ready for that level. I'd like to do some more unit testing and have some peer code review.

This is where I am a bit sceptical. I oppose having different parsers for compiler and interpreter on the ground that either we end up with 2 different languages or we get into maintenance hell trying to keep 2 parsers identical. 

To determine whether the script is a top-level definition, an expression or an import, currently the implementation is very naive. So I thought I could use a parser. Reusing the same parser as the compiler is the ideal solution. I looked at Grammar.y but I am not sure how to proceed with this. Any pointers on that direction would be great.

[Ingo W.]

Am Donnerstag, 26. April 2012 06:03:22 UTC+2 schrieb Marimuthu Madasamy:

To determine whether the script is a top-level definition, an expression or an import, currently the implementation is very naive. So I thought I could use a parser. Reusing the same parser as the compiler is the ideal solution. I looked at Grammar.y but I am not sure how to proceed with this. Any pointers on that direction would be great.

It's basically a YACC grammar, just with frege code instead of C code for the semantic actions. 

If you don't want to dive into this, you tell me what you need.

Currently, the grammar is like this:

    start: packageclause WHERE '{' definitions '}'

We could do something like

   start: packageclause WHERE '{' definitions '}'  { .... }
          | INTERPRETER interpreterstuff  { const }
   interpreterstuff:
          definition             { ReplDef }
          | expression        {\_\e -> ReplEx e}
          | command1        { .... }     
          | ....
          | commandN       { .... }

where INTERPRETER would be a pseudo keyword (token) that you'd need to prepend to the token list returned by the scanner.
We would need an appropriate datatype for the different parse results (currently, it's just a tuple with packagename and list of definitions). So for instance

data ParseResult = PackageForCompiler String [Definitions]
          | ReplDef Definition
          | ReplEx Expr
          | ...

To give you an idea how it works:

The code in the { ... } are semantic actions that determine the value associated with each rule. Every nonterminal (which names a single rule or a set of alternative rules) constructs a value of a certain type, while terminals (keywords, identifiers, literals, etc.) have just their Token as values (the token, in turn, contains the string the token was made of).
The rule value is actually computed by passing the values of the parsed items on the left to the associated semantic action, hence the latter must be an appropriate function. So for example, if the parser sees at the start an INTERPRETER token, it remembers this and tries to parse interpreterstuff, which has many alternative subrules. Suppose the parser finds a definition (of type Definition). Then it will apply the semantic action ReplDef to that Definition value. Because in our example ReplDef is a constructor that takes a Definition, this results in a value of ParseResult. The parser has now a Token (INTERPRETER) and a ParseResult on the stack and applies semantic action const to it (2nd alternative of start: rule), which results in the same ParseResult and this would be the overall parse result.

         

[Marimuthu Madasamy]

Thanks a lot Ingo. That is really helpful and very interesting. I will try to come up with something based on this.

[Ingo W.]

Is it possible to have unicode characters on both input and output?

frege> :t const

res10 :: ? -> ? -> ?

(The type pretty printing code wants to write greek letters here.)

But I found also that I can enter only ASCII characters, hence when I type:

show "mäßig"

it appears like

frege> show "mig"

res12 = "mig"

(This is on Windows 7 in cmd.exe with Unicode code page.)

Am Dienstag, 24. April 2012 06:31:33 UTC+2 schrieb Marimuthu Madasamy:

[Marimuthu Madasamy]

In Ubuntu, the unicode characters are displayed correctly. Java's System.out uses default encoding of the underlying platform. The following code prints the unicode character but along with another special character:

final String test = "\u03B1";

final PrintStream out = new PrintStream(System.out, true, "UTF8");

out.println(test);

displays,

α�

[Ingo W.]

Ok, then we have some platform compatibility problem here.

I have no idea how the line editing part is working, but in any case we need to recognize this in the "final polishing". Also, the line editing is working very slowly in a Windows command line window.

Looks like it is a good idea to keep the interpreter code as separated as possible from the frontend, so that one day we can have

- interpreter on the command line

- interpreter within eclipse

- interpreter with a simple gui (swing)

- interpreter with a web frontend, something like http://tryhaskell.org/

- ...

[Marimuthu Madasamy]

Hello all,

Here is the new Frege REPL: http://dl.dropbox.com/u/55737110/FregeReplDist/fregerepl.zip . It has the following improvements:

1) I have updated Frege language grammar to support interpreter. So now we can type all the expressions and declarations directly in the REPL as they are in a complete frege program without any additional annotations for the REPL.

2) As Ingo noted, the interpreter code is now completely independent of the front end. This is achieved through the new Frege Scripting Engine (JSR 223). So now we can use Frege as a scripting language inside a Java program,

final ScriptEngineManager mgr = new ScriptEngineManager();

final ScriptEngine engine = mgr.getEngineByExtension("fr"); //For JavaScript engine, we would specify the extension as "JavaScript"

engine.eval("filter even (1..20)");

Frege Script Engine internally uses a Frege compiler and a Java compiler which compile the scripts and the generated class files will be in memory. The REPL uses Frege Script Engine to execute scripts.

3) Now we can load definitions from a file using the ":load" command. (Package declaration is not supported)

Ex: If there is a file "testscript.fr"  with the following contents:

pure native parseInt Integer.parseInt :: String -> Int

import frege.IO

now = currentTimeMillis ()

We can load it in the REPL with,

frege> :load testscript.fr

All the definitions in the file will be loaded. In addition to that, the imports in the file will be preserved for the rest of the REPL session which is true for the :paste command as well.

4) All IO actions are executed and the result will be printed.

Example:

frege> import frege.IO 

frege> currentTimeMillis () 

res1 = 1338964419152

frege>  

5) The results will be displayed by invoking 'show' implicitly. If the results are not instances of 'Show', it will be an error but functions will be displayed with their type information.

For example:

frege> data YesNo = Yes | No 

frege> answer = Yes 

frege> answer 

E show res7:1: YesNo is not an instance of Show

E show res7:1: YesNo is not an instance of Show

frege> derive Show YesNo 

frege> answer

res10 = Yes

frege> const 

res12 :: α -> β -> α

frege>  

6) To make sure that the scripting engine is pluggable to different UIs, I have also developed a GUI REPL with my little Java Swing experience. The GUI REPL is also available in the attached zip file. Please see below to launch both console and GUI repls.

7) Following commands are supported:

:type <expr>      - To print the type of an expression

:paste            - To input multiple definitions at once

:load <File Path> - To load definitions from a file

:help             - 

Commands can be abbreviated (:t, :ty, :type all mean type command to print the type of an expression)

 

How to run?

1) Extract the attached zip file into a folder

2) In the terminal, switch to the folder and 

type the line below to launch console REPL (Make sure that the Java Compiler is available in the path)

java -jar fregerepl.jar

For GUI REPL, just pass in some argument

java -jar fregerepl.jar gui

In GUI REPL, ctrl - s -> to save all the REPL contents into the clipboard and ctrl-c, ctrl-v - to copy selection and paste respectively.

Please find attached a sample REPL (GUI) Session.

Regards,

Marimuthu

[Ingo W.]

This is so great, Marimuthu!

Luckily, tomorrow is a holiday in germany, so I'll hopefully find some time to try this out.

Am Mittwoch, 4. April 2012 10:07:47 UTC+2 schrieb Marimuthu Madasamy:

[Ingo W.]

Hi, it works great.

One thing I noted, though on empty input:

Welcome to Frege 3.19.124 (Oracle Corporation Java HotSpot(TM) Client VM, 1.7.0)

frege>       

Exception in thread "AWT-EventQueue-0" frege.RT$Undefined: Prelude.head []

at frege.prelude.PreludeBase.error(PreludeBase.java:16137)

at frege.prelude.PreludeList$IListLike__lbrack_rbrack.head(PreludeList.java:1271)

at frege.interpreter.FregeScriptCompiler$3.eval(FregeScriptCompiler.java:1216)

at frege.rt.Lam1$1._v(Lam1.java:71)

at frege.rt.Unknown._e(Unknown.java:384)

at frege.rt.Lambda.app(Lambda.java:60)

 .      ...

[Marimuthu Madasamy]

Thanks Ingo. Fixed.

[Travis Wellman]

Any intention to publish the source?

[Marimuthu Madasamy]

yes, I am testing out few things in online REPL. After that, I will be publishing the source.

[skg]

Hi Marimuthu,

 

can you please provide the link for the latest REPL jar file, I tried using parseInt function and got error "E parseInt "12234" :l:can't resolve `parseInt', did you mean `Ord_Int` perhaps"

 

may be I am using a old version, so it would be great if you upload a new version. This interface is a great way for novice like me to learn frege, so your help is highly appreciated.

 

Thanks & Regards,

Sudeep

[Ingo W.]

Hi Sudeep, sorry for intervening.

I am afraid your error has not much to do with a specific version of the REPL.

I rather suspect an error in your code.

The best place to resolve this would be to create a question on stackoverflow. Be sure to tag it with "frege"

[Ingo W.]

In addition, you might want to try this: "42".int

This should give you Right 42

[Marimuthu Madasamy]

Hi Sudeep,

The error is because there is no built-in function called 'parseInt'. You can define it like this:

pure native parseInt Integer.parseInt :: String -> Int

The REPL session which I have shown above has this function loaded through an external script file using ':load' command. 

You could also use built-in function 'int' as Ingo mentioned.

Thanks Ingo for the 'int' function.

Regards,

Marimuthu Madasamy

--
 
 

[Sudeep]

Ingo, Marimuthu, Thanks a lot for the clarification. In future, I will search and quote such query in stackoverflow only, apologies for my ignorance.

 

Warm regards,

Sudeep

 

PS: I am just starting in frege (and functional programming), and it has been good fun playing with both the eclipse IDE and REPL. However if you know of any help file/ documentation other than the language manual (Language-609.pdf) on code.google.. would anyone please direct me to it. Thanks!

[Ingo W.]

Am Dienstag, 14. August 2012 08:42:12 UTC+2 schrieb Sudeep:

 

PS: I am just starting in frege (and functional programming), and it has been good fun playing with both the eclipse IDE and REPL. However if you know of any help file/ documentation other than the language manual (Language-609.pdf) on code.google.. would anyone please direct me to it. Thanks!

Sounds funny, but you could walk through "Learn you a Haskell for great good."

Would be interesting to learn how this would have to be adapted or what is missing if one uses Marimuthus REPL instead of the Haskell ghci.

[Marimuthu Madasamy]

Hello all,

I have setup an online Frege REPL - "tryfrege" here. It is running Frege 3.19.112 and still in active development.

I would like to know whether we have any plans / ideas to host Frege tools online. I could see that we have created a domain for Frege http://www.frege-lang.org/. Are there any plans to host other web applications on this domain? 

[Ingo W.]

Is it just me?
On Firefox 16.0.1 I always get "Oops, REPL is down", while at the same time it works in IE and Chrome.

[Marimuthu Madasamy]

Have you tried in that Firefox instance before? It might be returning the cached results. I am planning for an REPL release today with the latest Frege compiler. I will try to handle this with that.

Sent from my iPhone

--
 
 

[Ingo W.]

I am not sure.
Hence I just restarted Firefox and now it works.

Long live Chrome! But google-software is forbidden where I work :-(

[Marimuthu Madasamy]

Yes, restart would do. I will also try to disable the cache.

And online REPL is now running latest Frege Compiler frege3.20.23.jar.

[Ingo W.]

Not sure if I understand the architecture correctly, but could this be a web app hosted, for example, as Google Web App (https://developers.google.com/appengine/)?

Am Mittwoch, 17. Oktober 2012 08:38:32 UTC+2 schrieb Marimuthu Madasamy:

[Marimuthu Madasamy]

Having had a quick look on Google App Engine, I guess we should deploy our applications as war and it runs Java 6 VM. I see two issues with this. Please correct me if I am wrong.

1) Currently the 'tryfrege' web application makes a call to another JVM to actually execute the code. So there must be a standalone application which accepts the code and executes it. Since the REPL is running on a seperate JVM (sandbox), in case the code doesn't terminate after a certain time, we could always terminate and restart the REPL. I am not sure whether it is possible with GAE.

2) The GAE Java page says "App Engine runs Java applications using the Java 6 virtual machine (JVM)". But Frege is compiled to Java 7 and I think Java 7 is not currently supported on GAE.

[Emanuele Ziglioli]

On Sunday, 21 October 2012 12:26:20 UTC+13, Marimuthu Madasamy wrote:

Having had a quick look on Google App Engine, I guess we should deploy our applications as war and it runs Java 6 VM. I see two issues with this. Please correct me if I am wrong.

1) Currently the 'tryfrege' web application makes a call to another JVM to actually execute the code. So there must be a standalone application which accepts the code and executes it. Since the REPL is running on a seperate JVM (sandbox), in case the code doesn't terminate after a certain time, we could always terminate and restart the REPL. I am not sure whether it is possible with GAE.

Instances are created on demand, a JVM spins up, it takes a few seconds. Then after some 30 seconds of inactivity it spins down.

If your app doesn't take long for initializing, that model would fit on a GAE free quota. There's something else called Backends you could use but I'm not sure it's available for free apps.

 

2) The GAE Java page says "App Engine runs Java applications using the Java 6 virtual machine (JVM)". But Frege is compiled to Java 7 and I think Java 7 is not currently supported on GAE.

It's being phased it with the latest SDK (in beta). Currently beta testers (invite only) can run a Java 7 runtime but not yet the general public

[Ingo W.]

Somehow a post of mine got lost, so I'll repeat me here.

2) The GAE Java page says "App Engine runs Java applications using the Java 6 virtual machine (JVM)". But Frege is compiled to Java 7 and I think Java 7 is not currently supported on GAE.

This is not a big issue since

a) the only code that currently really needs java7 is the ForkJoin.fr, but it isn't used in the compiler. Also, maybe something in the runtime that I used in the method handle experiments. Bottom line is that we can setup things for java6 in a few minutes.

b) as Emanuele points out, it'll probably not be too long until java7 is generally used even in the cloud. I am somehow surprised that they're still at java6.

Concerning the 2nd JVM, I can imagine it is not needed, because of this (from https://developers.google.com/appengine/docs/whatisgoogleappengine):

For example, when an application is called to serve a web request, it must issue a response within 60 seconds. If the application takes too long, the process is terminated and the server returns an error code to the user. The request timeout is dynamic, and may be shortened if a request handler reaches its timeout frequently to conserve resources.

 

[Marimuthu Madasamy]

Thanks Emanuele and Ingo. I'll learn more about it and try it out. I have just created an app id 'tryfrege'.

[Marimuthu Madasamy]

After having setup everything, I finally found out that the Java Compiler API (javax.tools.ToolProvider) is restricted in GAE and couldn't proceed further since the frege code is first compiled with fregec and then javac. Any other options or workaround?

Nevertheless it was actually fun playing around with app engine and building the frege compiler for Java 6.

Here are some notes which might be useful for those wanting to build the frege compiler for Java 6:

1) In Makefile, we need to set the compiler to Java 6 and comment out the lines referring ForkJoin.fr, ForkJoin.class, MH.class, FregeCompiler.class

2) In frege/compiler/JavaUtils.java, we need to remove references to `Files` class and implement slurp method. The following one-liner could help:

return new Scanner(new File(filename), encoding).useDelimiter("\\Z").next();

3) In frege/RT.java, Comment out the fork method and in the fjMain method, the use of ForkJoin needs to be removed and hence checking the 'parallel' option is unnecessary (can be removed).

4) In frege/StandardLibrary.fr, the import Lib.ForkJoin needs to be removed.

[Emanuele Ziglioli]

On Monday, 22 October 2012 15:10:05 UTC+13, Marimuthu Madasamy wrote:

After having setup everything, I finally found out that the Java Compiler API (javax.tools.ToolProvider) is restricted in GAE and couldn't proceed further since the frege code is first compiled with fregec and then javac. Any other options or workaround?

well, that sucks big time. You could always file a bug request to have those classes whitelisted but I suspect it'd be a feature only requested by selected fews.

One option could be to just grab those sourcea  and repackage them into your app and try to remove any dependency from non-whitelisted classes.

Mind that there's no File IO API on GAE, and that could be a big show stopper. You could mimic that with writes to memcache or to the datastore, or the blobstore (write only).

[Emanuele Ziglioli]

Shameless plug. If you're looking for the nicest Datastore API on GAE, look no further:

http://www.sienaproject.com/index.html

https://github.com/mandubian/siena

Thank you,

Emanuele

https://github.com/Sirtrack

[Ingo W.]

Am Montag, 22. Oktober 2012 09:01:32 UTC+2 schrieb Emanuele Ziglioli:

On Monday, 22 October 2012 15:10:05 UTC+13, Marimuthu Madasamy wrote:

After having setup everything, I finally found out that the Java Compiler API (javax.tools.ToolProvider) is restricted in GAE and couldn't proceed further since the frege code is first compiled with fregec and then javac. Any other options or workaround?

well, that sucks big time. You could always file a bug request to have those classes whitelisted but I suspect it'd be a feature only requested by selected fews.

One option could be to just grab those sourcea  and repackage them into your app and try to remove any dependency from non-whitelisted classes.

You probably couldn't do this for every line that is typed in the REPL.
 

Mind that there's no File IO API on GAE, and that could be a big show stopper. You could mimic that with writes to memcache or to the datastore, or the blobstore (write only).

File I/O is not really needed as everything happens in-memory, as far as I understod Marimuthu.
 

[Marimuthu Madasamy]

Here it is: "tryfrege" on Google App Engine http://tryfrege.appspot.com/ 

The Java Compiler API problem on GAE is solved by using Eclipse JDT Compiler. The other problem was that while building Frege compiler for Java 6, the Java 6 compiler was throwing the syntax error as mentioned in this post by Ingo http://stackoverflow.com/questions/5404750/javac-strange-syntax-error-illegal-start-of-expression. So I used Java 7 compiler and set the source and target level to 1.6 for compilation. On GAE, compilation happens in-memory so the application is not persisting anything other than the user scripts on the session. 

[Emanuele Ziglioli]

This is really impressive, congratulations! I'd suggest you to post a message to the GAE group, I'm sure they'd be interested in knowing about it and how you solved various problems: https://groups.google.com/forum/?fromgroups=#!forum/google-appengine

So, the JDT compiler implements the same API as the Java Compiler API but is only uses whitelisted classes, is that right?

Have you discovered what classes where not passing the GAE validation?

[Emanuele Ziglioli]

This opens a lot of possibilities for App Engine Java developers.

So far there hasn't been a REPL for Java runtimes. One could run Python shells either using Clojure or by deploying a non-default Python version (http://bit.ly/SgGkEp). Another possibility was to use the so-called 'remote-api' or Groovy/Gaelyc.

A REPL could help editing the datastore, given the limitations of the Web Console. And perhaps it could be used for monitoring other instances.

[Marimuthu Madasamy]

This is really impressive, congratulations!

Thanks!

 

 So, the JDT compiler implements the same API as the Java Compiler API but is only uses whitelisted classes, is that right?
Have you discovered what classes where not passing the GAE validation?

 

The latest JDT compiler implements Java Compiler API but again we can't use that in GAE since no classes from javax.tools package are whitelisted. So I am using version 3.7 JDT compiler which doesn't implement Java Compiler API.

 

A REPL could help editing the datastore, given the limitations of the Web Console. And perhaps it could be used for monitoring other instances.

GAE is not that much restricted as I initially thought, given the nature of REPL (dynamic class loading, reflection etc.). So it should be possible to extend this to other application areas that can fit in the sandbox model.

[Emanuele Ziglioli]

 

The latest JDT compiler implements Java Compiler API but again we can't use that in GAE since no classes from javax.tools package are whitelisted. So I am using version 3.7 JDT compiler which doesn't implement Java Compiler API.

I see ;-)

It all looks pretty responsive. Are you on a free quota or have you paid to keep one instance warm?

No idea how big JDT 3.7 is, have you done some appstats?

GAE is not that much restricted as I initially thought, given the nature of REPL (dynamic class loading, reflection etc.). So it should be possible to extend this to other application areas that can fit in the sandbox model.

They do tend to remove restrictions over time but some things are only possible in a rather awkward fashion, such as Thread Factories.

For other APIs, such as Datastore, Task Queues etc there's just no standard way of doing it. For example each NoSQL dbs offers a different API. 

I wish GAE promoted a more asynchronous way of doing things. For example, accessing the datastore via Futures is not the way to go IMHO.

[Marimuthu Madasamy]

It all looks pretty responsive. Are you on a free quota or have you paid to keep one instance warm?

I am on free quota. Actually it could be still better responsive if we don't have to serialize the objects on session. If we can keep any objects, the compiler parsed definitions can be stored on the session. Since those definitions are not serializable, currently the source scripts are stored in session and parsed every time.

No idea how big JDT 3.7 is, have you done some appstats?

I am still a GAE beginner :) yet to explore appstats.

[Emanuele Ziglioli]

On Wednesday, 24 October 2012 16:20:03 UTC+13, Marimuthu Madasamy wrote:

It all looks pretty responsive. Are you on a free quota or have you paid to keep one instance warm?

I am on free quota. Actually it could be still better responsive if we don't have to serialize the objects on session. If we can keep any objects, the compiler parsed definitions can be stored on the session. Since those definitions are not serializable, currently the source scripts are stored in session and parsed every time.

It looks like a good case for using memcache. As far as I know, GAE servlet sessions are written to datastore. Memcache access is way faster and cheap

 

[Ingo W.]

It is astonishing how you do such things almost instantly!
Great work!

Am Dienstag, 23. Oktober 2012 22:19:19 UTC+2 schrieb Marimuthu Madasamy:

Here it is: "tryfrege" on Google App Engine http://tryfrege.appspot.com/ 

The Java Compiler API problem on GAE is solved by using Eclipse JDT Compiler. The other problem was that while building Frege compiler for Java 6, the Java 6 compiler was throwing the syntax error as mentioned in this post by Ingo http://stackoverflow.com/questions/5404750/javac-strange-syntax-error-illegal-start-of-expression.

Didn't they still fix that yet!!!!
Do you remember which file it was where the error occured? Maybe I can put some parenthesis in so that it'll finally be possible to use a bare JDK 1.6. 

Regarding responsiveness, it looks to me like it ran slightly faster on mmhelloworld.

[Ingo W.]

Am Mittwoch, 24. Oktober 2012 05:20:03 UTC+2 schrieb Marimuthu Madasamy:

It all looks pretty responsive. Are you on a free quota or have you paid to keep one instance warm?

I am on free quota. Actually it could be still better responsive if we don't have to serialize the objects on session. If we can keep any objects, the compiler parsed definitions can be stored on the session. Since those definitions are not serializable, currently the source scripts are stored in session and parsed every time.

Do you think it is a good idea to make algebraic data types Serializable? Would it be a big deal? 

[Yo Eight]

You've done a great job. Just one point, it's a shame we have an automatic tab switch on hover.

Can't wait to see the source code.

Cheers

[Marimuthu Madasamy]

Do you think it is a good idea to make algebraic data types Serializable? Would it be a big deal?

I think that it would be useful in some areas. In this particular instance, I wanted to store the instances of `DefinitionT` (not sure this can be eligible for serialization) so that whenever a new script is submitted, we can just compare the definitions in the current script with the existing and eliminate the overridden definitions and just run the rest of the definitions as opposed to recompiling the entire history from source to bring back the last REPL state.

[Marimuthu Madasamy]

It is astonishing how you do such things almost instantly!
Great work!

Thanks Ingo! It is all really fun and exciting!! :)

Didn't they still fix that yet!!!!

It looks so! By the way, I am using OpenJDK.

 

Do you remember which file it was where the error occured? Maybe I can put some parenthesis in so that it'll finally be possible to use a bare JDK 1.6.

The error occurred in the Frege generated Java source file for PreludeBase.fr.

Here is the snippet from the file PreludeBase.java. The error was on line no 19084, at position after `Box.`:

return Box.<java.lang.Exception>mk(
                  (java.lang.Exception)(Box.<frege.RT.Undefined>box(arg$1._e()).j)
                );

I think we need to make changes in Java code generation to make Java 6 compiler happy.

Regarding responsiveness, it looks to me like it ran slightly faster on mmhelloworld.

This is because in this domain, the REPL state (in the form of parsed Definitions) is stored in the session and hence there is only one compilation with all the definitions combined at a time.

[Marimuthu Madasamy]

You've done a great job. Just one point, it's a shame we have an automatic tab switch on hover.

Thanks! I have removed the 'hover' event so now it opens only on clicking the tabs. 

[Ingo W.]

Am Donnerstag, 25. Oktober 2012 04:08:33 UTC+2 schrieb Marimuthu Madasamy:

Do you think it is a good idea to make algebraic data types Serializable? Would it be a big deal?

I think that it would be useful in some areas. In this particular instance, I wanted to store the instances of `DefinitionT` (not sure this can be eligible for serialization) so that whenever a new script is submitted, we can just compare the definitions in the current script with the existing and eliminate the overridden definitions and just run the rest of the definitions as opposed to recompiling the entire history from source to bring back the last REPL state.

As long as a value does not contain functions or non-serializable java objects, I guess it could be done. Yet, of course, all lazy values would have to be evaluated in the process of serialization. And, of course, there is the danger that somewhere an infinite sub-value would exist.

Not sure how Java Serialization/Deserialization copes with sharing, i.e. 

foo = map show (1..1000)

bar = (foo, tail foo)

Here we have potentially 1000  string objects in foo, and the two lists in bar share 999 of them along with the actual list nodes.

On serialization, foo and bar would have to be fully evaluated.

But how would it look at de-serialization? Would the two lists in bar still share the tail of foo?

[Yo Eight]

Stream doesn't implement Storable in Haskell, in that sense, it can't be marshalled (though I'm pretty sure alternative solutions exist).The biggest challenge is to handle infinite sub value but this leads us to data and codata distinction which very few compilers can handle.

[Ingo W.]

Why, Yorrik, infinite values just take a bit longer to serialize. :)

Seriously, this would of course change semantics. While it is perfectly valid to have undefined values lingering around in your data structures, this is not so with to-be-serialized data. Those data would have to work as if in a 100% strict language.

This, of course, appears as no good news from a theoretical point of view. Yet, in practice, one can imagine that it would be ok to do for some data for some purposes.

[Ingo W.]

Do you need the Definitions, or do you really need the symbol tables from the last Global state?

The latter can be reconstructed from the class file if you have that somewhere. This is a spezialized form of de-serialization (so to speak) which I made as fast as possible in order to minimize the overhead of import definitions. see Import.fr

Am Donnerstag, 25. Oktober 2012 04:08:33 UTC+2 schrieb Marimuthu Madasamy:

[Marimuthu Madasamy]

Excellent! Import.fr is all what we need. So we can serialize the bytecodes and then from loading the class, we can reimport everything. I will give it a try today.

Sent from my iPhone

--
 
 

[Ingo W.]

One caveat, though: most expressions are not recovered (nor are they generally recoverable). You get the types, classes, instances, etc., but not the expressions. 

[Ingo W.]

return Box.<java.lang.Exception>mk(
                  (java.lang.Exception)(Box.<frege.RT.Undefined>box(arg$1._e()).j)
                );

Any suggestions on how to write this perfectly valid code differently? This is a call of a static, generic method box, followed by member access j, and the result is casted to java.lang.Exception. 

The () around the expression that gets casted may not be necessary in this case (I am always unsure about that), but the (java.lang.Exception) is actually the specification for a native method, and the template for this is to generate (targetType)(J[x]), where J[x] is the java equivalent of some frege expression x. It could also make (double)(2+2) from (2+2).double and the () make sure that whatever the argument is, the cast operation will cover all of it and not emit things like (double)2 + 2

The error is:

build\frege\prelude\PreludeBase.java:18994: illegal start of expression
                  (java.lang.Exception)(Box.<frege.RT.Undefined>box(arg$1._e()).j)
                                            ^

 

[Marimuthu Madasamy]

It also implies that we need to store and import all the classes from the previous submissions, right? In that case, this is like our previous approach which we dropped because the expressions once bound are not reevaluated.

[Ingo W.]

Am Freitag, 26. Oktober 2012 02:40:37 UTC+2 schrieb Marimuthu Madasamy:

It also implies that we need to store and import all the classes from the previous submissions, right? In that case, this is like our previous approach which we dropped because the expressions once bound are not reevaluated.

Yes, this seems unfortunately  to be the case.

I must say that I like the current approach more.

Perhaps we should stop and think about it again. Is it really the case, that re-parsing a short script should be that time consuming (several seconds)? We are doing this in the eclipse IDE on almost every keystroke and yet this starts to really hurt only if one has several 100 lines.

I am not familiar with the architecture of the GAE, but is it possible that a new JVM is started for every line I type? For, if not, the compiler code should get JIT-ted more and more and should get faster over time.

[Marimuthu Madasamy]

Is it really the case, that re-parsing a short script should be that time consuming (several seconds)?

It is because If we are recompiling from source, it is not just fregec, javac also must be invoked hence the whole process takes time. In AWS instance, the last successful submission is stored in form of fregec parsed definitions so the next time, we don't need the sources from history, instead we can just include those definitions along with the definitions from the current script.

is it possible that a new JVM is started for every line I type?

I am also not sure but it looks like to be the case at least for most of the times.

Would it be a good idea or is it possible to serialize the compiler parsed definitions(DefinitionT instances) after a successful fregec compilation?

[Ingo W.]

Am Samstag, 27. Oktober 2012 19:37:00 UTC+2 schrieb Marimuthu Madasamy:

Is it really the case, that re-parsing a short script should be that time consuming (several seconds)?

It is because If we are recompiling from source, it is not just fregec, javac also must be invoked hence the whole process takes time. In AWS instance, the last successful submission is stored in form of fregec parsed definitions so the next time, we don't need the sources from history, instead we can just include those definitions along with the definitions from the current script.

I don't know if you use fregIDE.

Here, with every keystroke (except when the pause between 2 keystrokes is less than some preconfigured time) all compiler steps from lexical analysis up to type checking are done for the whole program text in the editor buffer.

Once the frege compiler code is "warm", the lexer and parser pass together take somewhere less than 1ms per line.

The most expensive pass is still "symbol table initialization and import", which takes about 200ms.

Overall, we achieve times of about lines*10 ms till completion of typechecking.

(I have a PC with i3 dual core processor, I guess google has faster ones.)

is it possible that a new JVM is started for every line I type?

I am also not sure but it looks like to be the case at least for most of the times.

I strongly suspect this, meanwhile. Given that my data have to cross the ocean probably anyway, I'd guess that network latency is about the same (when comparing  mmhelloworld and google).

Yet, Google is consistently slower, as far as my experience goes, and I am talking about several seconds here.

A new JVM for every request would be a consistent explanation for this, as the start up costs of frege are naturally rather high.

Would it be a good idea or is it possible to serialize the compiler parsed definitions(DefinitionT instances) after a successful fregec compilation?

In the light of where the time costly operations in the compiler are (see above), I guess Serialization would be even slower. Or at least would not save much, as they could save only the time spent in lexer and parser, and this is almost nothing in relation to the rest.

For example, on recompilation of the compiler on the command line there is this big Grammar.fr, that takes more than 2min to fully compile. The lexer and parser account for less than 7s here. Hence, 5%, give or take some fractions.

Given that to get an answer from Google takes 3 to 4s (for me), we could save no more than non-noticeable 15 to 20/100 seconds, if at all, because Java Serialization also has its cost AFAIK.

[Marimuthu Madasamy]

I don't know if you use fregIDE.
Here, with every keystroke (except when the pause between 2 keystrokes is less than some preconfigured time) all compiler steps from lexical analysis up to type checking are done for the whole program text in the editor buffer.

I am using fregIDE. I figured out that the only difference is invoking javac on every script which is not really necessary. So I changed 'tryfrege' to invoke 'javac' only when we need the class file.

For example, on recompilation of the compiler on the command line there is this big Grammar.fr, that takes more than 2min to fully compile. The lexer and parser account for less than 7s here. Hence, 5%, give or take some fractions.
Given that to get an answer from Google takes 3 to 4s (for me), we could save no more than non-noticeable 15 to 20/100 seconds, if at all, because Java Serialization also has its cost AFAIK.

Yes, you are right. I found out that the multiple invocations of fregec and javac for each script in the history were affecting the performance. Now on every run, to restore the history, only fregec is invoked, that too just once with all the source from the history. It looks little better now.

I have also added a new command ":h" to list the history of scripts compiled successfully so far.

[Marimuthu Madasamy]

The source code is now available on github: https://github.com/Frege. There are 3 projects:

frege-scripting: This is the main project which has an in-memory Frege compiler and Java compiler (Eclipse JDT) to interpret Frege source code snippets.

try-frege: This is the web application currently deployed in Google App Engine on http://tryfrege.appspot.com/. This project does not do much of its own apart from handling web requests and responses. It just passes the code snippets to 'frege-scripting' and collects the response.

frege-repl: This is like 'try-frege' but a standalone application that can be used as console REPL. It is also using 'frege-scripting' to evaluate frege code.

All these projects are currently targeted for Java 6 because of Google App Engine restriction. So we would need Frege compiler for Java 6 which is now available on https://github.com/Frege/frege/downloads. If not for try-frege on GAE, frege-scripting and frege-repl can be targeted for Java 7. Once GAE supports Java 7, we can drop support for Java 6.

[Ingo W.]

Great many thanks, Marimuthu!

Maybe you can explain in the README how to use frege-scriptig/frege-repl with the latest compiler code if you find a bit spare time.

[Yo Eight]

Thanks a lot !

[Marimuthu Madasamy]

I have updated README with a section on how to run the REPL. Latest Frege compiler, 3.20.48, and the other jars to run the REPL are now available on the Downloads page.

With respect to changes, earlier in REPL, the type annotation was not working. Now this issue has been fixed. For example,

frege> foo = one

frege> one
res0 :: Num α => α
frege> foo :: Double
res1 = 1.0

Another change is that now the type command ":t" doesn't bind the expression to a variable. It just prints the type of an expression.

frege> :t foo
Num α => α

[Ingo W.]

Yes, I already noticed this, and its much better this way.

I guess the right repository for bug reports is frege/scripting?

[Marimuthu Madasamy]

I guess the right repository for bug reports is frege/scripting?

scripting or repl both should be fine for bug reports. 

[Marimuthu Madasamy]

Hello,

Now in REPL, we can load and reload code snippets from a file like:

frege> :l Foo.fr

frege> :r

":l" and ":r" are the commands to load and unload script files respectively. Earlier ":l" was used to list the identifiers and ":r" was used to reset the REPL session but now those are renamed to ":list" and ":reset". The script files cannot contain module declarations since the code snippets in the files will be compiled into REPL's default module.