Embed Frege in Java/Scala

[zcourts]

I'm currently investigating means of providing a DSL in our project.

Currently options have been narrowed down to embedding Frege or providing a very trivial extended lambda calc. like DSL using "Implementing functional programming languages" as a basis.

That right now seems like the best option but the biggest prevailing concern is that we will undoubtedly end up implementing a much richer lang. and hence the case for just starting with a full env. to begin with.

Looking at Frege, it seems like the perfect fit for this. Having looked at https://github.com/Frege/frege-repl I think we can take that apart and embed Frege in a similar way.

If we are to do that, is that how you'd recommend doing it?

Are there any recommendations in that regard (embedding that is)?

Importantly, are there any reasons why we shouldn't embed Frege (effectively turning it into an interpreter)?

The use case is such that:

Some client will submit expressions (there's a concern here of isolation, it looks like it'll have access to the entire JVM environment).

We validate it (run it through Frege interpreter, make sure it parses, all  fns used exist etc)

At some later point (mins, hours, days later), some event occurs which triggers execution of the expression.

Given the clear difference between the submission and later execution of the expressions, I'm wondering if it's feasible to use the Frege compiler to actually generate code and keep reference to the compiled expression that is then executed later when some event triggers it. From reading https://github.com/Frege/frege/wiki/Compiler-Manpage however I get the impression that's probably not practical because Frege's compiler's generating Java which will need to be compiled after. If that's the case, what does the REPL do, how's it able to "compile" Frege and run in the same session?

Any suggestions/comments/advice on or around the above will be appreciated.

Places I've looked at while researching this:

https://groups.google.com/forum/#!searchin/frege-programming-language/embed%7Csort:date/frege-programming-language/p9gXVbmsgiE/E2-4MH0LVBIJ

https://github.com/Frege/frege/wiki/Calling-Frege-Code-from-Java

http://mmhelloworld.github.io/blog/2013/07/10/frege-hello-java/

https://github.com/Frege/frege/wiki/Calling-Frege-from-Java-(from-release-3.24-on)

[Marimuthu Madasamy]

Hi,

Your use case sounds exactly like Frege REPL as you figured out. In fact, it is more related to online REPL with your sandbox concerns. Luckily both command line REPL and online REPL use a base library called frege-interpreter. Frege Interpreter does the actual work of compiling Frege to Java code and then compiling Java code to bytecode without the UI concerns of command line REPL and online REPL. It compiles everything in memory. It intercepts Frege compiler passes to store the generated Java files in memory and then it uses Java compiler API to compile the Java files into bytecode in memory.

Frege interpreter also supports JSR 223, Java Scripting API using which we can use Frege interpreter as a library and evaluate Frege expressions from Java. Here are some examples:

Example 1: Evaluating a Frege expression

final ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine frege = factory.getEngineByName("frege");
final Object actual = frege.eval("show $ take 10 [2,4..]");
final Object expected = "[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]";

Example 2: Compile and then evaluate later

final Compilable compilableFrege = (Compilable) frege;
final CompiledScript compiled =
        compilableFrege.compile("fib = 0 : 1 : zipWith (+) fib (tail fib)");
compiled.eval();
final Object actual = frege.eval("show $ take 6 fib");
final Object expected = "[0, 1, 1, 2, 3, 5]";

There are more examples here.

As for sandboxing, Frege online REPL itself uses a Java SecurityManager as one might expect which allows only few permissions and blocks everything else which you can see from here.

Please let us know if you need any help.

- Marimuthu Madasamy

[zcourts]

Hi Marimuthu,

Thank you for the reply.

That is very insightful! I'll experiment today/tomorrow and see how I get on.

For the moment however, you're direction's given more than enough to dive in.

Thanks

[zcourts]

I've had a chance to play with this and have come up with a few questions.

1. I've figured from the tests that modules can be loaded by using `eval` one at a time. Is there a way to load multiple modules without knowing the dependency order? Using `eval` I'd have to make sure that the modules are loaded in the correct order otherwise imports to modules not loaded yet will fail. 

So the workflow that seems to work right now is forall modules without main eval(m); then main = frege.compile(main) and later main.eval()

2. Can I specify the file name to use when loading a module? I can already see it being a nightmare to debug when all errors reported show that the input file is `<console>.fr`. I've found this but can't see how I'd provide it via the scripting interface.

3. Can I load compiled Frege dependencies? Example, what i'm thinking at the moment is that we will have a maven sub-module in our project that is all Frege code. This will provide additional types/functions by default (by running eval(import OurModule) before others). If this isn't possible we'll just have to ship the Frege module's source and compile/eval them prior to external code.

4. How does one make a variable available? This test implies that just setting a binding should work but following the test my example fails with:

Exception in thread "main" frege.runtime.Undefined: [E <console>.fr:3: can't resolve `bar`, did you mean `or` perhaps?]

5.  Looking around, the scripting api also has `ScriptContext ` and bindings (I've figured is used to make vars available) but it wasn't obvious to me how or if they could be used to achieve the above.

The example code I've been toying with:

public static void main(String[] args) throws IOException, ScriptException {

  final ScriptEngineManager factory = new ScriptEngineManager();

  ScriptEngine fregeEngine = factory.getEngineByName("frege");

  final Compilable compilableFrege = (Compilable) fregeEngine;

  Bindings bindings = fregeEngine.getBindings(ENGINE_SCOPE);
  bindings.put("bar :: Integer", new BigInteger("12312332142343244"));

  System.out.println("Modules");
  String a, b, c;
  a = "module A where\na:: Int -> Int\na i = i * i\n";
  b = "module B where\nb:: Int -> Int\nb i = i + i";
  c = "module Main where\nimport A(a)\nimport B(b)\n_ = show $ (a 2) * (b 3) + bar"; //why can't bar be resolved?
  //fregeEngine.eval(a + "\n" + b + "\n" + c); //todo we don't get dependency resolution, can Frege interpreter load multiple modules and resolve deps
  System.out.println("Eval A");
  //ScriptContext ctx;
  fregeEngine.eval(a); //returns null for modules, can I set name to A.hs???

  System.out.println("Eval B");
  fregeEngine.eval(b);

  System.out.println("Compile C");
  CompiledScript cs = compilableFrege.compile(c);

  System.out.println("Eval");
  //https://github.com/Frege/frege-interpreter/blob/master/frege-interpreter-core/src/test/java/frege/interpreter/scriptengine/FregeScriptEngineTest.java#L173
  //System.out.println(fregeEngine.eval("show bar"));
  //with out without bindings param this fails with:
  //Exception in thread "main" frege.runtime.Undefined: [E <console>.fr:4: can't resolve `bar`, did you mean `or` perhaps?]
  Object res = cs.eval(bindings);
  System.out.println(res);
}

I'm depending on version 3.23.288-gaa3af0c instead of the latest 3.24 because I'm added the frege-maven-plugin and the latest 1.0.8 of the plugin requires that version otherwise there it fails with NoClassDefError. Assuming refactor/delete of class between the 3.23 and 3.24 versions. The idea here was that this maven module would have the Frege sources that are built with the project and loaded/imported later as a library. In the same maven module the ScriptManager stuff would load the Frege lib compiled by the maven plugin, eval some imports from this lib and then eval/compile smaller input scripts that make use of this lib at run time.

[Ingo W.]

Small correction here:

Am Montag, 29. Januar 2018 14:41:24 UTC+1 schrieb zcourts:

Given the clear difference between the submission and later execution of the expressions, I'm wondering if it's feasible to use the Frege compiler to actually generate code and keep reference to the compiled expression that is then executed later when some event triggers it. From reading https://github.com/Frege/frege/wiki/Compiler-Manpage however I get the impression that's probably not practical because Frege's compiler's generating Java which will need to be compiled after. If that's the case, what does the REPL do, how's it able to "compile" Frege and run in the same session?

Actually, the generation of Java code is just one step in the pipeline.

It's simply the most easy way to generate bytecode and class files when you run on a JDK.

Regards, Ingo

[Marimuthu Madasamy]

Hi,

Thank you for trying out. Lot of good findings!

1. Loading multiple modules all at once is not supported in the interpreter at the moment but compiler obviously supports it so I think we should be able to add support in interpreter also.

2. I just learned that we can actually provide file name with scripting API using: engine.put(ScriptEngine.FILE_NAME, fileName). This is not supported at the moment in Frege interpreter but we could easily support this one.

3. Loading compiled Frege modules should work as long as modules are on the classpath of the JVM that is running the interpreter.

4. Binding variables currently only work in the default module "<console.fr>" as the user modules are passed to the compiler as they are without any changes that are necessary to "include" those variables. As you are trying to use binding variables in your custom module 'Main', it doesn't work at the moment.

I will get to some of this but PRs are most welcome.