OSGi, class loading and class building

[Simon Schäfer]

For the extensible Scala IDE project I need to solve some problems
related to class loading, where I could need some help because I'm not
very familiar with the topic.

First, an explanation of what needs to be achieved: The Scala IDE should
be able to load Scala code that is located anywhere on the filesystem
with the purpose to execute it. It should do this in an user friendly
way, which means that in the best case users only have to edit a file
and the IDE automatically takes the changes and handles them. User
friendliness shouldn't be very important at the beginning but it may
already lead to a different design.

All in all there are two main problems that need to be solved:

(1) Make Scala IDE extensions available to the editor that is used to
edit the Scala files
(2) Load and run Scala files by the IDE

For (1) it would be nice if the already existing Scala editor could be
used, for which we probably would need a project that exists in the
workspace. Forcing users to create a project shouldn't be a problem
because they need a way to manage their Scala files anyway. Though I see
it as a problem when users need to install PDE in order to be able to
create a plugin project. I also see it as a problem when users have to
manage the classpath of their project, which is in some way required
because an interface that is defined in the IDE needs to be available to
users. Providing interfaces is probably the cleanest way to find out of
what type a Scala file is (it could be a save action, a refactoring
etc.). The easiest way to implement this requirement is to ask users to
add the dependency of the Scala IDE by themselves but I hope with a
custom builder this problem may be solved more elegantly. The builder
could enrich the classpath with everything needed, but what about other
features like code completion or refactorings? Would they work just with
an enhanced builder or are further steps required?

Because the IDE lives inside an OSGi context it is not that
straightforward to simply load classes at runtime with reflection, which
leads to (2). I couldn't find any information so far that could tell me
if it is possible at all to load classes at runtime that do not live in
an OSGi bundle. Is it possible and if not is it possible to work around
this limitation by creating an in memory bundle or a custom class loader
that can fake the existence of the bundle to the OSGi/Eclipse runtime?
Runtime loading of classes gets even more complicated by considering
that extensions created for the Scala IDE should be usable in multiple
workspaces or multiple running Eclipse processes. The IDE must even be
able to manage class files for different Scala versions. Not to mention
that access to the class files/the Scala sources needs to be thread
safe. I don't know much about OSGi and therefore I'm unsure where to
start looking for scalable solutions that fit well in the OSGi environment.

Simon

[iulian dragos]

Hi Simon,

These are important questions, and I have a few ideas about how it should work. I'll try to put them in a more detailed email tomorrow, and we'll have some time to iron it out during ScalaDays.

Regarding (2), this is for sure possible inside OSGi. The current REPL implementation (whose shortcut CMD-Shift-X stopped working in the nightlies, but can still be accessed by opening the Scala Interpreter view) is instantiating a Scala REPL and using 'eval' to compile and run on-the-fly Scala expression (you can even try to call System.exit, it will work!). The tricky points will be around what platform classes will be made available to REPL expressions.

cheers,

iulian

Simon

--
You received this message because you are subscribed to the Google Groups "Scala IDE Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-ide-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-ide-dev/5394EA99.4090909%40antoras.de.
For more options, visit https://groups.google.com/d/optout.

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

[Simon Schäfer]

On 06/09/2014 06:39 PM, iulian dragos wrote:

Hi Simon,

These are important questions, and I have a few ideas about how it should work. I'll try to put them in a more detailed email tomorrow, and we'll have some time to iron it out during ScalaDays.

Regarding (2), this is for sure possible inside OSGi. The current REPL implementation (whose shortcut CMD-Shift-X stopped working in the nightlies, but can still be accessed by opening the Scala Interpreter view) is instantiating a Scala REPL and using 'eval' to compile and run on-the-fly Scala expression (you can even try to call System.exit, it will work!). The tricky points will be around what platform classes will be made available to REPL expressions.

I'm not sure if the idea to use an interpreter would work for editor features. Code completion etc. should also benefit from an extended classpath. What do you think, would the integration of an interpreter into a custom builder work?

cheers,

iulian

On Mon, Jun 9, 2014 at 12:58 AM, Simon Schäfer <ma...@antoras.de> wrote:

For the extensible Scala IDE project I need to solve some problems related to class loading, where I could need some help because I'm not very familiar with the topic.

First, an explanation of what needs to be achieved: The Scala IDE should be able to load Scala code that is located anywhere on the filesystem with the purpose to execute it. It should do this in an user friendly way, which means that in the best case users only have to edit a file and the IDE automatically takes the changes and handles them. User friendliness shouldn't be very important at the beginning but it may already lead to a different design.

All in all there are two main problems that need to be solved:

(1) Make Scala IDE extensions available to the editor that is used to edit the Scala files
(2) Load and run Scala files by the IDE

For (1) it would be nice if the already existing Scala editor could be used, for which we probably would need a project that exists in the workspace. Forcing users to create a project shouldn't be a problem because they need a way to manage their Scala files anyway. Though I see it as a problem when users need to install PDE in order to be able to create a plugin project. I also see it as a problem when users have to manage the classpath of their project, which is in some way required because an interface that is defined in the IDE needs to be available to users. Providing interfaces is probably the cleanest way to find out of what type a Scala file is (it could be a save action, a refactoring etc.). The easiest way to implement this requirement is to ask users to add the dependency of the Scala IDE by themselves but I hope with a custom builder this problem may be solved more elegantly. The builder could enrich the classpath with everything needed, but what about other features like code completion or refactorings? Would they work just with an enhanced builder or are further steps required?

Because the IDE lives inside an OSGi context it is not that straightforward to simply load classes at runtime with reflection, which leads to (2). I couldn't find any information so far that could tell me if it is possible at all to load classes at runtime that do not live in an OSGi bundle. Is it possible and if not is it possible to work around this limitation by creating an in memory bundle or a custom class loader that can fake the existence of the bundle to the OSGi/Eclipse runtime? Runtime loading of classes gets even more complicated by considering that extensions created for the Scala IDE should be usable in multiple workspaces or multiple running Eclipse processes. The IDE must even be able to manage class files for different Scala versions. Not to mention that access to the class files/the Scala sources needs to be thread safe. I don't know much about OSGi and therefore I'm unsure where to start looking for scalable solutions that fit well in the OSGi environment.

Simon

--
You received this message because you are subscribed to the Google Groups "Scala IDE Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-ide-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-ide-dev/5394EA99.4090909%40antoras.de.
For more options, visit https://groups.google.com/d/optout.

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

--
You received this message because you are subscribed to the Google Groups "Scala IDE Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-ide-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-ide-dev/CAOwe9faQ_nZ%3DBZVMtPqQ_rPppm5Ffj71LmVqeKAQCqG3RW1nmQ%40mail.gmail.com.

[iulian dragos]

Here's a bit more detail on how I think this feature would work. Users should be allowed to write (relatively) short Scala snippets that are invoked by Scala IDE at some well-defined points. It helps to take 2 examples, so we have some concrete user-interaction scenarios.

1. Save actions: Scala code gets invoked every time the editor is saved. The snippet has access to the current editor and make make modifications before it gets saved. For example, it might remove all trailing spaces.

2. Detail formatters: The Variables view in the debugger may delegate the String rendering for a given type to some Scala code. The code will have access to the JDI object mirrors that allow retrieval of fields and invocation of methods inside the debugged VM.

The first issue we need to solve is how to compile and execute these snippets. What should be on their classpath? In my opinion, they should have the classpath of the running Eclipse IDE (meaning snippets can't add their own dependencies). What runtime objects (from the running IDE) can they access? I think this will be specific to each use case. For Save Actions, this would be something like "currentDocument: IDocument". For the detail formatter it might be "currentValue: IValue" and maybe "frame: IFrame" and "thread: IThread", which are all JDI types.

Execution should be handled through the Scala toolbox compiler (or the interpreter). Scala reflection has an "eval" method, and a 'bind' method that allows one to inject variables from the "outside" into the context where the expression is evaluated. We'd probably pre-compile and save the bytecode somewhere int he configuration area.

Regarding where these snippets live, I think they need to be workspace specific. They should appear inside the regular Scala settings dialogs, and should be enabled/disabled independently. When they're changed, they should be compiled (errors reported, etc.). It would be great to be able to use the regular Scala editor, but I don't think that will be easy. Maybe a simpler solution is to reuse the worksheet editor, which is sort-of bare-bones and easier to disentangle from the rest of the IDE. It will be challenging to make it run without a project (as Simon correctly noticed). However, I feel this aspect is not a blocker for the first "release". I expect these snippets to be relatively short, and we can gain already a LOT by having them.

You can have a look at how the eval-expression branch is doing (Jedd's work), and his gist for the Repl assistance: https://gist.github.com/jhaberstro/2a5f71093c6ac4ee754e

cheers,

iulian

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-ide-dev/5398ABA5.4060102%40antoras.de.

For more options, visit https://groups.google.com/d/optout.

[Simon Schäfer]

On 06/13/2014 03:19 PM, iulian dragos wrote:

Here's a bit more detail on how I think this feature would work. Users should be allowed to write (relatively) short Scala snippets that are invoked by Scala IDE at some well-defined points. It helps to take 2 examples, so we have some concrete user-interaction scenarios.

1. Save actions: Scala code gets invoked every time the editor is saved. The snippet has access to the current editor and make make modifications before it gets saved. For example, it might remove all trailing spaces.

2. Detail formatters: The Variables view in the debugger may delegate the String rendering for a given type to some Scala code. The code will have access to the JDI object mirrors that allow retrieval of fields and invocation of methods inside the debugged VM.

The first issue we need to solve is how to compile and execute these snippets. What should be on their classpath? In my opinion, they should have the classpath of the running Eclipse IDE (meaning snippets can't add their own dependencies). What runtime objects (from the running IDE) can they access? I think this will be specific to each use case. For Save Actions, this would be something like "currentDocument: IDocument". For the detail formatter it might be "currentValue: IValue" and maybe "frame: IFrame" and "thread: IThread", which are all JDI types.

Execution should be handled through the Scala toolbox compiler (or the interpreter). Scala reflection has an "eval" method, and a 'bind' method that allows one to inject variables from the "outside" into the context where the expression is evaluated. We'd probably pre-compile and save the bytecode somewhere int he configuration area.

Regarding where these snippets live, I think they need to be workspace specific. They should appear inside the regular Scala settings dialogs, and should be enabled/disabled independently. When they're changed, they should be compiled (errors reported, etc.).

A lot of nice ideas, I'm going to respond to them at ScalaDays, I'm too lazy to do that now. ;)

It would be great to be able to use the regular Scala editor, but I don't think that will be easy.

Actually, I implemented it this way and it was trivial. ;)

Ok, the concept was the simple thing to grasp, the actual implementation turned out to need a lot of debugging. What I did is, I extended the classpath of the presentation compiler and the builder. There are still some problems (code completion and find references don't find the additional members) but all other features of the editor just work out of the box this way. This also solves the problem of how to produce binaries.

Maybe a simpler solution is to reuse the worksheet editor, which is sort-of bare-bones and easier to disentangle from the rest of the IDE. It will be challenging to make it run without a project (as Simon correctly noticed). However, I feel this aspect is not a blocker for the first "release". I expect these snippets to be relatively short, and we can gain already a LOT by having them.

You can have a look at how the eval-expression branch is doing (Jedd's work), and his gist for the Repl assistance: https://gist.github.com/jhaberstro/2a5f71093c6ac4ee754e

I already had a look to the worksheet and the repl implementation but I see some more challenges than just simply executing the code, but lets discuss this in more detail at ScalaDays.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-ide-dev/CAOwe9fbNAiD2BqGo%2B79oUujz64syy-dhQEsB0tbhhwXs62GP3w%40mail.gmail.com.