[scala-ide-dev] scala library as osgi bundle

[Stefan Langer]

Hi,

I'm currently working on integrating sbt into eclipse (see
http://www.assembla.com/spaces/sbt-eclipse-integration) and I'm
running into the problem of providing a scala version for the plugin.
Currently I simply depend on the bundle provided by scala ide plugin
making me depend on the installation of scala-ide which currently is
the nightly builds so no update site for sbt-eclipse-integration yet
;).
I also found the scala osgi bundle at
http://github.com/weiglewilczek/scala-lang-osgi from heiko seeberger.
Now the problem is I would only like to provide my own scala osgi
bundle when nessecary and would prefer if there was a updatesite where
osgi bundles can be downloaded from with the correct version
identifier.
Is anyhting like this planed for scala-ide?
If not would it be feasable to get a unified osgi bundle deployed
somewhere where each project depending on scala in a osgi setting can
depend on and download it from and not everybody must ship their own
version? During this process I also strongly advise to make the osgi
bundle a well behaved bundle that declares its uses clauses and its
version of exported packages properly, etc..

What are your thoughts on this?

-Stefan

[Miles Sabin]

This is something I'd very much like to do.

What do people want to see? An update site which provides the Scala
library and compiler bundles separately? I'd be happy to host that on
www.scala-ide.org.

I'm a bit wary of introducing Scala library modularization by the
backdoor, but this might also be an opportunity to split up the Scala
library bundle somewhat. What do people think?

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

[Jin Mingjian]

scala.tools.nsc (compiler?) is osgized itself when born from the scala-lang. It is best that the scala librarys can add some mf infos to be osgized when they born.  One thing is, heiko seeberger's ScalaModules has become one subproject of Eclipse-Equinox. I guess ScalaModules will affect the Scala adoption of Eclipse community.

2010/5/11 Stefan Langer <mailto...@googlemail.com>

[Stefan Langer]

2010/5/11 Jin Mingjian <jin...@gmail.com>:

> scala.tools.nsc (compiler?) is osgized itself when born from the scala-lang.
> It is best that the scala librarys can add some mf infos to be osgized when
> they born.  One thing is, heiko seeberger's ScalaModules has become one
> subproject of Eclipse-Equinox. I guess ScalaModules will affect the Scala
> adoption of Eclipse community.
>

Even more important to get this unified across the whole developer
community. For special cases there is still the option to ship ones
own bundle.
Important here is that the produced osgi bundle provides all the
information it can this should include the version for the exported
packages. As Heiko already has a build inplace to produce this
information (he uses maven but he also provided a sbt plugin) it might
be beneficial to use this for the bundle used by scala-ide too.


- Stefan

[Heiko Seeberger]

On 11 May 2010 10:43, Miles Sabin <mi...@milessabin.com> wrote:

This is something I'd very much like to do.

What do people want to see? An update site which provides the Scala
library and compiler bundles separately? I'd be happy to host that on
www.scala-ide.org.

IMHO an update site can only be an additional source for OSGified Scala libraries. They have to be available via Maven repo and simple download, too.

 

I'm a bit wary of introducing Scala library modularization by the
backdoor, but this might also be an opportunity to split up the Scala
library bundle somewhat. What do people think?

I am all for splitting the Scala library, if that's possible. As far as I understand one big (maybe the biggest) issue is the current build process. If EPFL manages to switch to SBT, OSGifying would be a piece of cake using bnd4sbt.

I would like to add that for me it is very important that the Scala libraries are OSGified properly. E.g. the manifests must contain versions and version ranges on the packages, must use the uses directive, etc. Of course separate bundles for scala-library, scala-compiler, etc.

Such bundles are already available from the scala-lang-osgi project. I would be glad to transfer this project to whatever prominent place. It would be great, if the Scala IDE could use these bundles ...

Heiko

Company: weiglewilczek.com
Blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net
Stambecco, highly scalable computing: stambecco.org

[Miles Sabin]

On Tue, May 11, 2010 at 11:32 AM, Heiko Seeberger
<heiko.s...@googlemail.com> wrote:
> IMHO an update site can only be an additional source for OSGified Scala
> libraries. They have to be available via Maven repo and simple download,
> too.

Doesn't this become an non-issue with Tycho and Maven 3.0?

> I am all for splitting the Scala library, if that's possible. As far as I
> understand one big (maybe the biggest) issue is the current build process.
> If EPFL manages to switch to SBT, OSGifying would be a piece of cake using
> bnd4sbt.
> I would like to add that for me it is very important that the Scala
> libraries are OSGified properly. E.g. the manifests must contain versions
> and version ranges on the packages, must use the uses directive, etc. Of
> course separate bundles for scala-library, scala-compiler, etc.

Agreed.

> Such bundles are already available from the scala-lang-osgi project. I would
> be glad to transfer this project to whatever prominent place. It would be
> great, if the Scala IDE could use these bundles ...

I'd love to have this under the Scala Eclipse banner if you're willing.

I think my only real concern here is that we need to be able to
support (even if not exclusively) a manifest-first approach ... that's
pretty much a precondition for self-hosted development of the IDE and
is consistent with the move to Tycho.

Does that work for you?

[Heiko Seeberger]

Miles,

On 11 May 2010 12:44, Miles Sabin <mi...@milessabin.com> wrote:

Doesn't this become an non-issue with Tycho and Maven 3.0?

Have not tried yet ...

I'd love to have this under the Scala Eclipse banner if you're willing.

Sure, very good place IMHO.

 

I think my only real concern here is that we need to be able to
support (even if not exclusively) a manifest-first approach ... that's
pretty much a precondition for self-hosted development of the IDE and
is consistent with the move to Tycho.

Does that work for you?

Hum, not sure! Maybe we generate the manifest via BND and put it in the bundle for each version of Scala?

Heiko

[Stefan Langer]

Hi Miles,

why the need for the manifest first approach?

-Stefan

2010/5/11 Heiko Seeberger <heiko.s...@googlemail.com>:

[Miles Sabin]

On Wed, May 12, 2010 at 2:19 PM, Stefan Langer
<mailto...@googlemail.com> wrote:
> why the need for the manifest first approach?

To support self-hosted development.

[Stefan Langer]

In which way is this only possible when you do the manifest first
approach! Sorry if I do not get it but shouldn't it be enough to have
bnd in the build that creates the manifest from the scala libraries
and compiler jars?

-Stefan

2010/5/12 Miles Sabin <mi...@milessabin.com>:

[Miles Sabin]

On Wed, May 12, 2010 at 2:38 PM, Stefan Langer
<mailto...@googlemail.com> wrote:
> In which way is this only possible when you do the manifest first
> approach! Sorry if I do not get it but shouldn't it be enough to have
> bnd in the build that creates the manifest from the scala libraries
> and compiler jars?

Right, but I don't what to have to build .jars continuously.

With Eclipse projects acting as unpacked bundles (with an explicit
manifest in /META-INF) I can launch an Eclipse instance straight out
of the development workspace. I'm not averse to using a tool like bnd
to generate the manifests, but I don't want to have to do a complete
build each time I want to launch an instance from my workspace.

[Stefan Langer]

ok now I understand. So a valid approach would be to generate the
nessecary manifest files and then load the generated file into the
appropriate META-INF folder for the plugin in order to prevent
rebuilding of the jars all the time.

Can this be easily realized with the maven build or is there a need to
code some java stuff to realize this? Sorry I'm not too familiar with
coding mojos for maven builds, among other things it is stuff like
this why I prefer sbt or rake as a build tool.

Sorry if I sound like I'm pushing this but I want to get this resolved
in a community friendly way ASAP in order to go on with my sbt eclipse
plugin. If I can do anything to resolve this issue feel free to give
me a task and I will try my best.

-Stefan

2010/5/12 Miles Sabin <mi...@milessabin.com>:

[Miles Sabin]

On Wed, May 12, 2010 at 3:32 PM, Stefan Langer
<mailto...@googlemail.com> wrote:
> ok now I understand. So a valid approach would be to generate the
> nessecary manifest files and then load the generated file into the
> appropriate META-INF folder for the plugin in order to prevent
> rebuilding of the jars all the time.

I assume so.

> Can this be easily realized with the maven build or is there a need to
> code some java stuff to realize this? Sorry I'm not too familiar with
> coding mojos for maven builds, among other things it is stuff like
> this why I prefer sbt or rake as a build tool.

I don't know ... hence my question to Heiko.

> Sorry if I sound like I'm pushing this but I want to get this resolved
> in a community friendly way ASAP in order to go on with my sbt eclipse
> plugin. If I can do anything to resolve this issue feel free to give
> me a task and I will try my best.

That's fine. I'm really pleased that you want to work on sbt integration :-)

There are lots of tasks I could give you, but I'd like to know what
the blocker is right now for _your_ project ... I would have thought
that you could make progress with the ant based setup that we have
currently (OK, you might have to hold your nose while you're doing it,
but it should be workable ;-).

[Stefan Langer]

2010/5/12 Miles Sabin <mi...@milessabin.com>:
> [...}

> There are lots of tasks I could give you, but I'd like to know what
> the blocker is right now for _your_ project ... I would have thought
> that you could make progress with the ant based setup that we have
> currently (OK, you might have to hold your nose while you're doing it,
> but it should be workable ;-).

Not really a blocker more like a inconvenience. Since I'm still in
early development without a official release yet I can easly change
things around so I would prefer to have a unified osgi scala library
bundle that can be downloaded somewhere instead of having to ship our
own scala bundle. Thats all.

I just thought if nobody else has the time to do the changes in the
build I can step in and try to resolve this. I must say I haven't
looked at scala-ides build setup yet so no promises made ;-).

I will see how far I get with the ant based setup and the stuff heiko
is doing and keep an eye on this thread in the hopes that we can
produce a update sites for scala artifacts. A lot of projects should
benefit from this.

-Stefan

[Miles Sabin]

On Wed, May 12, 2010 at 3:51 PM, Stefan Langer
<mailto...@googlemail.com> wrote:
> 2010/5/12 Miles Sabin <mi...@milessabin.com>:
>> [...}
>> There are lots of tasks I could give you, but I'd like to know what
>> the blocker is right now for _your_ project ... I would have thought
>> that you could make progress with the ant based setup that we have
>> currently (OK, you might have to hold your nose while you're doing it,
>> but it should be workable ;-).
> Not really a blocker more like a inconvenience. Since I'm still in
> early development without a official release yet I can easly change
> things around so I would prefer to have a unified osgi scala library
> bundle that can be downloaded somewhere instead of having to ship our
> own scala bundle. Thats all.
>
> I just thought if nobody else has the time to do the changes in the
> build I can step in and try to resolve this. I must say I haven't
> looked at scala-ides build setup yet so no promises made ;-).

If you're willing to see if you can get a proof of concept of a Tycho
build of the scala-plugin and scala-plugin-aspects projects up and
running that would be wonderful. You could take a look at what Mirko
has done with the refactoring project and his blog,

http://misto.ch/eclipse-plugins-scala-maventycho/

as a guide.

It should be me doing that right now, but I'm currently wrestling with
Amazon EC2 for the build host ... I'm boggling that the most recent
Fedora image that seems to be even vaguely sane is Fedora 8! :-(

[Heiko Seeberger]

Hey guys,

I don't know ... hence my question to Heiko.

Not exactly sure what you are asking, but if you want to know how you can create a proper manifest for scala-library.jar using Maven take a look at http://github.com/weiglewilczek/scala-lang-osgi. To be honest, this POM will not only create a manifest, but also create bundle. But the docs of the Maven bundle plugin contain a description for creating a manifest only: http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html.

Heiko

[Stefan Langer]

I'll get on it and see what I can do.

-Stefan

2010/5/12 Miles Sabin <mi...@milessabin.com>:

[Miles Sabin]

On Wed, May 12, 2010 at 4:13 PM, Heiko Seeberger
<heiko.s...@googlemail.com> wrote:
>> I don't know ... hence my question to Heiko.
>
> Not exactly sure what you are asking, but if you want to know how you can
> create a proper manifest for scala-library.jar using Maven take a look
> at http://github.com/weiglewilczek/scala-lang-osgi. To be honest, this POM
> will not only create a manifest, but also create bundle. But the docs of the
> Maven bundle plugin contain a description for creating a manifest
> only: http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html.

I'm sure that's fine for a bnd + vanilla Maven solution, but I'm keen
to at least attempt to go with Tycho. If it turns out that bnd can be
used to periodically generate manifests which can be used for
self-hosted development and consumed by Tycho builds then that would
be wonderful. But I've never done that, and I'm hoping for some
insights from the community :-)

[mwg]

On May 12, 5:22 pm, Miles Sabin <mi...@milessabin.com> wrote:
> I'm sure that's fine for a bnd + vanilla Maven solution, but I'm keen
> to at least attempt to go with Tycho. If it turns out that bnd can be
> used to periodically generate manifests which can be used for
> self-hosted development and consumed by Tycho builds then that would
> be wonderful. But I've never done that, and I'm hoping for some
> insights from the community :-)

By using bnd, you'll lose some of the compile time checks that you'd
ordinarily get in Eclipse with a manifest first approach. If using bnd
in conjunction with Maven, you get other problems because Maven
doesn't know or care about OSGi.

With Tycho, dependencies are calcaluted _before_ compilation in the
OSGi way. This leads to a correct dependency graph and a correct
classpath for each bundle (at least for Java - Scala still needs a bit
of work to fully honour import/export directives).

The thing with bnd is that it's sort of a remedy to fix issues with
Maven and OSGi (and to wrap existing non-OSGi jars which it is quite
good at). bnd only kicks in _after_ your code has been compiled. At
that point bnd will automatically calculate import/export OSGi headers
based on what your code's package layout and what you've used not
coming from your own project.

But since bnd and Maven do not know about OSGi and modularity, what's
calculated may be wrong. You can for instance easily produce bundles
that access private classes in other bundles. You'll just get runtime
errors, which is unfortunate.

Additionally with Maven, since there's no OSGi-like dependency
resolution and no respecting import/export headers; your classpath
when compiling may be completely wrong. Again you risk runtime errors
rather than compile time errors.

Another point is that if you rely on bnd to calculate export/import
OSGi headers, you won't be able to resolve OSGi dependencies before
compiling: you'll need to compile code before knowing what's used, but
you can't do that because your compile time classpath depends on the
OSGi dependency resolution.

To overcome that issue, you'd need to specify everything yourself if
the bnd configuration file, which would be semantically equivalent to
specifying everything in the manifest. And since we're working in
Eclipse where the latter is the default and where built-in tooling for
that task exist, what would be the point of using bnd?

To recap the two alternatives:
* Manifest first: (the Eclipse way)
- Maintain bundle/package imports and exports explicitly in
MANIFEST.MF with
help from Eclipse tooling.
- This allows for self-hosted development
- Use OSGi's finer-grained dependency model:
You can depend on packages not knowing where they come from and
Eclipse will
resolve your bundle dependencies from your target platform.
- This works with Tycho that reads Eclipse project files and
MANIFEST.MF

* Generate MANIFEST.MF
- Maintain dependencies in external build files
(Maven POMs, Ant build.xml, etc.)
- More coarse-grained dependency model:
We can only specify bundle-level dependencies not package
dependencies and
will not have the OSGi's way of resolving bundle-dependencies
against the
target platform from import/export package declarations.
- You might encounter runtime errors (packages that cannot be
resolved)

Regards,
Martin & Michael

[Miles Sabin]

On Fri, May 14, 2010 at 7:58 AM, mwg <m...@maconomy.com> wrote:
<snip/>

Many thanks for that ... very, very helpful :-)

In particular, I find this,

> To overcome that issue, you'd need to specify everything yourself if
> the bnd configuration file, which would be semantically equivalent to
> specifying everything in the manifest. And since we're working in
> Eclipse where the latter is the default and where built-in tooling for
> that task exist, what would be the point of using bnd?

extremely compelling.

Heiko, Neil ... what's your take on this?

[Heiko Seeberger]

Hi,

I agree that only manifest first enables compile time support for OSGi. As I have been using PDE for quite a while, I know about the benefits, but in the end I think this is nice to have, but not too important.

Another thing that I find very important is the calculation of the uses directive. BND does that, PDE only if you press a button. Hence the 

Heiko

--
Heiko Seeberger

[Jin Mingjian]

one note: Apache Felix seemly uses the bnd+maven to build itself long time.

2010/5/14 Heiko Seeberger <heiko.s...@googlemail.com>

[Neil Bartlett]

Miles has asked me to respond to these points. I should declare my
interest because I have what may be regarded as a competing solution
-- Bndtools, a Bnd-based Eclipse IDE for OSGi. But I still believe in
the right tool for the job, and in fact I think Tycho would be a fine
choice for Scala IDE.

Nevertheless there are some misconceptions about Bnd which I feel I
must challenge.

Regarding the loss of compile time checks. Note that the compiler in
both cases is not Bnd nor Tycho -- it is javac, ecj or scalac. None of
these compilers knows anything about exported packages or private
packages: when we add a JAR or folder to the compilation classpath,
all of the classes therein are visible. So whatever solution we choose
must be layered on top of the compiler.

When using PDE/Tycho on Java code, a rather complex system of
visibility constraints is used. The challenge is that the compiler
must restrict the visible packages to the intersection of two sets:
the exported packages of the dependencies, AND the imported packages
of the current bundle. Ecj, the Eclipse Java Compiler, includes some
enhanced options to hide packages, but even then it is not
particularly close to the runtime checks that exist in OSGi. Also
neither javac nor (AFAIK) scalac support those additional options. As
you say "Scala still needs a bit of work"... quite an understatement
IMHO.

You state that with Tycho, dependencies are calculated before
compilation. That is incorrect... nothing is calculated, it must be
manually entered and maintained. This is the essence of the so-called
manifest first approach: the MANIFEST.MF file is the primary source
file which describes the dependencies and all other metadata for the
bundle. You make changes directly in MANIFEST.MF, and the file you
edit appears as-is in the resulting bundle, so you have to list all of
your Import-Packages by hand.

Bnd's approach is to treat the Java/Scala code (actually the compiled
bytecode) as prime. The dependencies are derived from what is actually
used, and Bnd does an excellent job of ensuring that ALL those
dependecies that actually exist in the code -- and ONLY those
dependencies -- are added to the Import-Package statement.

Let me go back to the compilation issue. Yes, we must find a way to
compile the current bundle, but this is actually easier than with PDE/
Tycho because we only need to restrict the visible packages to the
exports of the dependencies. The solution I have implemented in
Bndtools is to simply strip non-exported packages from the bundles
that I store in my repository -- this works with ANY compiler. Of
course I still keep the unstripped bundles around for use at runtime!

But suppose I don't bother to do that stripping -- what happens if a
bundle built with Bnd tries to use an internal package from another
bundle, e.g. "org.foo.internal"? The compilation will succeed, and Bnd
will detect the dependency and add "org.foo.internal" to the Import-
Package for my bundle. When I deploy the bundle at runtime it will
refuse to resolve, because no other bundle will export
"org.foo.internal".

Admittely this is a runtime error, but it is a LOT different from
something like a NoClassDefFoundError or ClassNotFoundException. The
evil thing about those exceptions is they only get thrown when a
certain code path is executed. But a resolution problem due to missing
imports is immediately apparent: your bundle will be stuck in the
INSTALLED state, and running the "diag" command on it will inform you
"Missing imported package: org.foo.internal".

Also I admit that we need to maintain some kind of module-level
dependencies, but the need from these is different from what you
state. They are needed only for two things: (1) compiling modules in
the correct order, and (2) ensuring enough JARs are visible on the
compiler classpath. However this does NOT affect the runtime
dependencies, which come only from Import-Packages and the package-
dependencies which ACTUALLY exist in the code. With a Maven/Bnd-style
build, there is no cost associated with adding extra module-level
dependencies (well, the compilation might be fractionally slower).

In contrast, a spurious Import-Package or Require-Bundle added to
MANIFEST.MF with a manifest-first tool will go straight through to
your bundle, which now will refuse to resolve when the
"org.dont.need.this" package is absent, even though your code doesn't
actually use the package.

So to summarise:

1) The compile-time checking in PDE is only produced at the great
expense of manual maintenance of Import-Packages, and even then is not
a great match for the true runtime checks performed by OSGi;

2) Bnd automates the production of Import-Package, and does so more
accurately and reliably than any manual process can;

3) Compile-time checks as good as PDE's can be achieved with Bnd by
using stripped bundles at compile time.

Now, going back to the more relevant question of which tooling Miles
should choose, as much as I would like such a visible project to be
using Bndtools, I would probably recommend Miles to choose Tycho, for
the following reasons:

1) Tycho is based on PDE and requires very little change (if any?) to
the layout of an existing PDE project;

2) As I understand it, Scala IDE has relatively few large-grained
bundles. Problems with manually maintaining manifests become much more
acute with larger numbers of bundles and packages, but I don't believe
that is Miles' number one problem;

3) Tycho is more mature than Bndtools and has the backing of a company
with deep pockets. Bndtools is something that I sincerely hope will
become widely adopted, but it is clearly not there yet.

Kind regards
Neil

[Neil Bartlett]

Another point while I'm in the thread... Miles states that manifest-
first is required for self-hosting. It is not.

Bnd can build an average bundle in a matter of milliseconds. Bndtools
uses this to incrementally build a new bundle whenever the code inside
it changes. So as soon as you save your Java/Scala code, a new bundle
is produced. Therefore Bndtools absolutely supports self-hosting,
using the actual bundle JARs; there is no need for a build step before
running.

In fact self-hosting in PDE requires some hackery, because a PDE
project typically compiles its classfiles to the "bin" directory
rather than the root of the bundle. Equinox includes a "dev mode",
enabled by the -dev command line switch, to make classfiles in bin
appear to be at the root. Goodness knows how they are able to launch
other frameworks such as Felix. Needless to say I do feel that my
solution is somewhat cleaner :-)

Regards
Neil

[Miles Sabin]

On Fri, May 14, 2010 at 12:58 PM, Neil Bartlett <njbar...@gmail.com> wrote:
> Miles has asked me to respond to these points.

Also incredibly helpful ... thanks.

I'm still unclear whether or not it's possible to profitably use both
bnd (either with or without bndtools) and Tycho. Tycho requires
manifests and bnd can create those ... is there any reason why we
can't use bnd's output as Tycho's input? Here I'm assuming that
running of bnd would be done manually, quite infrequently and would be
used to generate the manifests that live at
<project>/META-INF/MANIFEST.MF and support self hosted development.

And although I would be quite happy to go my own way for the IDE
components, we still need a solution for the Scala compiler and
libraries that is suitable for general consumption. I think that Tycho
is less compelling here, because the Scala compiler and library
project has no external binary dependencies, and depends on an Ant
build script in any case.

So I would be open to the idea of using bnd(tools) for the Scala
compiler and libraries, if that got on well with the approach that we
want to take to the OSGi-fication of Scala beyond the IDE, *and* if we
could come up with some scheme which supported self-hosted development
in the context where we have source projects in the workspace for the
compiler and libraries (see my "goals" mail).

[Miles Sabin]

On Fri, May 14, 2010 at 1:11 PM, Neil Bartlett <njbar...@gmail.com> wrote:
> Another point while I'm in the thread... Miles states that manifest-
> first is required for self-hosting. It is not.
>
> Bnd can build an average bundle in a matter of milliseconds. Bndtools
> uses this to incrementally build a new bundle whenever the code inside
> it changes. So as soon as you save your Java/Scala code, a new bundle
> is produced. Therefore Bndtools absolutely supports self-hosting,
> using the actual bundle JARs; there is no need for a build step before
> running.
>
> In fact self-hosting in PDE requires some hackery, because a PDE
> project typically compiles its classfiles to the "bin" directory
> rather than the root of the bundle. Equinox includes a "dev mode",
> enabled by the -dev command line switch, to make classfiles in bin
> appear to be at the root. Goodness knows how they are able to launch
> other frameworks such as Felix. Needless to say I do feel that my
> solution is somewhat cleaner :-)

I still have to try this, but as we've discussed before, my concern is
that by going down this route we would have to give up on JDI-level
hotswapping (because that isn't going to work across regeneration of
jar files).

But maybe if we restrict this to the Scala compiler and library
projects that isn't such a big deal?

So, opinions on the following would be welcome,

* bnd for the Scala compiler and libraries
* Tycho/Maven for the IDE projects
* all coexisting happily in a single Eclipse workspace
* supporting self-hosted development (including JDI hotswap for at
least the IDE projects)
* either source projects or binary dependencies for the Scala compiler
and libraries

Can this be made to work?

[Neil Bartlett]

As I said in our Skype chat, you don't give up JDI debugging.
Hotswapping of OSGi bundles is always an option, irrespective of the
build tool you use. If you don't want to do it, then don't.

Bnd for Scala compiler and libs plus Tycho for Scala IDE is certainly
possible. Actually the Maven Bundle Plug uses Bnd internally so this
is just Maven + Tycho. The EGit project does this already: JGit is
built with Maven and EGit (the Eclipse plug-in part) is built with
Tycho.

Another possibility is to use Bnd for everything, and Bndtools to give
you a nicer development experience on top of Bnd. To be honest at this
stage in Bndtools' evolution I would have to work closely with you to
iron out any problems. I'm more than happy to do that but you might
prefer to go with something more proven.

Regards
Neil




On May 14, 1:21 pm, Miles Sabin <mi...@milessabin.com> wrote:

[Miles Sabin]

On Fri, May 14, 2010 at 2:06 PM, Neil Bartlett <njbar...@gmail.com> wrote:
> Bnd for Scala compiler and libs plus Tycho for Scala IDE is certainly
> possible. Actually the Maven Bundle Plug uses Bnd internally so this
> is just Maven + Tycho. The EGit project does this already: JGit is
> built with Maven and EGit (the Eclipse plug-in part) is built with
> Tycho.

Properly Maven-izing the Scala compiler and libraries is something I
would much rather leave to EPFL. But using bnd from the Ant script
seems like a possibility.

I'd need to coordinate this with EPFL, because it adds another tool to
the set they need to build a distribution and because the main build
script would need some small tweaks ... so I'd like to be confident
that this approach would be useful for people in the general case, not
just in the context of IDE development.

Opinions?

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com

[Jin Mingjian]

Bnd is more lightweight than PDE in fact. I think "all roads lead to Rome".

IMHO, a simpler build system is better for potential hacking and contributing.

2010/5/14 Miles Sabin <mi...@milessabin.com>

[Martin Gamwell Dawids]

On May 14, 2:14 pm, Miles Sabin <mi...@milessabin.com> wrote:
> I'm still unclear whether or not it's possible to profitably use both
> bnd (either with or without bndtools) and Tycho. Tycho requires
> manifests and bnd can create those ... is there any reason why we
> can't use bnd's output as Tycho's input? Here I'm assuming that
> running of bnd would be done manually, quite infrequently and would be
> used to generate the manifests that live at
> <project>/META-INF/MANIFEST.MF and support self hosted development.

Taking the risk into consideration that I have misunderstood
something, isn't this a catch 22 situation?

In order for bnd to calculate import/export declarations and generate
manifests it must know the bundle dependencies. Otherwise it does not
know in which order the bundles should be compiled and which bundles
to put on the classpath. And to calculate the import/export it uses
the compiled class files.

Tycho/PDE, on the other hand, needs to know the import/export
constraints stated in the manifests. It uses these to calculate the
bundle dependency graph.

So, if we use bnd to generate manifests, we must already have manually
set up our dependency graph, and then there is no need for Tycho/PDE
to calculate the dependency graph.

If we use Tycho, we already have the manifests and then there is no
need for bnd.

So isn't it either or?

Kind regards,
Martin

[Neil Bartlett]

We don't use both Bnd and Tycho to build the *same* bundles.

Bnd would be used to build a set of bundles. Tycho would then consume
those bundles, almost as if they were third party dependencies.

Certainly issues arise if there are dependencies in both directions --
i.e. a dependency cycle. In that case we would indeed need to use just
one tool.

Neil

[Miles Sabin]

On Fri, May 14, 2010 at 3:43 PM, Martin Gamwell Dawids <m...@maconomy.com> wrote:

> So, if we use bnd to generate manifests, we must already have manually
> set up our dependency graph, and then there is no need for Tycho/PDE
> to calculate the dependency graph.
>
> If we use Tycho, we already have the manifests and then there is no
> need for bnd.
>
> So isn't it either or?

I don't see that. Tycho/Maven does a great deal besides calculating
the dependency graph. Managing the dependencies on binary artefacts
being the main benefit that I see. Or am I misunderstanding and binary
artefacts are all managed by vanilla Maven?

[Miles Sabin]

On Fri, May 14, 2010 at 3:56 PM, Neil Bartlett <njbar...@gmail.com> wrote:
> We don't use both Bnd and Tycho to build the *same* bundles.
>
> Bnd would be used to build a set of bundles. Tycho would then consume
> those bundles, almost as if they were third party dependencies.

That's correct in the scenario where we use Ant+bnd for the Scala
compiler and library projects and Tycho for the IDE projects.

I was also wondering whether there would any benefit from mixing bnd
and Tycho for the IDE project: have bnd compute the manifests, and
have Tycho manage the target platform and binary artefacts given the
bnd-generated manifest as input. If this is gibberish, please do just
say ;-)

> Certainly issues arise if there are dependencies in both directions --
> i.e. a dependency cycle. In that case we would indeed need to use just
> one tool.

There aren't any dependency cycles to worry about ...

[Neil Bartlett]

Ah I see! No I don't believe that's possible. Tycho's raison d'etre is
to adapt manifest-first style projects into the Maven build, which
naturally favours generated manifests (or "pom-first"). If you're
going to generate the manifest then there's probably little point in
using Tycho -- just use Maven Bundle Plugin or Bnd directly.

Neil

[Martin Gamwell Dawids]

On May 14, 4:56 pm, Neil Bartlett <njbartl...@gmail.com> wrote:
> We don't use both Bnd and Tycho to build the *same* bundles.

Miles wrote previously:

> Tycho requires
> manifests and bnd can create those ... is there any reason why we
> can't use bnd's output as Tycho's input?

and

> Here I'm assuming that
> running of bnd would be done manually, quite infrequently and would be
> used to generate the manifests that live at
> <project>/META-INF/MANIFEST.MF and support self hosted development.

Therefore, I assumed Miles was talking about maintaining the manifests
for some bundles using bnd and using these generated manifests when
building the _same_ set of bundles using Tycho.

Kind regards,
Martin

[Miles Sabin]

On Fri, May 14, 2010 at 4:11 PM, Martin Gamwell Dawids <m...@maconomy.com> wrote:
> Therefore, I assumed Miles was talking about maintaining the manifests
> for some bundles using bnd and using these generated manifests when
> building the _same_ set of bundles using Tycho.

You understood me correctly, and it appears that I'm wrong: bnd +
Maven would make sense, as would Tycho/Maven, but bnd + Tycho/Maven
doesn't make sense.

[Martin Gamwell Dawids]

When I wrote

> So isn't it either or?

I meant either bnd or Tycho for building the same bundle.

Kind regards,
Martin

[Neil Bartlett]

Yes, I apologise, I misunderstood what Miles was asking.

Regards,
Neil

Sent from my BlackBerry®

[Martin Gamwell Dawids]

> Tycho/Maven does a great deal besides calculating
> the dependency graph. Managing the dependencies on binary artefacts
> being the main benefit that I see. Or am I misunderstanding and binary
> artefacts are all managed by vanilla Maven?

As far as I understand, Tycho resolves dependencies on binary
artifacts
a) either by from external p2 repositories and caches these in your
local m2 repository under
~/.m2/repository/p2
This target platform resolution is done independently for each
module.
b) or using an explicitly stated target platform using "-
Dtycho.targetPlatform=<folder>"
which you must then maintain manually.

You can read (a bit more) here:
http://tycho.sonatype.org/targetplatformmanagement.html

/Martin

[Miles Sabin]

On Fri, May 14, 2010 at 4:40 PM, Martin Gamwell Dawids <m...@maconomy.com> wrote:
> You can read (a bit more) here:
> http://tycho.sonatype.org/targetplatformmanagement.html

Right, this is definitely what I'm after.

So it seems that Tycho has a lot to offer even if manifest generation
were delegated to bnd. Is it just insanity to consider using the two
tools together (on a single bundle) or might there be some advantages
to doing that?

[Martin Gamwell Dawids]

On May 14, 1:58 pm, Neil Bartlett <njbartl...@gmail.com> wrote:
> Nevertheless there are some misconceptions about Bnd which I feel I
> must challenge.

Firstly, I must admit that Michael and I are not even close to being
experts on bnd. Also, we have had no intention of bashing bnd.

We have tried to use bnd (not Bndtools yet) but found Tycho more
compelling for our project. This is mainly because we are sharing an
already existing codebase (with a large number of bundles) with
another project in our company. This other project is already using
Eclipse PDE and a manifest-first approach. This existing codebase had
an external build based on Ant + PDE export and required a lot of
maintenance. Tycho lets us reuse the Eclipse PDE project definitions
and manifests.

> You state that with Tycho, dependencies are calculated before
> compilation. That is incorrect... nothing is calculated, it must be
> manually entered and maintained.

It is correct that Tycho does not calculate import/export directives.
However, it calculates the bundle dependency graph -- with Maven+bnd
you must manually maintain it.

Furthermore, the calculated dependency graph is used for
a) determining the correct compilation order of bundles
b) calculating the classpath for each bundle
(including package restrictions)

> When using PDE/Tycho on Java code, a rather complex system of
> visibility constraints is used. The challenge is that the compiler
> must restrict the visible packages to the intersection of two sets:
> the exported packages of the dependencies, AND the imported packages
> of the current bundle.

As I understand it, when PDE compiles a bundle it sets up the
classpath to contain all bundles on which the bundle to be compiled
depends...

> Ecj, the Eclipse Java Compiler, includes some
> enhanced options to hide packages,

...and this classpath is "annotated" with a string on each classpath
entry telling the Eclipse Java Compiler which packages of each
classpath element it is allowed to access.

> ...but even then it is not

> particularly close to the runtime checks that exist in OSGi.

It is correct that Eclipse/PDE can only give static guarantees about
bundle dependencies as long as the combined set of bundles in the
workspace and target platform are also the same set of bundles present
at runtime. Also, Eclipse/PDE cannot give static guarantuees about the
availability of services.

Is this what you are thinking of or am I missing someting? (I am not
an expert of OSGi.)

> As you say "Scala still needs a bit of work"... quite an
> understatement IMHO.

It is not that difficult. We have managed to make the Scala compiler
enforce the import/export restrictions in our Tycho builds by
a) writing our own very small OSGi Scala plug-in for Maven.
This uses Tycho functionality to calculate the classpath
annotations.
b) writing a very small plug-in for the Scala compiler to which we
pass these
classpath annotations. The plug-in then checks these restrictions
against referenced classes.

There are a few corner cases it does not handle yet (because it
currently kicks in after classes has been resolved/looked up from the
classpath) but it works in our builds to give us compile-time errors
when accessing non-accessible classes in other bundles.

> This is the essence of the so-called
> manifest first approach: the MANIFEST.MF file is the primary source
> file which describes the dependencies and all other metadata for the
> bundle. You make changes directly in MANIFEST.MF, and the file you
> edit appears as-is in the resulting bundle, so you have to list all of
> your Import-Packages by hand.

I completely agree. This all really boils down to whether one finds it
better to

a) Manually maintain the bundle dependency graph and get the manifests
generated automatically (Maven + bnd)

b) Manually maintain import/export declarations in MAINFEST.MF and
have PDE
calculate your bundle dependency graph automatically (Eclipse PDE,
Tycho)

I (currently) believe b) is more in the spirit of OSGi as one (in most
situations) should think of a bundle as a module consuming packages
(and services) provided by other bundles, but exactly which bundles
will be resolved by OSGi. For me, manually maintaining the dependency
graph is mentally close(r) to using "Require-Bundle" declarations in
the manifest which is discouraged in most situations.

For now, I still prefer b) although I will not promise that I might
change my mind in the future. :-)

Kind regards,
Martin

[Martin Gamwell Dawids]

On 14 Maj, 17:51, Miles Sabin <mi...@milessabin.com> wrote:
> So it seems that Tycho has a lot to offer even if manifest generation
> were delegated to bnd. Is it just insanity to consider using the two
> tools together (on a single bundle) or might there be some advantages
> to doing that?

I think it is either bnd or Tycho/PDE for a given set of bundles. As a
wrote previously.
a) Either you maintain the bundle dependency graph manually and get
your manifests generated.
b) Or you maintain your manifests (import/export declarations in
particular) and get you bundle dependency graph calculated.

This does not mean you cannot use Maven+bnd / Eclipse+Bndtools for one
set of bundles in your project and Tycho/PDE for another set of
bundles in the same project, but I believe using both for the same
bundle or set of bundles makes no sense.

Neil, please correct me if I got this wrong.

Regards,
Martin

[Heiko Seeberger]

Let me add to this discussion that EPFL is thinking about moving the Scala build to SBT (Simple Build Tool). There is a BND plugin for SBT I wrote (bnd4sbt) which would make it easy to create OSGi bundles from the scala-libraries.

Heiko

[Miles Sabin]

On Sat, May 15, 2010 at 7:17 AM, Heiko Seeberger
<heiko.s...@googlemail.com> wrote:
> Let me add to this discussion that EPFL is thinking about moving the Scala
> build to SBT (Simple Build Tool). There is a BND plugin for SBT I wrote
> (bnd4sbt) which would make it easy to create OSGi bundles from the
> scala-libraries.

Yes, although there's no definite timetable for that. Nevertheless,
I've now settled on the opinion that bnd is the way to go for the
Scala compiler and libraries.

I'm now also pretty much persuaded that for the IDE projects there's
no benefit to be had from attempting to mix Tycho and Maven (even if
it were possible) and that Tycho is the way to go.

Just to wrap things up, I'm wondering if bnd would be able to cope
with the AspectJ based project? Would bnd be able to infer bundle
dependencies which derive from references in pointcuts and advice?

[Heiko Seeberger]

On 15 May 2010 10:27, Miles Sabin <mi...@milessabin.com> wrote:

Just to wrap things up, I'm wondering if bnd would be able to cope
with the AspectJ based project? Would bnd be able to infer bundle
dependencies which derive from references in pointcuts and advice?

BND works on the compiled classes => If you go for compile-time weaving BND should work, but compile-time weaving and OSGi does not work (well) and as far as I know you are using Equinox Aspects for load-time weaving. BND does not know/understand AspectJ source code.

Heiko

[Miles Sabin]

On Sat, May 15, 2010 at 10:00 AM, Heiko Seeberger
<heiko.s...@googlemail.com> wrote:
> On 15 May 2010 10:27, Miles Sabin <mi...@milessabin.com> wrote:
>>
>> Just to wrap things up, I'm wondering if bnd would be able to cope
>> with the AspectJ based project? Would bnd be able to infer bundle
>> dependencies which derive from references in pointcuts and advice?
>
> BND works on the compiled classes => If you go for compile-time weaving BND
> should work, but compile-time weaving and OSGi does not work (well) and as
> far as I know you are using Equinox Aspects for load-time weaving. BND does
> not know/understand AspectJ source code.

If I'm following you correctly that means that bnd is a no-go for the
AspectJ project (which does depend on Equinox Aspects and load-time
weaving).

In which case I think that settles thinks in favour of Tycho/Maven for
the IDE projects.

[Heiko Seeberger]

On 15 May 2010 11:12, Miles Sabin <mi...@milessabin.com> wrote:

If I'm following you correctly that means that bnd is a no-go for the
AspectJ project (which does depend on Equinox Aspects and load-time
weaving).

In which case I think that settles thinks in favour of Tycho/Maven for
the IDE projects.

100% correct!

Heiko

[Neil Bartlett]

It means that with bnd you will need to manually add the dependencies that are weaved in at runtime, though the rest will be generated/analysed.

With PDE/Tycho you must manually add all the dependencies, including the compiled-in ones.

Neil

------Original Message------
From: Miles Sabin
To: scala-...@googlegroups.com
Cc: Neil Bartlett
Subject: Re: [scala-ide-dev] Re: scala library as osgi bundle
Sent: 15 May 2010 10:12

On Sat, May 15, 2010 at 10:00 AM, Heiko Seeberger
<heiko.s...@googlemail.com> wrote:
> On 15 May 2010 10:27, Miles Sabin <mi...@milessabin.com> wrote:
>>
>> Just to wrap things up, I'm wondering if bnd would be able to cope
>> with the AspectJ based project? Would bnd be able to infer bundle
>> dependencies which derive from references in pointcuts and advice?
>
> BND works on the compiled classes => If you go for compile-time weaving BND
> should work, but compile-time weaving and OSGi does not work (well) and as
> far as I know you are using Equinox Aspects for load-time weaving. BND does
> not know/understand AspectJ source code.

If I'm following you correctly that means that bnd is a no-go for the
AspectJ project (which does depend on Equinox Aspects and load-time
weaving).

In which case I think that settles thinks in favour of Tycho/Maven for
the IDE projects.

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
http://www.chuusai.com/
http://twitter.com/milessabin

Sent from my BlackBerry®

[Jin Mingjian]

From the technical aspect, runtime dependencies can be checked by integration test easily. (GUI test is one kind of great integration test.) No big reason to show runtime dependency checking at build-time is necessary or better than that at real-runtime.

2010/5/15 Neil Bartlett <njbar...@gmail.com>