Import-Package header support

[Nicolas Lalevée]

Hi,

Lately I was very interested in improving dependencies management of
OSGi bundles, so I am interested to bushel.

I saw on the TODO list that bushel will support the Import-Package and
Export-Package headers. That sounds great.
But dependencies management via package or via bundle/module id is
very different. And Ivy currently only support dependencies management
via module id.
So I am interested to know how the support of theses directives will
be supported.

Nicolas

[alexr]

Hi Nicolas,

There are two hurdles to overcome:
1) As you mentioned, Ivy is artifact based and therefore requires a
known location for an artifact to extract meta-data to resolve any
dependencies.
2) Further to problem 1), to resolve package imports you need to know
all available package exports independent of the artifact name.

So one possible solution is to scan the repository directory (easy for
for a file system resolver) and extract all the Export-Package
directives. For each export, you can create a duplicate metadata file
that resolves to the real bundle artifact, it's a bit wasteful of
space as you have a lot of redundant metadata. But it would allow Ivy
to resolve a package import in it's natural manner.

For example:
com.acme.delta-4.0.0.jar
Import-Package: com.acme.echo.aaa;version="5.0.0"
Import-Package: com.acme.echo.bbb;version="5.0.0"
Import-Package: com.acme.echo.ccc;version="5.0.0"

com.acme.echo-5.0.0.jar
Export-Package: com.acme.echo.aaa
Export-Package: com.acme.echo.bbb
Export-Package: com.acme.echo.ccc

Using Bushel, if the com.acme.delta bundle uses a Require-Bundle
directive for com.acme.echo it would result in something like the
following ivy.xml fragment:
<dependency org="com.acme" name="echo" rev="[5.0.0,6.0.0)"
conf="default->default" />

Ivy would then look for a metadata descriptor in the cache for:
{ivy-cache-dir}/com.acme/echo/ivy-5.0.0.xml

If, in a future version of Bushel, it scanned com.acme.echo for Export-
Package and produced a set of virtual artifact metadata for each
package, such as:
{ivy-cache-dir}/_pkg_/com.acme.echo.aaa/ivy-5.0.0.xml
{ivy-cache-dir}/_pkg_/com.acme.echo.bbb/ivy-5.0.0.xml
{ivy-cache-dir}/_pkg_/com.acme.echo.ccc/ivy-5.0.0.xml

Then Bushel could convert the com.acme.delta bundle Import-Package
directives into:
<dependency org="_pkg_" name="com.acme.echo.aaa" rev="5.0.0"
conf="default->default" />
<dependency org="_pkg_" name="com.acme.echo.bbb" rev="5.0.0"
conf="default->default" />
<dependency org="_pkg_" name="com.acme.echo.ccc" rev="5.0.0"
conf="default->default" />

These metadata descriptors would actually publish the real bundle
artifact, allowing it to be resolved by Ivy.

At this stage it's just a theoretical solution, I haven't written any
code yet, but it seems like it should work.

Cheers,
Alex


On Aug 25, 6:41 pm, Nicolas Lalevée <nicolas.lale...@hibnet.org>
wrote:

[Nicolas Lalevée]

Le 26 août 08 à 09:30, alexr a écrit :

Ok I see. But then I see three issues.

The first is about the bandwidth and the number of HTTP connection
required for remote repositories. First a client have to retrieve the
"_pkg_" metadata, so for one bundle metadata, there are about ten
different ivy.xml to retrieve. That is a lot, and can be quite painful
for big projects.

Then there an issue with trying to satisfy an import package against
an export package that has not yet been cached. A bundle in the
repository can export some package, but its jar has not been retrieved
yet because not module explicitly required it. So there is no way on
the client side to know that that bundle will satisfy some import
package. Unless that on the client side you try to retrieve every
bundle that exist on the repository, which will be like getting the
entire remote repository locally.

And I see a issue with the conflict management.
If my bundle A require the bundle B, and import the version 1.0 of a
package of C. And my bundle B require the version 1.1 of C.
So here there are some ambiguities about the version of C to retrieve
that need to be resolved.
But Ivy will only know about two different module id: a
"_pkg_.com.me.C.package" and a "com.me.C".
Or you are willing to implement a special resolver ? So when Ivy will
ask to the resolver a module id with an organization name "_pkg_", it
will try to resolve the metadata of a "_pkg_.com.me.C.package", but
will return the real module descriptor : "com.me.C".

But if you are at implementing a resolver, I think that it would be
better to support a real OSGi repository. The idea would be to get the
OSGi repository descriptor instead of creating a lot of fake module
(here is an exemple of descriptor of OSGi repository: http://felix.apache.org/obr/releases.xml)
. On the repository side, instead of having a process that create the
_pkg_ modules hierarchy, we would have a process that generate that
repository.xml. Thought the ivy.xml generated by bushel should sill
contain the dependency declaration against "_pkg_" fake modules.
So every package dependency is retrieve once, so you save a lot of
HTTP connection. And I think that the process on the repository side
will be more efficient.
But as said, it will require some process on the repository side. I
don't think this should be a blocker. I think that production ready
projects should have their own repository, even if they are just
replicas of public ones, at least they have their own backup and can
have an hand on it.

I have to admit that I got these ideas along with writing that reply,
and I got more but I am not sure yet of the direction to get. So what
do you think ?

Nicolas

[alexr]

Hi Nicolas,

I don't think I properly explained how it works, basically Bushel
treats an
OSGI bundle as both the artifact-metadata and the artifact. So in
response
to the HTTP bandwidth issue, only one file is downloaded and the _pkg_
fake
metadata files are created in the local ivy cache only. As you know,
this
means subsequent calls for the imported _pkg_ will come from the local
cache
for as long as the cache TTL allows.

Also regarding satisfying the uncache bundles, there will definately
be
techincal problems to overcome with regard to unkown package imports.
What I
was proposing is scanning the entire remote repository once and
caching the
_pkg_ metadata in the local cache (note, just the metadata, not any
unused
artifacts). Subsequent updates to the remote repo would result in
incremental changes to the _pkg_ metadata. Unless the remote
repository is
huge, it don't see this as a show stopper because it should be very
fast
once cached. Another alternative is to create a Bushel-managed file in
the
Ivy cache directory that can provide the mappings, rather than using
the
builtin Ivy behaviour, but I would like to avoid this if possible.

I have not looked into conflict management yet, I know it may cause me
some
problems. Also, I like the idea of support the OBR services, I have
created
a ticket (http://code.google.com/p/bushel/issues/detail?id=10) and
will
hopefully get to work on it in the near future.

I have to admit, Bushel started off with more modest goals and it
seems that
I could keep adding more and more functionality. For example, once the
ground work is done, I think Ivy would make a good provisioning backed
for
OSGi services. I think it would also help if I got in touch with the
Ivy
developers, rather than working completely in isolation. I may have to
do
that soon.

Cheers,
Alex



On Aug 26, 12:42 pm, Nicolas Lalevée <nicolas.lale...@hibnet.org>
wrote:

[alexr]

I meant OSGi provisioning *back-end*.

[Nicolas Lalevée]

Le 27 août 08 à 09:37, alexr a écrit :

Once cached yes. But to get it cached it will take a lot of time.
And getting the update will require as many http connection as they
are artifacts, so it is very consuming too.

I have an environment where we put every of our libraries in svn. It
contains nows nearly 400 modules. Doing a svn up of that svn
repository is really a pain, so we changed a little bit our config:
we can now retrieve the artifacts remotely from svn via ivy, not via a
manual svn up, so ivy can just select the needed artifacts, it saves a
lot of time waiting for tools. So I don't think having a local copy of
a remote repository is very workable.

Maybe you can take advantage of the fact that the META-INF directory
is always the first content in the stream of the jar (see 2.3.1 in
OSGI core spec R4.1), so you could cut the download early, but this
still need a lot of HTTP requests.

I think we can agree on one process: try to get a repository.xml, and
if there no such file, then does a processing of every jars. WDYT ?

> Another alternative is to create a Bushel-managed file in
> the
> Ivy cache directory that can provide the mappings, rather than using
> the
> builtin Ivy behaviour, but I would like to avoid this if possible.
>
> I have not looked into conflict management yet, I know it may cause me
> some
> problems. Also, I like the idea of support the OBR services, I have
> created
> a ticket (http://code.google.com/p/bushel/issues/detail?id=10) and
> will
> hopefully get to work on it in the near future.
>
> I have to admit, Bushel started off with more modest goals and it
> seems that
> I could keep adding more and more functionality. For example, once the
> ground work is done, I think Ivy would make a good provisioning backed
> for
> OSGi services.

I agree, Ivy can be a good backend. I looked to the OGSi Bundle
Repository service implemented in Felix, and the dependency management
is quite poor (it is a really new project thought, nothing to blame on
them), the confict management is just a "first match wins".

> I think it would also help if I got in touch with the
> Ivy
> developers, rather than working completely in isolation. I may have to
> do
> that soon.

You are not totally in isolation, I am there :). Well I mainly
contribute to IvyDE, but I have some Ivy knowledge.
But of course, every discussion related to Ivy improvement should
happen on the ant-dev list.

Nicolas

[Jérôme]

Hi Nicolas, and Alex,

First, i'm the second developper (frenchy like you :)) of the young
Bushel project.

How to manage OSGi Import-Package and Export-Package is an excellent
debate! Indeed, combine dependencies management via package and via
module id is not easy.

I don't sure that use "<dependency org="_pkg_"
name="com.acme.echo.aaa" ... is the best way.
I think we must prefer use specific extra medatas to add OSGi semantic
in Ivy data model.

In example :

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:osgi="http://ant.apache.org/ivy/osgi">
<info osgi:bundle="true" organisation="com.acme" module="test"
revision="1.0.1" status="release"/>
<publications>
<artifact name="com.acme.test" type="jar"
osgi:Export-Package="com.acme.pkg1,com.acme.pkg2"/>
</publications>
<dependencies defaultconf="bundle">
<!-- like Require-bundle -->
<dependency org="com.acme" name="alpha" rev="1.0.0"/>
<!-- like Import-Package -->
<dependency org="com.acme" name="beta" rev="1.0.0"
osgi:Import-Package="com.acme.alpha.pkg1, com.acme.alpha.pkg2"/>
</dependencies>
</ivy-module>

And I agree to use repository.xml file publish on repository side like
http://felix.apache.org/obr/releases.xml to optimitize HTTP
connections!

What do you think about it ?

Cheers,
Jérôme.

--
"Code is a model, Show me the model!"
http://www.benois.fr

[Nicolas Lalevée]

Le 28 août 08 à 23:40, Jérôme a écrit :

Unfortunately this won't work. Because when specifying an import
package, we don't know yet which bundle will fit that requirement. So
we cannot have a org="com.acme" name="beta". Althougth it might be a
good idea to decorate the ivy.xml with some custom attributes.
About having a fake _pgk_ organisation, I don't think we can bypass it
as Ivy need an module id to compute dependencies. One way or the
other, even with a pure OSGi repository.xml, that fake _pgk_ would
have to exists.

Nicolas

[alexr]

Agreed. I too thought we could solve this using extra attributes, but
hit the same wall. I don't think the _pkg_ solution is ideal, but
don't see much of an alternative.

Jerome, I think your idea could work where the bundle name and the
package imports were similar, such as your example. But in the wild,
bundle names and package imports are named independently. For example,
look at the bundle names in the SpringSource bundle repo, log4j looks
something like: com.springsource.org.apache.log4j. If you had a Import-
Package for org.apache.log4j, you couldn't find the bundle you need
from the package name alone. Hence the needs to pre-process the
bundles and produce the fake _pkg_.

Sorry if I've been a bit lax in replying, I've been trying to setup an
Hudson build on a fresh Ubuntu server and have been struggling a bit.
I will hopefully finish it up today.

Also, first things first, Bushel needs a URL Resolver, it shouldn't be
too hard, but all this _pkg_ talk is academic until we can get that
finished. I am aiming to do some work on Sunday, so if anyone has any
input just post it here and I'll have a read then.

If you are interested Nicolas, I can make you a project member, Jerome
do you have any objections? The more the merrier I say ;-)

Cheers,
Alex

On Aug 29, 1:48 am, Nicolas Lalevée <nicolas.lale...@hibnet.org>
wrote:

> >http://felix.apache.org/obr/releases.xmlto optimitize HTTP

[Nicolas Lalevée]

Le 29 août 08 à 09:01, alexr a écrit :

>
> Agreed. I too thought we could solve this using extra attributes, but
> hit the same wall. I don't think the _pkg_ solution is ideal, but
> don't see much of an alternative.
>
> Jerome, I think your idea could work where the bundle name and the
> package imports were similar, such as your example. But in the wild,
> bundle names and package imports are named independently. For example,
> look at the bundle names in the SpringSource bundle repo, log4j looks
> something like: com.springsource.org.apache.log4j. If you had a
> Import-
> Package for org.apache.log4j, you couldn't find the bundle you need
> from the package name alone. Hence the needs to pre-process the
> bundles and produce the fake _pkg_.
>
> Sorry if I've been a bit lax in replying, I've been trying to setup an
> Hudson build on a fresh Ubuntu server and have been struggling a bit.
> I will hopefully finish it up today.
>
> Also, first things first, Bushel needs a URL Resolver, it shouldn't be
> too hard, but all this _pkg_ talk is academic until we can get that
> finished. I am aiming to do some work on Sunday, so if anyone has any
> input just post it here and I'll have a read then.

Could it just be a wrapper of another resolver instead ? So that OSGi
resolver is independent of the transport layer, and benefit of every
existing implementation.

>
>
> If you are interested Nicolas, I can make you a project member, Jerome
> do you have any objections? The more the merrier I say ;-)

I would be happy to join and contribute !
As you should know my principal interest here will be to support the
OSGi bundle repository specification.

Nicolas

[Jérôme]

Hi,

Alex, I use already Hudson with Ivy2 with this patch
(http://www.benois.fr/hudsonIvy2.patch), maybe it will help you.

You're write my proposal doesn't solve Import-Package use case. And I
think also the _pkg_ solution isn't ideal. But maybe you can use fake
org=_pkg_ in internal and extends ivy semantic with a complete osgi
namespace.

In example this bloc:
<dependencies>

<dependency org="com.acme" name="alpha" rev="1.0.0"/>

<osgi:Import-Package name="com.acme.alpha.pkg1" rev="1.0.0"
visibility="reexport"/>
<osgi:Import-Package name="com.acme.alpha.pkg2" rev="1.0.0"
resolution="optional" />
</dependencies>

represent this :
<dependencies>

<dependency org="com.acme" name="alpha" rev="1.0.0"/>

<dependency org="_pkg_" name="com.acme.alpha.pkg1" rev="1.0.0"
osgi:visibility="reexport"/>
<dependency org="_pkg_" name="com.acme.alpha.pkg2" rev="1.0.0"
osgi:resolution="optional" />
</dependencies>

It's similar concept of Namespace supported by Spring Framework.
Indeed, spring provide an extensible syntax grammar with pluggable
NamespaceHandlerSupport, see : http://tinyurl.com/5n4a4w

Maybe we can choose the same way to propose clean OSGI Ivy module
descriptor without visible fake _pkg_.

What do you think ?

And a great +1 if Nicolas want to became a project member of Bushel!

[alexr]

> Maybe you can take advantage of the fact that the META-INF directory
> is always the first content in the stream of the jar (see 2.3.1 in
> OSGI core spec R4.1), so you could cut the download early, but this
> still need a lot of HTTP requests.

The current implementation may already do as you suggest. The file
resolver uses the JarHandlingRepository I wrote, which itself uses the
JarInputStream, to pick out the MANIFEST.MF jar entry. This should
hopefully not need to extract the entire Jar when loaded via a URL.

> > Also, first things first, Bushel needs a URL Resolver, it shouldn't be
> > too hard, but all this _pkg_ talk is academic until we can get that
> > finished. I am aiming to do some work on Sunday, so if anyone has any
> > input just post it here and I'll have a read then.
>
> Could it just be a wrapper of another resolver instead ? So that OSGi  
> resolver is independent of the transport layer, and benefit of every  
> existing implementation.

That is the end goal, but at the moment there is some custom version
handling behaviour required to handle the qualifier in
major.minor.micro.qualifier versions. Because Ivy requires an explicit
version I had to convert a version, for example 1.2.3 into
[1.2.3,1.2.4), to take advantage of the built-in Ivy pattern
resolving. There is probably a better way to do this, but I'm learning
the Ivy internals as I go :-)

> > If you are interested Nicolas, I can make you a project member, Jerome
> > do you have any objections? The more the merrier I say ;-)
>
> I would be happy to join and contribute !
> As you should know my principal interest here will be to support the  
> OSGi bundle repository specification.

Great to hear, I've added you as a member. I'm happy for you to
contribute whatever you are interested in.

Cheers,
Alex